Alternate loss functions and regression models that achieve robustness to outliers by modulating the learning rate

A new machine learning paper proposes two loss functions that make regression models robust to outliers by automatically scaling down the learning rate when large errors are encountered, rather than letting those errors dominate training [1]. The work, submitted to arXiv on 20 June 2026 and revised on 23 June 2026 by Mathew Mithra Noel, introduces the Square Root Loss (SRL) and the Smooth Mean Absolute Error (SMAE) loss [1]. The core mechanism is a factor embedded in the loss function that reduces the magnitude of parameter updates for very large loss values, preventing the model from selectively fitting outliers [4]. For the SRL-based model, a factor of \(\frac{e}{l}\) scales the learning rate, while the SMAE-based model uses a factor of \(\left(\frac{l(1-l)}{e}+e\right)\) to achieve a similar dampening effect [4]. Standard loss functions struggle with contaminated data. The Mean Squared Error (MSE) assigns very high importance to large errors, while the Mean Absolute Error (MAE), though most robust to outliers, is non-differentiable at the origin and has large derivative values near its minimum, causing instability during training [3]. Differentiable alternatives like the Huber and log-cosh losses have been used in the past to approximate MAE, but the new SRL and SMAE losses are designed to more closely approximate the absolute error while remaining infinitely differentiable [3]. The SRL loss is strictly convex, and the SMAE loss is strictly quasi-convex [3]. The paper presents two new robust linear regression models built on these losses [2]. Linear regression remains the most widely used model in statistics and machine learning due to its low computational cost and interpretability, despite the popularity of deep learning [4]. The authors provide highly vectorized parameter update formulas for both stochastic and batch gradient descent, designed to take advantage of modern GPU hardware [2]. The approach is evaluated on a wide variety of benchmarks and datasets contaminated with outliers, where the SRL and SMAE losses demonstrated superior performance compared to models trained with Huber and log-cosh losses [2]. The broader context of robust loss design has been an active research area. A 2017 paper introduced a general loss function that subsumes several common robust losses, including the Cauchy, Geman-McClure, and Welsch functions, allowing robustness to be treated as a continuous, tunable hyperparameter during training [5]. That work demonstrated how adaptive robustness can improve performance on tasks such as image registration, clustering, and unsupervised monocular depth estimation when paired with neural network training [5].