Variance-Shrunk Residuals for Robust Gradient Boosting

Gradient boosting has established itself as a dominant paradigm for structured data. However, its indiscriminate trust in empirical residuals renders it susceptible to noise and outliers. To this end, we propose Uncertainty-Regularized Boosting (URBoost), a framework that leverages the ensemble's internal prediction variance as a proxy for optimization instability. URBoost shrinks pseudo-residual targets according to trajectory variance and solves the resulting variance-regularized tree update through an equivalent weighted least-squares formulation. Theoretically, we formulate URBoost as a variance-regularized functional optimization problem and establish a descent guarantee under smoothness and a sufficiently small learning rate. Statistically, we prove that under a local perturbation model, the trajectory-variance shrinkage coefficient coincides with the optimal Minimum Mean Squared Error linear shrinkage coefficient. Extensive experiments across ten benchmark datasets demonstrate that URBoost achieves strong out-of-the-box performance and robustness against severe label and compound noise, highlighting that an ensemble's internal disagreement provides a useful meta-signal for robust learning.