acceptodds
Under review as a conference paper at ICLR 2027

Variance-Shrunk Residuals for Robust Gradient Boosting

Abstract

Gradient boosting has established itself as a dominant paradigm for structured data. However, its indiscriminate trust in empirical residuals renders it susceptible to noise and outliers. To this end, we propose Uncertainty-Regularized Boosting (URBoost), a framework that leverages the ensemble's internal prediction variance as a proxy for optimization instability. URBoost shrinks pseudo-residual targets according to trajectory variance and solves the resulting variance-regularized tree update through an equivalent weighted least-squares formulation. Theoretically, we formulate URBoost as a variance-regularized functional optimization problem and establish a descent guarantee under smoothness and a sufficiently small learning rate. Statistically, we prove that under a local perturbation model, the trajectory-variance shrinkage coefficient coincides with the optimal Minimum Mean Squared Error linear shrinkage coefficient. Extensive experiments across ten benchmark datasets demonstrate that URBoost achieves strong out-of-the-box performance and robustness against severe label and compound noise, highlighting that an ensemble's internal disagreement provides a useful meta-signal for robust learning.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Ensemble learning, Boosting, Machine learning, Classification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1VcZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper proposes a variant of gradient boosting with an uncertainty-based regulariser for improved robustness to noise and outliers.
  • This is formulated by adding a variance weighted regularisation term to the tree's residual fitting objective which is mathematically equivalent to weighted least squares, predicting a shrunken residual.
  • Experiments are conducted on 10 binary classification datasets from OpenML, where the proposed method shows gains in classification performance over the current state of the art. The proposed method is also shown to be robust under label and feature noise.
Critical strengths
  • Simple and principled method: The proposed idea is simple and interpretable. Using training history to adapt regularisation on specific datapoints is an intuitive and sensible direction, and only involves minor modifications to existing boosting frameworks. The mathematical derivation also is sound and correct.
  • The paper includes multiple ensemble baselines and a tabular foundation model (TabPFN), provides supplementary tuning, runtime, memory, and sensitivity experiments. The reported improvements on datasets such as Madelon and Bioresponse are very substantial, if reproducible and correct (see the weaknesses below)
Critical weaknesses
  • The theoretical results appear correct under their assumptions, but offer limited support for the proposed formulation. Lemma 2.1 is a standard weighted least-squares orthogonality result, Theorem 2.2 follows from the smoothness descent inequality, and both apply to arbitrary per-sample quadratic penalties and therefore do not justify the particular trajectory-variance statistic. Theorem 2.3 assumes that residual-noise variance is proportional to trajectory variance, however this proportionality is the central premise requiring justification. Thus, the theory supports the optimisation formulation but does not support noise identification or improved generalisation.
  • The reported results raise suspicion about the evaluation protocol and implementation. Please find below some examples which suggest this:
    • Table 4, which reports URBoost on Jasmine with 50% label noise at ACC 0.966, higher than at 10% noise (0.937), which in turn is higher than the noise-free Table 1 figure (0.9126). Regardless of noise protocol, it seems strange that accuracy is increasing monotonically with label corruption as high as 50%
    • Figure 4 shows shrinkage coefficients are extremely close to one, with average values between 0.9998–1.0000. These small changes accompany unusually large performance improvements.
    • Table 4 shows essentially identical results for λ∈[0.1,100]\lambda \in [0.1, 100]. The λ=1\lambda=1 and λ=100\lambda=100 rows are identical to four decimals on all four metrics. Such a large difference in the regularisation parameter would be expected to produce a visibly different model. Identical outputs across three orders of magnitude of λ\lambda seems strange.
    • An ablation study with λ=0\lambda=0 would allow us to assess if the performance gains arise from the proposed variance reduction method, or from some other implementation artefact.
  • The paper only compares against vanilla baselines, but not any of the related work identified by the authors. In particular ITBoost (Su et. al. 2026) also uses training history to assess sample reliability, thus evaluating it under identical test splits and noise injection procedures would strengthen the paper significantly.
Questions for authors
  • What was the noise injection protocol that was followed for feature / label noise in the label experiments? Was noise injected for both the train / test data?
  • What were the hyper parameters used for the proposed method?
Reviewer_ugRbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes URBoost to use the internal prediction variance of trees to make the gradient boosting algorithm more robust against noise. More specifically, it modifies the gradient boosting by damping updates on samples whose “trajectory variance” is high. The paper did experiments on many datasets and reported gains over baselines.

Critical strengths

The paper performed the experiment over a large set of datasets (~10) and the logic of the paper is clear.

Critical weaknesses

The novelty over ITBoost, Active Bias, GBDT training-dynamics methods and Robust-GBDT is incremental and is not discussed or tested empirically.

I found several closely related works.

  1. ITBoost: Information-Theoretic Trust for Robust Boosting (Su et al., arXiv preprint arXiv:2605.04671) The core idea is very similar. ITBoost examines the evolution of each sample’s residuals across iterations rather than instantaneous error. It then treats samples whose residual patterns fluctuate irregularly as less trustworthy and down-weights them during boosting. URBoost cites ITBoost only in one sentence at the very end of Related Work, does not compare its results with ITBoost, and

  2. Active Bias (Chang, Learned-Miller & McCallum, NeurIPS 2017) This paper re-weights training samples using the variance in predicted probability across iterations of mini-batch SGD. That is the same signal as URBoost’s trajectory variance, in a neural-network setting.

  3. Improving Data Quality with Training Dynamics of GBDTs (arXiv 2210.11327) This work proposes metrics computed from the training dynamics of GBDTs to assess the behavior of each training example. It targets noisy-label detection to remove or relabel samples, and compares against confident learning and a robust boosting algorithm. It is directly relevant prior art for per-sample trajectory statistics in GBDTs and is not cited.

  4. Robust-GBDT: Leveraging robust loss for noisy and imbalanced classification with gbdt (Luo et al., 2025) This is a noise-robust boosting model that integrates the GBDT framework with robust (non-convex) losses. The paper cites it in a list but never compares against it.

Questions for authors

Clarifications that could change my assessment:

  1. Compare with ITBoost, Active Bias, GBDT training-dynamics methods and Robust-GBDT baselines.
  2. How exactly is label noise injected (flip rate, symmetric/asymmetric, applied before or after the train/test split), and are test labels clean or corrupted? How can URBoost reach ~97% accuracy and AUC 0.994 at 50% label noise on Jasmine, when uniform 50% flipping makes labels independent of features? Why does accuracy increase with noise in Table 4, and why does it exceed the clean-data accuracy in Table 1?
  3. What are URBoost’s default hyperparameters and base implementation, and how do they compare to the baselines’? Please provide results for the same implementation with λ = 0 on all datasets and noise levels. This is the single most important missing experiment.
Reviewer_G8XKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes URBoost, a gradient-boosting variant aimed at robustness to label noise. At each iteration, every sample's pseudo-residual rr is shrunk by a factor W=1/(1+λσ2)W = 1/(1+\lambda\sigma^2), where σ2\sigma^2 is the variance of the previous trees' outputs on that sample ("trajectory variance"). The tree is then fit by weighted least squares, with targets WrWr and weights 1+λσ21+\lambda\sigma^2. The authors show this is exactly equivalent to adding a per-sample ridge penalty λσ2h(x)2\lambda\sigma^2 h(x)^2 to the least-squares tree fit. The resulting leaf value is ∑r/∑(1+λσ2)\sum r / \sum (1+\lambda\sigma^2), with both sums taken over the samples in the leaf.

Theoretically, the paper shows that:

  • (i) the fitted tree satisfies a weighted orthogonality condition and is positively aligned with the residual (Lemma 2.1);
  • (ii) the training loss decreases for a sufficiently small step size under LL-smoothness (Thm 2.2);
  • (iii) WW coincides with the MMSE linear shrinkage coefficient if the residual's noise variance is assumed proportional to σ2\sigma^2 (Thm 2.3).

Empirically, URBoost with default λ=1\lambda=1 is reported to outperform RF, AdaBoost, GBDT, XGBoost, LightGBM, CatBoost and TabPFN on ten OpenML binary classification datasets (Table 1), and to outperform tuned baselines on five of them (Table 2). It is also reported to remain stable under 0–50% label and compound noise on Madelon and Jasmine (Figs. 2–3). Shrinkage-weight trajectories are visualized in Fig. 4, and a λ\lambda-sensitivity study is given in Table 4.

Critical strengths

A simple, cheap mechanism with an exact optimization reading. Rewriting the variance-regularized objective (Eq. 2) as an exactly equivalent weighted least-squares problem (Eq. 4) is clean. It makes the method drop into any tree learner that accepts sample weights, and it keeps the update interpretable as a per-sample trust region. Using per-sample statistics of the optimization trajectory as a reliability signal is a reasonable and practically appealing idea. This would matter for acceptance if the paper showed the mechanism is responsible for the reported gains; however, see the weaknesses below.

Critical weaknesses

Notation: throughout, sums run over the samples in a single leaf, and nn is the number of samples in that leaf.

1. The reported gains cannot be attributed to the proposed mechanism, and some results are impossible.

(a) By the paper's own figure, the shrinkage is numerically negligible. Fig. 4 (Jasmine, 10% noise, λ=1\lambda=1) shows all individual weights W≳0.9991W \gtrsim 0.9991, with cohort averages of about 0.99993 (clean) vs. 0.99991 (noisy) for label noise. Inverting W=1/(1+λσ2)W=1/(1+\lambda\sigma^2) gives λσ2≲9×10−4\lambda\sigma^2 \lesssim 9\times10^{-4} at worst and ≈8×10−5\approx 8\times10^{-5} typically. From Eq. (5), every leaf value is plain GBDT's leaf value times a leaf-level factor:

cURB=cGBDT⋅11+λσˉ2,c^{\text{URB}} = c^{\text{GBDT}} \cdot \frac{1}{1+\lambda\bar\sigma^2},

where σˉ2\bar\sigma^2 is the average of σ2\sigma^2 over the leaf. So leaves differ from GBDT by at most about 0.09%.

For split selection, each child's score (∑r)2/∑(1+λσ2)(\sum r)^2/\sum(1+\lambda\sigma^2) is GBDT's score (∑r)2/n(\sum r)^2/n times a factor in [1−δ,1][1-\delta, 1], with δ≈9×10−4\delta \approx 9\times10^{-4}. Hence a split preferred by GBDT can be overturned only if its score exceeds the alternative's by less than about 0.09% (relative), i.e., only near-exact ties can change. This cannot produce the systematic gains reported, e.g., +22 points over GBDT on Madelon (0.9569 vs. 0.7327) and +11 over TabPFN on Bioresponse (0.9147 vs. 0.8019) (Table 1).

(b) Table 4 confirms this. At λ=0.1\lambda=0.1 the penalty is about 10−510^{-5}, so URBoost is numerically GBDT. Yet it reaches 0.9364 accuracy on Jasmine with 10% noise, versus 0.8083 for GBDT on clean Jasmine (Table 1). Results are also unchanged across a 1000× range of λ\lambda (0.1–100). Both facts indicate the gains come from something other than variance regularization: a different base configuration, implementation, or evaluation pipeline.

Also URBoost's own configuration (number of trees, depth, learning rate) is never reported ("Official recommended parameters" do not exist for a new method). The decisive λ=0\lambda=0 ablation under URBoost's configuration is missing as well.

(c) Some noise results violate basic information limits. If each training label is flipped with probability ρ\rho, then

P(y~=1∣x)=(1−ρ) p(x)+ρ (1−p(x)),P(\tilde y=1 \mid x) = (1-\rho)\,p(x) + \rho\,(1-p(x)),

which equals 0.50.5 at ρ=0.5\rho=0.5: the observed labels are independent of the features. The paper's own baselines confirm this regime, dropping to about 0.5 at 50% noise in Fig. 2. Yet Table 4 reports Jasmine accuracy 0.9658 and AUC 0.9944 at 50% label noise. Accuracy also rises with noise: 0.9126 clean (Table 1), 0.9378 at 10%, 0.9658 at 50%. Fig. 2(a) appears to show the same on Madelon.

Separately, Jasmine baselines collapse to about 0.5 under 50% label noise (Fig. 2b) but stay at about 0.80–0.84 under 50% compound (label + feature) noise (Fig. 3b), although compound noise should be at least as harmful. These point to an error in the noise-injection or evaluation pipeline, so the robustness claims (4.2) cannot currently be trusted.

(d) Reporting overstates the results. Appendix F claims URBoost "consistently outperforms all competitors across every dataset and metric," but Table 2 reports only five of ten datasets. The five omitted include all three where URBoost is beaten in Table 1, even by untuned baselines:

  • Elevators log loss: URBoost 0.2875 ranks 5th of 8 (TabPFN 0.2171, XGB 0.2619, LGBM 0.2732, CAT 0.2828).
  • Letter: AUC 0.9988 ranks 6th; log loss 0.0134 ranks 5th; TabPFN is higher on accuracy and F1.
  • Pol AUC: TabPFN 0.9998 vs. URBoost 0.9995.

In Table 1, baselines that beat URBoost are marked as significantly different. On Elevators log loss, TabPFN is marked at p<0.01p<0.01 and XGB, LGBM and CAT at p<0.05p<0.05; on Letter log loss, TabPFN is marked at p<0.01p<0.01. The significance markers therefore cannot be read as "URBoost is significantly better," as the text implies.

2. URBoost's mechanism is a known regularizer in a new form, and its "instability" signal mostly measures how large a sample's residual is, not whether its label is wrong.

(a) URBoost is XGBoost's L2 leaf penalty, with the penalty strength set by trajectory variance. The paper does not mention this connection, but it follows directly from Eq. (5).

In XGBoost, each leaf outputs w=−G/(H+Λ)w = -G/(H+\Lambda). Here GG and HH are the sums of the first and second derivatives of the loss over the samples in the leaf, and Λ\Lambda is the L2 penalty (reg_lambda). For squared error, each sample's first derivative is −r-r and its second derivative is 1. The leaf value therefore becomes ∑r/(n+Λ)\sum r/(n+\Lambda): the average residual, pulled slightly toward zero.

URBoost's leaf value (Eq. 5) is ∑r/(n+λ∑σ2)\sum r/(n+\lambda\sum\sigma^2). This is the same formula, with the fixed constant Λ\Lambda replaced by λ∑σ2\lambda\sum\sigma^2. Expanding URBoost's objective for a single leaf confirms it:

∑[(r−w)2+λσ2w2]=const+2[Gw+12(H+λ∑σ2)w2].\sum\big[(r-w)^2+\lambda\sigma^2w^2\big] = \text{const} + 2\Big[Gw + \tfrac12\big(H+\lambda\textstyle\sum\sigma^2\big)w^2\Big].

The right-hand side is XGBoost's leaf objective, up to a factor of 2 and a constant, with that replacement. Because the objectives match, the split scores match too, not only the leaf values. Put another way, URBoost is XGBoost with no L2 penalty, where each sample's second derivative is raised from 1 to 1+λσ21+\lambda\sigma^2.

This has two consequences. First, the novelty is limited to making an existing penalty depend on trajectory variance. Second, the natural baselines are missing: XGBoost with tuned reg_lambda and min_child_weight, and URBoost implemented as an XGBoost custom objective (gradient −r-r, second derivative 1+λσ21+\lambda\sigma^2). Appendix D does not tune reg_lambda at all.

How much does each method shrink a leaf? Compare each leaf's value with the plain average of its residuals, ∑r/n\sum r/n:

  • XGBoost (default Λ=1\Lambda=1):
    • Under squared error, the leaf value is ∑r/(n+1)\sum r/(n+1), which is smaller than the plain average by a fraction 1/(n+1)1/(n+1). That is 1% for a 100-sample leaf. Even the largest leaf possible on Jasmine (a 10-fold training set holds about 2,686 samples) is shrunk by 1/2687≈1/2687 \approx 0.037%.
    • Under log loss, XGBoost's shrinkage relative to its own unpenalized Newton step −G/H-G/H is 1/(H+1)1/(H+1), where H=∑p(1−p)H=\sum p(1-p). Since p(1−p)≤1/4p(1-p)\le 1/4, we have H≤n/4H\le n/4, so the shrinkage is even larger: at least 1/(2686/4+1)≈1/(2686/4+1)\approx 0.15% on Jasmine.
  • URBoost: the leaf value is the plain average times 1/(1+λσˉ2)1/(1+\lambda\bar\sigma^2), so it is shrunk by about λσˉ2\lambda\bar\sigma^2. Fig. 4 puts this at roughly 0.008% typically and 0.09% at most.

So even XGBoost's weakest shrinkage is at least four to five times larger than URBoost's typical shrinkage, and about 19 times larger under log loss, which is what XGBoost uses on Jasmine. If pulling leaves toward zero were what made URBoost accurate, default XGBoost should do at least as well. Instead, it scores 0.80 on Jasmine against URBoost's 0.91 (Table 1). URBoost does aim its shrinkage at leaves with unstable samples rather than shrinking every leaf evenly, but even its strongest shrinkage (about 0.09%) is far too small to plausibly explain an 11-point gap.

(b) Trajectory variance is high for clean samples too. Take one clean sample that the trees fit well under squared loss. Write r(k)r^{(k)} for its residual when tree kk is fit, and h(k)h^{(k)} for that tree's output on it. Fitting well means h(k)=r(k)h^{(k)} = r^{(k)}. Each update with learning rate ν\nu then shrinks the residual by a constant factor, r(k+1)=(1−ν)r(k)r^{(k+1)} = (1-\nu)r^{(k)}, so the trees' outputs decay smoothly: h(k)=r(1)(1−ν)k−1h^{(k)} = r^{(1)}(1-\nu)^{k-1}.

This is exactly the "stable descent" the paper associates with clean data. Yet the sequence still has nonzero variance. Taking the variance over k=1,…,Kk = 1, \dots, K:

σ2=(r(1))2⋅Var[(1−ν)k−1]≈0.066(r(1))2for ν=0.1 and K=20 trees.\sigma^2 = \big(r^{(1)}\big)^2 \cdot \mathrm{Var}\big[(1-\nu)^{k-1}\big] \approx 0.066 \big(r^{(1)}\big)^2 \quad \text{for } \nu=0.1 \text{ and } K=20 \text{ trees}.

So a sample's trajectory variance grows with the square of its starting residual, whether or not its label is correct. Samples that start with large residuals include mislabeled ones, but also clean samples that are simply hard to fit.

The conclusion concedes that both kinds can have high variance, but the paper never measures how often this happens. Fig. 4(b) fits this reading: the clean cohort's weights also fall steadily over iterations, and the clean–noisy gap is only about 2×10−52\times10^{-5}.

Questions for authors

Clarifications (answers could change my assessment):

  1. (Most important.) Please run URBoost with λ=0\lambda = 0 under exactly the configuration used in the paper, on all ten datasets, both with and without noise. If λ=0\lambda = 0 reproduces the gains in Tables 1–2 and Figs. 2–3, those gains do not come from variance regularization. Please also explain how Jasmine accuracy reaches 0.9658 (AUC 0.9944) under 50% label noise (Table 4), which is higher than the clean result of 0.9126 (Table 1).

  2. What is URBoost's full configuration: number of trees, maximum depth, learning rate, loss, leaf estimator, and base implementation? Is it built on the same tree learner as the GBDT baseline?

  3. What exactly is the noise protocol? Specifically: label flipping or random reassignment; noise injected into training folds only, or into test folds as well; and how compound noise is constructed (does "50% compound noise" include 50% label noise?). How do you reconcile Fig. 2(b), where baselines drop to about 0.5, with Fig. 3(b), where they stay at about 0.80?

  4. Please report the distribution of λσ2\lambda\sigma^2 on each dataset, and the fraction of split decisions that differ between λ=1\lambda = 1 and λ=0\lambda = 0. Fig. 4 implies shrinkage of about 10−410^{-4}; how can that produce gains of 10–20 points?

  5. Why does Table 2 include only five of the ten datasets? Please report tuned baselines on all ten, and revise the claim in Appendix F accordingly.

  6. What do the significance markers denote? In Table 1, several baselines that outperform URBoost are marked as significantly different (e.g., Elevators log loss: TabPFN, XGB, LGBM, CAT).

Suggestions (would strengthen the paper but are not decisive on their own):

  1. Compare against XGBoost with tuned reg_lambda, and against URBoost implemented as an XGBoost custom objective with gradient −r-r and hessian 1+λσ21 + \lambda\sigma^2. This would isolate the variance mechanism from implementation differences.

  2. Please also compare against robust-boosting baselines: Huber or truncated-loss GBDT, Robust-GBDT, and ITBoost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.