acceptodds
Under review as a conference paper at ICLR 2027

Gradient Boosting with Learnable and Incrementally Refined Trees

Abstract

Gradient boosting builds an ensemble by sequentially adding weak learners while keeping previously learned learners fixed. We revisit this design choice and ask whether boosting can benefit from jointly refining previously added learners as new learners are introduced. We investigate this question for gradient-boosted decision trees by replacing discontinuous split thresholds with differentiable routing functions, treating both tree decision boundaries and leaf values as continuously learnable parameters. Based on this formulation, we propose the Co-adaptive Gradient Boosting (CGB) framework and three algorithmic variants: CGB-AP, CGB-OB, and CGB-RO. All three methods perform gradient-based updates to the parameters of previously added trees. CGB-AP restricts boundary updates to axis-parallel directions, while CGB-OB permits oblique boundary updates. In both methods, newly added trees are initialized using standard purity-based greedy construction. CGB-RO instead bypasses greedy initialization entirely, starting each new tree randomly as an oblique decision tree and performing a localized gradient-based update before incorporating it into the ensemble. Across standard tabular benchmark datasets, we find that continuously updating previously added trees consistently matches or improves upon conventional discrete gradient boosting. These results suggest that freezing earlier weak learners is not a strict necessity of the boosting framework, and that incremental, differentiable refinement of the ensemble provides a promising alternative to the standard stagewise optimization paradigm.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Gradient Boosting, XGBoost, Oblique Decision Trees, Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_HqNdOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes CGB, a differentiable gradient boosting framework that updates previously added weak learners before each new tree is added.

Critical strengths

The idea of differentiable weak learner is a natural extension of boosting, and the paper is easy to follow.

Critical weaknesses

The experiments are too limited, comparing CGB only against XGBoost. Being a generic framework for differentiable boosting, it should be able to be combined with any existing boosting algorithms (e.g., LightGBM, CatBoost). The paper lacks a systematic view of the experiments. A single baseline is far from enough.

Questions for authors

The experiments limit XGBoost to 30 trees, while giving CGB up to 100 Adam epochs in every iteration. Currently the number of optimization steps is very unbalanced, ~30 for XGBoost vs up to ~30×100 for CGB. How does CGB compare against XGBoost and other non-differentiable boosting algorithms when they have closer optimization budgets (e.g., dropping the 30-tree limit for XGBoost)? If the gap closes, the gains may simply come from more optimization rather than from updating previous trees.

Reviewer_fpbCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes to learn an ensemble of soft decision trees. Instead of learning it end-to-end, it uses a gradient-boosting-style, stage-wise approach. There are three variants of the method: CGB-AP, CGB-OB, and CGB-RO. All of them iterate in the gradient boosting manner, but before adding a new tree, they retrain the current ensemble end-to-end. They differ in the way the new tree is added. In both CGB-AP and CGB-OB, the new tree is grown in a greedy top-down manner. CGB-AP fixes the chosen features and converts the hard tree into an axis-aligned soft tree, while CGB-OB converts it into an oblique soft tree. CGB-RO, instead of using greedy top-down induction, starts from random initial oblique parameters and learns the tree end-to-end on the residuals before adding it to the ensemble. Experiments are performed on 24 public datasets, and the method is compared with an XGBoost baseline.

Critical strengths

None

Critical weaknesses
  • The paper learns an ensemble of soft decision trees. Because these are differentiable end-to-end, there is no need to use a gradient boosting (GB) approach. The only thing this approach might help with is better initialization/generalization, but this was not the focus of the paper. For example, when training neural networks, one does not first learn LL layers, then add one more hidden layer fitted on the residuals, then retrain end-to-end, and so on. Instead, one initializes the whole neural network and trains it end-to-end with backpropagation. In this respect, an ensemble of soft decision trees is closer to a neural network than to an ensemble of hard axis-aligned decision trees. In the latter case, the model is not differentiable, and that is the whole reason greedy top-down induction and gradient boosting approaches are used.
  • In the experiments, the comparison with XGBoost is not fair. XGBoost learns an ensemble of hard axis-aligned decision trees, where only one root-to-leaf path in each tree is used during inference. In CGB, as far as I understand, all root-to-leaf paths are used. Therefore, this is not an apples-to-apples comparison. Another issue is that the hyperparameters of XGBoost were not tuned at all, and the ones used are far from optimal: 30 trees of depth 3 is far too small a model. Apart from XGBoost, other baselines, including the differentiable tree ensembles listed in the paper’s related work, should be included.
  • Several relevant works are missing: “Additive Groves of Regression Trees” by D. Sorokina et al. and “Towards Better Decision Forests: Forest Alternating Optimization” by M. Carreira-Perpiñán et al., in both of which previously added trees are iteratively re-optimized. Another work, “Predictive Learning via Rule Ensembles” by J. Friedman and B. Popescu, jointly optimizes all the leaf parameters of all the trees in the ensemble. All of these works operate on hard trees, without converting them into soft decision trees.
Questions for authors

No questions.

Reviewer_fAiFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores whether boosting can improve by jointly refining past learners as new ones are introduced since GBM adds weak learners sequentially while keeping earlier learners fixed. Hence, the paper proposes Co-adaptive Gradient Boosting (CGB) framework with three variants: CGB-AP, CGB-OB, and CGB-RO; these variants propose different techniques to update previously added trees' parameters through gradient-based methods. CGB-AP limits updates to axis-parallel directions, while CGB-OB allows for oblique updates. CGB-RO starts new trees randomly without greedy initialization. The proposed paper is evaluated on standard datasets, showing that continuously updating previous trees with a fixed time window enhances or matches the performance of conventional discrete gradient boosting, indicating a potential shift away from the traditional approach of freezing early learners in boosting.

Critical strengths

The main strengths of the paper are (i) The paper is well-written, even though it is missing information during the learning stage. (ii) The idea of converting the decision trees into differentiable trees to allow gradient-based techniques to improve the trees once they are added.

Critical weaknesses

However, there are two main weaknesses of the paper. The main weakness is the validation of the proposed approach since the method is only compared with binary classification and XGBoost. This means that multiclass classification is missing, and the paper does not compare the proposed approach with state-of-the-art methods such as LightGBM or other techniques that refine trees after they have been trained.

The second weakness is related to the formulation of the problem and the validation. What happens if we only update the splitting functions and compute v_i using the decision tree formulation? There is no reference about how the refinement improves the decision performance and how the parameters change. Is there any ping pong effect of the decision trees when we move the window? What is the influence of the window, since the windows break the pseudo labels when the new tree is added.

Questions for authors

The critical weaknesses include all my main questions of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.