acceptodds
Under review as a conference paper at ICLR 2027

BILEVEL-BOOST: INTERLEAVING HYPERPARAMETER OPTIMIZATION WITH GRADIENT BOOSTING

Abstract

Gradient-boosted trees are powerful tabular predictors, but tuning them can require many expensive complete training runs. This cost can dominate model development and becomes especially burdensome in repeated experiments and rolling estimation. We replace repeated full-model search with HPO decisions made inside a single boosting trajectory. BILEVEL-BOOST grows alternative next blocks from a shared accepted prefix and uses validation feedback to adapt hyperparameters stage by stage, spending computation on local counterfactuals rather than discarded com- plete ensembles. This creates a tradeoff: smaller blocks increase control resolution but also increase the number of validation-dependent decisions. With tree budget T , block size b, HPO sample size nH , and at most q outcomes per intervention, we bound validation-selection error by O(pT log q/(bnH )). Combined with block- wise approximation error O((b/T )α), this yields a principled block-size scaling law. On a 100-seed drifting Friedman benchmark, BILEVEL-BOOST matches 8-trial Optuna in MSE using about 5× less wall-clock and 6× fewer trees; 40-trial Optuna improves MSE by only 1.06 at about 18× the wall-clock. Matched-trial open-loop schedules underperform, while block-size sweeps exhibit the predicted interior optimum and its qualitative shift with HPO sample size. In a rolling- window application to non-stationary financial data, BILEVEL-BOOST achieves roughly comparable predictive performance to a large-scale SOBER-tuned baseline using about 1/30 of the CPU-hours.

Primary area:
optimization
Keywords:
Gradient boosting, XGB, Hyperparameter optimization, Statistical ML

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m2uBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Bilevel-Boost, a method that integrates HPO into the GBDT training process to reduce the computational cost of repeatedly training complete models. Instead of selecting a fixed set of HPs for the entire training process, Bilevel-Boost divides trees into blocks and adjusts HPs at each stage. Starting from a shared model prefix, it trains and evaluates candidate blocks with different configurations, then selects one to continue training. The method combines learning-rate grid search, L2 regularization probes, and sign-SPSA to adapt hyperparameters based on validation feedback.

The paper also provides a theoretical analysis of the tradeoff introduced by block size. Smaller blocks allow more frequent hyperparameter adjustments but increase the risk of overfitting to the validation data. Experiments on a drifting Friedman synthetic benchmark with 100 random seeds show that the proposed method achieves comparable MSE to 8-trial Optuna while using approximately 5× less wall-clock time and 6× fewer trained trees. Ablation studies suggest that the SPSA-based adaptation of structural and sampling parameters contributes most of the improvement, while block-size experiments qualitatively support the theoretical prediction. The authors also report a rolling-window experiment on proprietary financial data, where Bilevel-Boost achieves roughly comparable predictive performance to a large-scale baseline using about 1/30 of the CPU-hours.

Critical strengths

S1. The paper is well organized and clearly presented. It starts by discussing the computational cost of conventional HPO and naturally motivates the idea of integrating HPO into the boosting process. The method and theoretical analysis are presented clearly, with figures and pseudocode that make the overall approach easy to follow.

S2. The method is intuitive and has practical potential. Bilevel-Boost takes advantage of the incremental nature of GBDT training by reusing previously trained model prefixes, reducing the need to repeatedly train complete models. The approach does not require differentiable optimization and can be integrated into existing GBDT frameworks without major modifications.

S3. The theoretical analysis is interesting. The paper examines the tradeoff between block size and adaptive HPO. Smaller blocks allow more frequent hyperparameter updates but may introduce greater validation-selection error. The derived block-size scaling law offers a possible explanation for this tradeoff and provides some guidance on choosing the intervention frequency.

Critical weaknesses

W1. Limited dataset diversity and scale, with insufficient details about the financial experiment. The main experiments are conducted on a single synthetic dataset with only 8,000 samples and 10 features. Although the authors use 100 random seeds, the evaluation lacks datasets of different sizes, feature dimensions, and public real-world benchmarks. Moreover, the financial experiment reports substantial computational savings but provides few details about the dataset size, feature dimensions, rolling-window settings, or specific prediction metrics. This makes it difficult to assess the generalizability and practical effectiveness of the method.

W2. Lack of evaluation on an independent test set. Appendix E shows that the audit interval is used both to select the final model length and to report the main performance metrics. Although this does not necessarily lead to future data leakage, using the same data for model selection and evaluation may introduce selection bias. The authors are encouraged to evaluate their method on a separate test interval to better support the reported accuracy–efficiency tradeoff.

W3. Limited comparison with GBDT-specific optimization methods. The paper mainly compares against traditional HPO methods such as Optuna and Hyperband, with limited discussion of methods designed specifically for GBDT optimization. For example, SigOpt Mulch (2023) combines meta-learning and multi-fidelity optimization to reduce GBDT tuning costs, while Better by Default (NeurIPS 2024) uses pre-tuned hyperparameters to reduce the need for repeated HPO. Comparisons with or further discussion of these methods would help better establish the advantages of Bilevel-Boost.

Questions for authors

Q1. Could the authors provide experiments on larger and more diverse datasets, along with more details about the financial experiment? (see W1)

Q2. How would the reported results change if the models were evaluated on a fully independent test interval? (see W2)

Q3. It would be helpful to include comparisons with or further discussion of GBDT-specific optimization methods, such as SigOpt Mulch and Better by Default. (see W3)

Reviewer_HxCSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

According to the authors, their first contribution is "a warm-started controller that spends HPO computation on paired next-block counterfactuals along one accepted booster trajectory". To understand the algorithm more concretely, my interpretation of its core mechanism is the following:

(1) Start with the current model and its hyperparameter search center.

(2) Generate two nearby hyperparameter configurations using symmetric random perturbations.

(3) Starting from the same current model, train an additional block of 10 trees with each configuration.

(4) Evaluate both extended models on validation data and retain only the better block.

(5) Update the hyperparameter search center by taking a small step toward the winning configuration.

(6) Repeat.

Additionally, the authors also derive a theoretical bound that makes a useful prediction: more validation observations permit more frequent hyperparameter updates. In other words, every time the algorithm selects a block using validation data, it risks overfitting to the validation set. With more validation data, the algorithm can afford to make more such block selections. The bound is not specific to SPSA or the specific winner-branch controller introduced in this work. The authors give a theoretical analysis of the class of finite-branching adaptive controllers. While the analysis provides justification for the tradeoff between intervention frequency and validation overfitting, it does not establish that the particular controller proposed in this paper is preferable to simpler alternatives.

Critical strengths
  1. The authors propose a hyperparameter controller for gradient-boosted trees that, at each training stage, evaluates two candidate blocks grown from the same existing ensemble, retains the better-performing block, and updates the hyperparameter search center using a sign-SPSA rule. The claimed novelty lies in combining these mechanisms to adapt hyperparameters along a single boosting trajectory, rather than repeatedly training complete models under different hyperparameter configurations.

  2. The authors provide a theoretical bound that makes a useful prediction: more validation observations permit more frequent hyperparameter updates (more details above).

Critical weaknesses

See the algorithmic steps (1)–(6) described above. The general principles underlying this procedure—adapting hyperparameters during training, reusing trained model states, and selecting promising continuations—have already been explored in previous work [1,2]. In particular, [1] proposes repeated hyperparameter search starting from an existing gradient-boosted tree ensemble, while [2] develops population-based hyperparameter adaptation. The specific contribution here therefore seems to lie in the proposed combination of paired next-block evaluations, winner-branch selection, and sign-SPSA updates of a persistent hyperparameter search center. However, I find the experimental justification for this particular controller insufficient. In particular, a natural and computationally identical baseline would retain steps (1)–(4), but replace step (5) with direct adoption of the winning configuration as the next search center. In other words, instead of taking a small step toward the winner, why not simply move to the winner? This would amount to replacing the proposed sign-SPSA update with a greedy winner-adoption update, while keeping the same candidate evaluations, accepted blocks, and computational budget. This comparison seems especially important because the two procedures share the main proposed computational architecture. It would help establish whether the particular controller update offers a meaningful advantage beyond the already established principles of warm-started adaptive hyperparameter search. The existing no-SPSA ablation does not appear to answer this question, since it does not isolate the effect of the persistent-center update.

In short, it seems to me that a lot of the proposed method's novelty lies within very specific additional machinery (sign-SPSA update, small symmetric perturbations) that lacks an ablation.

[1] Patrick Crenshaw (2021), Re-searching Hyperparameters for Training Boosted Tree Models. https://www.crowdstrike.com/en-us/blog/re-searching-hyperparameters-for-training-boosted-tree-models/

[2] Jaderberg et al. (2017), Population Based Training of Neural Networks. https://arxiv.org/abs/1711.09846

Questions for authors

The questions that would affect my assesment the most are:

Why are two symmetric SPSA perturbations better than random configurations near the current center?

Why take a small step toward the winning configuration rather than adopt the winning configuration directly?

Essentially, does the proposed method improve upon a straightforward greedy blockwise HPO algorithm?

Reviewer_Z5TQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper replaces outer-loop hyperparameter search for gradient-boosted trees with decisions made inside one boosting trajectory. At each block, Bilevel-Boost grows candidates from the shared accepted prefix: an eta grid probe, one paired sign-SPSA comparison over six structural controls whose winner is committed, and a discarded L2 probe that updates the controller state. A finite-sample bound trades validation-selection error, which grows with intervention frequency, against schedule approximation error, giving a block-size scaling law. On a 100-seed drifting synthetic benchmark the method matches 8-trial Optuna at about a fifth of the wall-clock, trails 40-trial Optuna by a small but significant margin, beats trial-matched open-loop schedules, and shows the predicted interior block-size optimum. A proprietary rolling-window financial application reports comparable performance to SOBER tuning at a thirtieth of the CPU-hours.

Critical strengths
  1. The decomposition and the block-size analysis give the paper real explanatory value. Fixed, open-loop, and feedback schedules are nested and each is tested at a matched budget (Section 6.2), and the finite-sample bound predicts an interior block-size optimum that the sweep shows (Section 5, Appendix H.4).This frames in-trajectory HPO as a measurable tradeoff between schedule resolution and validation reuse, which is useful beyond this method.

  2. The evaluation is carefully designed. One hundred shared seeds with paired confidence intervals, informative ablations (removing eta adaptation helps slightly), a measured common-random-numbers effect, and a compute frontier. This is consequential because the central claim is an efficiency-frontier claim, and the paired design makes it credible on the benchmark it covers.

Critical weaknesses
  1. The evaluation segment is also the method's selection segment. Algorithm 1 returns the prefix with the best audit loss, and the best configuration also early-stops on audit, while the Optuna baselines select model length on the meta split and never touch audit (Appendix E, J.2). Every reported metric is computed on audit. The parity with 8-trial Optuna is a statistical tie, and no result on data outside the estimation window is reported; The authors might rerun the benchmark with meta-early-stop for Bilevel-Boost, or score all methods on observations after the estimation window, which the intended rolling use requires anyway.

  2. The closest prior work is absent, and the public validation is synthetic. Population Based Training (Jaderberg et al., 2017) performs HPO inside training with PB2 (Parker-Holder et al., 2020) an extension; Freeze-Thaw Bayesian optimization (Swersky et al., 2014) allocates tuning effort across partially trained models; FLAML (Wang et al., 2021) targets cheap GBDT tuning directly. None are cited, so both the novelty claim and the baseline set are incomplete, and no public real dataset is evaluated. The authors ought to position against this line, add a PBT-style or FLAML baseline at matched compute, and validate on public tabular data.

Questions for authors
  1. What are the primary-benchmark results when Bilevel-Boost selects trajectory length on the meta split, with audit used only for final scoring?

  2. How were the controller constants chosen (eta grid, perturbation radii, sign steps, probe radius, patience), and on what data? Section 4 reports perturbation radii of 0.5 while Appendix B.3 and the Proposition 4 example use 0.1.

  3. In the block-size sweep, the tree-fit sample shrinks as the HPO sample grows, and every entry of the 1600 row beats the 3200 row (Table 11). Does the predicted shift in optimal block size hold with the tree-fit sample held fixed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.