- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Bilevel-Boost, a method that integrates HPO into the GBDT training process to reduce the computational cost of repeatedly training complete models. Instead of selecting a fixed set of HPs for the entire training process, Bilevel-Boost divides trees into blocks and adjusts HPs at each stage. Starting from a shared model prefix, it trains and evaluates candidate blocks with different configurations, then selects one to continue training. The method combines learning-rate grid search, L2 regularization probes, and sign-SPSA to adapt hyperparameters based on validation feedback.
The paper also provides a theoretical analysis of the tradeoff introduced by block size. Smaller blocks allow more frequent hyperparameter adjustments but increase the risk of overfitting to the validation data. Experiments on a drifting Friedman synthetic benchmark with 100 random seeds show that the proposed method achieves comparable MSE to 8-trial Optuna while using approximately 5× less wall-clock time and 6× fewer trained trees. Ablation studies suggest that the SPSA-based adaptation of structural and sampling parameters contributes most of the improvement, while block-size experiments qualitatively support the theoretical prediction. The authors also report a rolling-window experiment on proprietary financial data, where Bilevel-Boost achieves roughly comparable predictive performance to a large-scale baseline using about 1/30 of the CPU-hours.
Critical strengths
S1. The paper is well organized and clearly presented. It starts by discussing the computational cost of conventional HPO and naturally motivates the idea of integrating HPO into the boosting process. The method and theoretical analysis are presented clearly, with figures and pseudocode that make the overall approach easy to follow.
S2. The method is intuitive and has practical potential. Bilevel-Boost takes advantage of the incremental nature of GBDT training by reusing previously trained model prefixes, reducing the need to repeatedly train complete models. The approach does not require differentiable optimization and can be integrated into existing GBDT frameworks without major modifications.
S3. The theoretical analysis is interesting. The paper examines the tradeoff between block size and adaptive HPO. Smaller blocks allow more frequent hyperparameter updates but may introduce greater validation-selection error. The derived block-size scaling law offers a possible explanation for this tradeoff and provides some guidance on choosing the intervention frequency.
Critical weaknesses
W1. Limited dataset diversity and scale, with insufficient details about the financial experiment. The main experiments are conducted on a single synthetic dataset with only 8,000 samples and 10 features. Although the authors use 100 random seeds, the evaluation lacks datasets of different sizes, feature dimensions, and public real-world benchmarks. Moreover, the financial experiment reports substantial computational savings but provides few details about the dataset size, feature dimensions, rolling-window settings, or specific prediction metrics. This makes it difficult to assess the generalizability and practical effectiveness of the method.
W2. Lack of evaluation on an independent test set. Appendix E shows that the audit interval is used both to select the final model length and to report the main performance metrics. Although this does not necessarily lead to future data leakage, using the same data for model selection and evaluation may introduce selection bias. The authors are encouraged to evaluate their method on a separate test interval to better support the reported accuracy–efficiency tradeoff.
W3. Limited comparison with GBDT-specific optimization methods. The paper mainly compares against traditional HPO methods such as Optuna and Hyperband, with limited discussion of methods designed specifically for GBDT optimization. For example, SigOpt Mulch (2023) combines meta-learning and multi-fidelity optimization to reduce GBDT tuning costs, while Better by Default (NeurIPS 2024) uses pre-tuned hyperparameters to reduce the need for repeated HPO. Comparisons with or further discussion of these methods would help better establish the advantages of Bilevel-Boost.
Questions for authors
Q1. Could the authors provide experiments on larger and more diverse datasets, along with more details about the financial experiment? (see W1)
Q2. How would the reported results change if the models were evaluated on a fully independent test interval? (see W2)
Q3. It would be helpful to include comparisons with or further discussion of GBDT-specific optimization methods, such as SigOpt Mulch and Better by Default. (see W3)