acceptodds
Withdrawn from ICLR 2027

CRAFT-GBM: Selecting Best Features for Gradient-Boosted Machine Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feature subsampling is widely used to accelerate gradient-boosted trees, yet standard approaches choose the number of features in advance and largely ignore the split-gain evidence accumulated during training. We introduce CRAFT-GBM (Confidence-Ranked Adaptive Feature Tracking), a native LightGBM extension that uses this evidence to adapt which—and, by default, how many—features are searched in each boosting round. CRAFT-GBM transforms root-split gains into bounded observations and constructs joint time-uniform bounds that remain valid under adaptive, non-stationary training histories. In its default endogenous mode, after initialization, CRAFT-GBM retains exactly the features that cannot yet be ruled out as best, allowing the selected feature-set size to emerge from the observed training history rather than from a pre-specified fraction. With probability at least , every feature with the largest expected gain is retained jointly across all rounds after initialization. When a feature fraction is specified instead, CRAFT-GBM ranks features by their interval point estimates and provides an approximate top- guarantee for the corresponding feature budget . We implement CRAFT-GBM directly within LightGBM while preserving its native feature-bundling representation, and evaluate it across 14 public and synthetic dataset families together with four noise-augmented settings. In the fully endogenous setting, CRAFT-GBM reduces per-fit training time by 39.74% while achieving 1.04% lower test loss in the geometric mean across all 18 RQ1 experiments. Under pre-specified feature fraction , Exogenous CRAFT-GBM also yields aggregate training-time reductions up to 13.52 percentage points larger than random feature subsampling, with no increase in aggregate test loss relative to full-feature LightGBM. These results show that accumulated training evidence can support adaptive feature search with statistical guarantees while retaining the practical advantages of native gradient boosting.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
gradient boosting, adaptive feature selection, confidence sequences, efficient training, tabular learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zfsDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CRAFT-GBM, a feature selection framework for LightGBM, where the feature selection pool in each round is chosen based on the historical split gains.

Critical strengths

A feature selection helper for the popular boosting library LightGBM has potential real-world impact, since high-dimensional training with LightGBM is widely used and expensive in practice.

Critical weaknesses

The theoretical analysis is about the feature's average gain at the root slit, not wheater the feature is useful for all of the current tree. For example, the features that are useful at deeper level may not be useful at the root level, and can be removed by CRAFT-GBM. Further analysis/ablation on this would consolidate the finding.

Questions for authors

How does CRAFT-GBM compare with a simple baseline that trains on all features for the first 20 rounds and then only uses the fixed top-K features ranked by accumulated gain? It would be important to see how much a naive implementation of the same intuition achieves in practice compared to this more sophisticated setup. If the simple baseline performs similarly, the benefit of the proposed method would be unclear.

Reviewer_vnRLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CRAFT-GBM, an adaptive feature selection method for gradient boosted trees. It uses historical root-split gains and sequential confidence bounds to determine which features should be considered in the subsequent boosting rounds. The method provides theoretical guarantees for retaining features with the highest historical expected gains and introduces both endogenous and exogenous feature selection strategies. Experiments over 14 public datasets show improvements in training efficiency while maintaining predictive performance in general.

Critical strengths
  1. The paper introduces a statistically motivated approach to adaptive feature selection in gradient boosting. The use of confidence sequences provides a novel way to account for uncertainty in historical feature gains.
  2. The method is integrated directly into LightGBM and demonstrates improvements in training efficiency. The empirical evaluation covers multiple datasets and includes comparisons with existing feature subsampling methods.
Critical weaknesses
  1. The method relies on historical average split gains, which may not reflect the change of importance of features during boosting. Features that are initially not important can become useful after other features are fitted, this is an issue discussed by iterative sure independence screening (ISIS)[1]. It doesn’t seem CRAFT-GBM can take this phenomenon into consideration.
  2. The choice of a fixed 20-round initialization is not sufficiently justified. The initialization length may depend on the learning rate, as smaller learning rates lead to slower changes in the residuals. The authors should investigate the interactions between the learning rate and initialization rounds as they affect both feature selection and predictive performance.

[1]. Fan, Jianqing, and Jinchi Lv. "Sure independence screening for ultrahigh dimensional feature space." Journal of the Royal Statistical Society Series B: Statistical Methodology 70.5 (2008): 849-911.

Questions for authors

How sensitive is CRAFT-GBM to feature cardinality? Could differences in the number of candidate splits introduce bias into the gain-based feature selection procedure?

Reviewer_uxEPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an advanced extension for LightGBM, called Confidence-Ranked Adaptive Feature Tracking Gradient Boosting Machine (CRAFT-GBM). The main idea is to enhance feature subsampling in gradient-boosted trees. Instead of using traditional methods that preselect a number of features and ignore the split-gain evidence accumulated during training, the proposed approach adaptively determines which features to search based on training evidence using a bandits formulation. This means that the root-split gains are transformed into bounded observations and that time-uniform bounds are constructed, which are valid for non-stationary training histories. Hence, the proposed approach allows reducing approx. 40% of the training time with a 1% test loss (log loss for classification), validating the use of accumulated training evidence for adaptive feature selection while benefiting from gradient boosting’s efficiency.

Critical strengths

As the paper summary mentions, the main strength of the paper is tracking the split-gain evidence for feature selection. Hence, the proposed approach avoids computing the gain for all features and reduces the training computation time. Moreover, the paper merges bandit theory to ensure that the gains remain bounded.

Critical weaknesses

However, the main weaknesses of the paper are the validation of the proposed approach and the state-of-the-art analysis. First, the proposed approach is validated against three methods (LightGBM-Fixed, XGBoost and CatBoost) since LightGBM-Full is used as a model reference. For instance, LightGBM-Full already trains faster than XGBoost due to its leaf-wise tree growth approach, whereas CatBoost is mainly used to handle categorical features. Therefore, this validation is unfair because the proposed approach is compared against traditional methods instead of methods that tackle the same problem. Second, the state-of-the-art analysis or related work is missing. Only a brief description in the introduction is done, and some related work is missing, like

  • Zhixiang Eddie Xu, Gao Huang, Kilian Q. Weinberger, Alice X. Zheng: Gradient Boosted Feature Selection. CoRR abs/1901.04055 (2019).
Questions for authors
  • The proposed approach aims to reduce training time by selecting features based on split-gain evidence. However, how this feature selection affects the inference time. The main idea of feature selection is also to improve inference time as done in [1,2].
  • If "Feature subsampling is widely used to accelerate gradient-boosted trees, yet standard approaches choose the number of features in advance and largely ignore the split-gain evidence accumulated during training," why is it not compared against these methods?
  • Accuracy should be reported together with other metrics to compare the performance of the proposed approach since log loss does not reflect correctly the performance of the model.
  • Figure 1 and 2 should be self-explanable without missing information in the caption.

References:

[1] Shibal Ibrahim, Kayhan Behdin, Rahul Mazumder. End-to-end Feature Selection Approach for Learning Skinny Trees. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:2863-2871, 2024. [2] Xu, Z., Weinberger, K. Q., & Chapelle, O. (2012). The Greedy Miser: Learning under Test-time Budgets. In Proceedings of the 29th International Conference on Machine Learning (ICML 2012), Edinburgh, Scotland, pp. 1299-1306.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.