- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tries to improve a tuned XGBoost model by improving where it makes errors, using LLMs + tabular foundation models at training time. Each stage has 3 steps:
- Bin the current model's out-of-fold residuals into 3 states.
- An LLM sees aggregated error statistics and proposes up to 8 features plus a focus condition, which pick an in-context set for a tabular foundation model (LiMiX-2M). LiMiX outputs soft state probabilities for each row.
- A correction XGBoost takes [X, state probabilities] and fits the residuals.
After up to 5 stages, the teacher's OOF predictions are blended with labels to train a single XGBoost student on X alone. The authors find that the student improves on base XGB on a set of 18 OpenML datasets they try and has the best average rank among 10 models, although I have some concerns about their evaluation.
Critical strengths
- The core idea is easy to follow. Figure 1 lays out the pipeline well, and the idea of removing the foundation models at inference time is nice.
- The problem and method are well-motivated.
Critical weaknesses
- The comparison with base XGBoost is problematic. XGB uses an internal 80:20 split and never sees the external validation set, while RECAST picks the best among 25 students on it, after the external validation. That could easily account for the 2% gain on its own.
- Relatedly, 7 of the 27 classification seed-runs select α = 0, where the student is just XGBoost trained on y with F0's hyperparameters, so there shouldn't be an improvement on those seeds?
- It's hard to tell what the contribution of the LLM piece is because The no-LLM control is random feature selection, and it uses all features whenever p ≤ 8, which is half the datasets.
- The improvements seem relatively small against RealMLP and TabM
- The dataset selection looks arbitrary. The paper gives no reason for these 18 OpenML datasets, and they are not a standard suite. 11 of the 18 have fewer than 1,000 rows, most have 10 or fewer features, and the list mixes old analcatdata toy sets with an anomaly-detection time series (art_daily). The baselines are even tuned with TabArena search spaces, but the evaluation doesn't use TabArena's datasets. It'd be nice to instead see results on an established benchmark like TabArena.
Smaller note: Jayawardhana et al. 2025 ("Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes") already fit GBDTs on residuals of LLM/TabPFN predictions and should be discussed.
Questions for authors
- How do the results look when re-running base XGB with access to the same external validation set (e.g., select n_estimators on it, or refit on train+val)?
- Can you add a better no-LLM baseline?
2 comments
Prompting for Discussion
Hi Reviewer bJWY,
Has the authors' response addressed your concerns? Can you please reply?
Regards, AC
Response to Reviewer bJWY
We address your comments on fairness in validation-set access, the role of the LLM, and benchmark composition below. Shared experimental results are provided in the General Response (Tables G1–G5).
W1. Fairness of validation access
Response. We addressed your point that reusing hyperparameters alone does not provide the same opportunity for model selection using the external validation set. RECAST also outperforms XGBoost(+
n_estimators), which selects F₀'s number of boosting rounds on the same external validation split, on 38 of 50 datasets, improving mean NRMSE by 1.33% and mean AUROC by 0.19% (General Response, Table G2).W1.1. Runs with alpha = 0
Response. In the original submission records we checked, α = 0 was selected in 6 of the 27 classification runs; in the revised analysis, it was selected in 4 of the 24 classification runs. All four runs in the revised analysis have exactly the same AUROC as F₀ under the same seed. Improvements in the dataset-level averages come from the α > 0 results in other seeds.
W2. Contribution of the LLM
Response. Using all features when p ≤ 8 is the rule for the original random-feature control. RECAST's LLM determines both the feature subset and its size, k. Even on
pc1_req, which has eight features, it selects five or six features, depending on the seed. The SHAP control with the same k constructs its context from high-residual examples. When k = p, the comparison captures the difference between the LLM's focus conditions and this example-selection strategy (General Response, Table G3(a)).W3. Comparison with RealMLP and TabM
Response. RECAST outperforms RealMLP on 37 of 50 datasets, with a 1.41% lower mean NRMSE and a 1.31% higher mean AUROC. Its mean NRMSE is close to TabM's, at 0.7506 versus 0.7504, while its inference time in the representative measurements is approximately one-fifth of TabM's (General Response, Tables G1 and G4). This performance is achieved with a single XGBoost student that uses only the original inputs.
W4. Benchmark coverage
Response. We excluded three tasks from the original evaluation, added 15 TabArena datasets and 20 TALENT datasets, and included OCTree and DeLTa in the comparison. On TabArena's official repeat 0, folds 0–2, RECAST ranks fourth among ten models and improves over the original F₀ on all 15 datasets (NRMSE −1.41%, AUROC +0.48%).
Smaller note. Related work on Transformer-guided GBDTs
Response. Jayawardhana et al., Section 3 initialize boosting with Transformer predictions and add tree corrections to those predictions at inference. In contrast, RECAST refines a teacher using residual states defined from an existing XGBoost model's OOF errors, then distills the result into a single XGBoost student. Final inference therefore uses only the student's S(x) on the original inputs, without Transformer predictions.
Q1. Validation-matched XGBoost comparison
Response. Following your suggestion, we kept the other hyperparameters of F₀ fixed and selected
n_estimatorson the same external validation split. RECAST outperforms this control on 17 of 20 regression datasets and 21 of 30 classification datasets, with two-sided Wilcoxon tests yielding p = 4.8256 × 10⁻⁴ and p = 0.0473, respectively (General Response, Table G2).Q2. Stronger no-LLM baseline
Response. We added a control on 15 datasets that combines SHAP-based selection with a matched feature count and context construction from high-residual examples. RECAST wins against this SHAP control on 8/15 datasets overall, 5/9 with
n < 1000, and 3/4 withn < 1000, p > 8(General Response, Table G3(a)). The observed win rate is highest in the last subset, suggesting an advantage for LLM contexts under small-sample conditions with more features.