acceptodds
Under review as a conference paper at ICLR 2027

RECAST: RESIDUAL-STATE REPRESENTATION LEARNING WITH FOUNDATION MODELS FOR LIGHTWEIGHT TABULAR PREDICTION

Abstract

Even well-trained tabular models can exhibit systematic error patterns in particular regions of the feature space. We propose RECAST, which learns residual-state representations to improve an existing model, rather than directly relearning its residuals from the original features alone. At each correction stage, RECAST partitions out-of-fold XGBoost errors into three states. An LLM uses feature semantics and aggregated error statistics to select feature subsets and propose a focus condition that guides in-context example selection, while a tabular foundation model (TFM) estimates residual-state representations for new inputs. A teacher augments the original features with these representations to predict residual corrections, and the residual states and context are updated after each stage. The teacher’s out-of-fold predictions are then blended with ground-truth labels to train a single XGBoost student using only the original features. Foundation models are used only during training; inference requires only the student and the original inputs. Across nine regression and nine binary classification datasets from OpenML, RECAST achieves the best average rank among ten compared models: 4.11 for regression, 3.00 for classification, and 3.56 overall. It improves the baseline XGBoost’s three-seed mean performance on all 18 datasets. Controlled comparisons further show better task-level mean performance than residual relearning without state representations and direct distillation of the TFM’s original-target predictions. These results support residual-state estimation as a training-time use of foundation models for improving an existing tabular predictor while retaining lightweight inference.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
tabular prediction, residual correction, representation learning, tabular foundation models, knowledge distillation, large language models, XGBoost

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bJWYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tries to improve a tuned XGBoost model by improving where it makes errors, using LLMs + tabular foundation models at training time. Each stage has 3 steps:

  1. Bin the current model's out-of-fold residuals into 3 states.
  2. An LLM sees aggregated error statistics and proposes up to 8 features plus a focus condition, which pick an in-context set for a tabular foundation model (LiMiX-2M). LiMiX outputs soft state probabilities for each row.
  3. A correction XGBoost takes [X, state probabilities] and fits the residuals.

After up to 5 stages, the teacher's OOF predictions are blended with labels to train a single XGBoost student on X alone. The authors find that the student improves on base XGB on a set of 18 OpenML datasets they try and has the best average rank among 10 models, although I have some concerns about their evaluation.

Critical strengths
  • The core idea is easy to follow. Figure 1 lays out the pipeline well, and the idea of removing the foundation models at inference time is nice.
  • The problem and method are well-motivated.
Critical weaknesses
  1. The comparison with base XGBoost is problematic. XGB uses an internal 80:20 split and never sees the external validation set, while RECAST picks the best among 25 students on it, after the external validation. That could easily account for the 2% gain on its own.
    1. Relatedly, 7 of the 27 classification seed-runs select α = 0, where the student is just XGBoost trained on y with F0's hyperparameters, so there shouldn't be an improvement on those seeds?
  2. It's hard to tell what the contribution of the LLM piece is because The no-LLM control is random feature selection, and it uses all features whenever p ≤ 8, which is half the datasets.
  3. The improvements seem relatively small against RealMLP and TabM
  4. The dataset selection looks arbitrary. The paper gives no reason for these 18 OpenML datasets, and they are not a standard suite. 11 of the 18 have fewer than 1,000 rows, most have 10 or fewer features, and the list mixes old analcatdata toy sets with an anomaly-detection time series (art_daily). The baselines are even tuned with TabArena search spaces, but the evaluation doesn't use TabArena's datasets. It'd be nice to instead see results on an established benchmark like TabArena.

Smaller note: Jayawardhana et al. 2025 ("Transformers Boost the Performance of Decision Trees on Tabular Data across Sample Sizes") already fit GBDTs on residuals of LLM/TabPFN predictions and should be discussed.

Questions for authors
  1. How do the results look when re-running base XGB with access to the same external validation set (e.g., select n_estimators on it, or refit on train+val)?
  2. Can you add a better no-LLM baseline?
2 comments
AuthorsOpenReview ↗

Response to Reviewer bJWY

We address your comments on fairness in validation-set access, the role of the LLM, and benchmark composition below. Shared experimental results are provided in the General Response (Tables G1–G5).

W1. Fairness of validation access

Response. We addressed your point that reusing hyperparameters alone does not provide the same opportunity for model selection using the external validation set. RECAST also outperforms XGBoost(+n_estimators), which selects F₀'s number of boosting rounds on the same external validation split, on 38 of 50 datasets, improving mean NRMSE by 1.33% and mean AUROC by 0.19% (General Response, Table G2).

W1.1. Runs with alpha = 0

Response. In the original submission records we checked, α = 0 was selected in 6 of the 27 classification runs; in the revised analysis, it was selected in 4 of the 24 classification runs. All four runs in the revised analysis have exactly the same AUROC as F₀ under the same seed. Improvements in the dataset-level averages come from the α > 0 results in other seeds.

W2. Contribution of the LLM

Response. Using all features when p ≤ 8 is the rule for the original random-feature control. RECAST's LLM determines both the feature subset and its size, k. Even on pc1_req, which has eight features, it selects five or six features, depending on the seed. The SHAP control with the same k constructs its context from high-residual examples. When k = p, the comparison captures the difference between the LLM's focus conditions and this example-selection strategy (General Response, Table G3(a)).

W3. Comparison with RealMLP and TabM

Response. RECAST outperforms RealMLP on 37 of 50 datasets, with a 1.41% lower mean NRMSE and a 1.31% higher mean AUROC. Its mean NRMSE is close to TabM's, at 0.7506 versus 0.7504, while its inference time in the representative measurements is approximately one-fifth of TabM's (General Response, Tables G1 and G4). This performance is achieved with a single XGBoost student that uses only the original inputs.

W4. Benchmark coverage

Response. We excluded three tasks from the original evaluation, added 15 TabArena datasets and 20 TALENT datasets, and included OCTree and DeLTa in the comparison. On TabArena's official repeat 0, folds 0–2, RECAST ranks fourth among ten models and improves over the original F₀ on all 15 datasets (NRMSE −1.41%, AUROC +0.48%).

Smaller note. Related work on Transformer-guided GBDTs

Response. Jayawardhana et al., Section 3 initialize boosting with Transformer predictions and add tree corrections to those predictions at inference. In contrast, RECAST refines a teacher using residual states defined from an existing XGBoost model's OOF errors, then distills the result into a single XGBoost student. Final inference therefore uses only the student's S(x) on the original inputs, without Transformer predictions.

Q1. Validation-matched XGBoost comparison

Response. Following your suggestion, we kept the other hyperparameters of F₀ fixed and selected n_estimators on the same external validation split. RECAST outperforms this control on 17 of 20 regression datasets and 21 of 30 classification datasets, with two-sided Wilcoxon tests yielding p = 4.8256 × 10⁻⁴ and p = 0.0473, respectively (General Response, Table G2).

Q2. Stronger no-LLM baseline

Response. We added a control on 15 datasets that combines SHAP-based selection with a matched feature count and context construction from high-residual examples. RECAST wins against this SHAP control on 8/15 datasets overall, 5/9 with n < 1000, and 3/4 with n < 1000, p > 8 (General Response, Table G3(a)). The observed win rate is highest in the last subset, suggesting an advantage for LLM contexts under small-sample conditions with more features.

Area_Chair_ejtUOpenReview ↗

Prompting for Discussion

Hi Reviewer bJWY,

Has the authors' response addressed your concerns? Can you please reply?

Regards, AC

Reviewer_y4i9OpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RECAST, a framework for improving pretrained XGBoost models using residual-state representations without increasing inference costs. The method first partitions out-of-fold prediction errors into three residual states. An LLM then selects relevant features and proposes a focus condition for context construction. A tabular foundation model estimates residual-state probabilities, which are appended to the original features for iterative XGBoost residual correction. Finally, the corrected teacher's OOF predictions are blended with ground-truth labels to train a lightweight XGBoost student. The entire teacher pipeline uses cross-fitting to prevent label leakage.

Experiments on 18 OpenML datasets (nine regression and nine classification) show that RECAST achieves the best overall average rank and improves the baseline XGBoost's three-seed mean performance on all datasets. Ablations suggest that residual-state representations outperform direct residual relearning and TFM distillation. The theoretical analysis explains how these representations may reduce approximation error under restricted function classes.

Critical strengths
  1. The authors apply nested cross-fitting throughout teacher construction, including preprocessing, residual-state estimation, and context selection. Query samples are excluded from their own state contexts.
  2. Replacing the default LLM or TFM with alternative models generally preserves improvements over baseline XGBoost, suggesting that the method is not dependent on a specific checkpoint.
Critical weaknesses
  1. The average improvement over XGBoost is approximately 2%, with several datasets showing gains below 0.3%. The Full versus X-only improvement is also small (~0.8%), making its reliability unclear.
  2. Although RECAST achieves the best average rank, its mean regression NRMSE (0.8232) is worse than RealMLP (0.8114) and TabM (0.8150), while its mean classification AUROC (0.7857) is below TabM (0.7915). This discrepancy receives limited discussion.
  3. Several seeds select α=0\alpha=0, meaning that the student is trained only on ground-truth labels. Since it reuses baseline XGBoost hyperparameters, its performance should be similar to the baseline. Any improvements in these cases require further explanation.
  4. The no-LLM ablation simultaneously removes feature selection and focus conditions. Since many datasets have fewer than ten features, feature selection may contribute little. Moreover, no-LLM performs worse than X-only on regression (0.8364 vs. 0.8297), suggesting that poorly constructed contexts can make state representations harmful. Simple heuristic context-selection baselines are missing.
Questions for authors
  1. When α=0\alpha=0 is selected, how does the student differ from baseline XGBoost in training configuration? Does it still improve performance? Could the authors separately report gains for α=0\alpha=0 and α>0\alpha>0?
  2. Since no-LLM underperforms X-only on regression, under what conditions do residual-state representations become harmful? Could validation-based gating prevent such degradation?
3 comments
AuthorsOpenReview ↗

Response to Reviewer y4i9

We address your comments on fairness in validation-set access, the role of the LLM, and benchmark composition below. Shared experimental results are provided in the General Response (Tables G1–G5).

W1. Reliability of the improvements

Response. RECAST improves over the original F₀ on all 50 datasets, with a dataset-level two-sided sign test yielding p = 1.7764 × 10⁻¹⁵ (General Response, Table G2). We also observed additional improvements of 0.78% in NRMSE and 0.76% in AUROC over the X-only control, which retains iterative correction, distillation, and the student architecture (Table G3(b)). Performance differences from prior work and public benchmarks are presented in Table G5.

W2. Average rank versus mean performance

Response. Average rank reflects relative ordering across datasets, whereas mean scores also reflect the magnitude of performance differences. In the revised evaluation, RECAST again has a better average rank than TabM, while TabM has slightly better mean NRMSE and AUROC (General Response, Table G1). We therefore do not claim superiority on every mean metric; rather, we emphasize that RECAST provides competitive predictive performance at approximately one-fifth of TabM's inference time (Table G4).

W3. Runs with alpha = 0

Response. As you point out, when α = 0 and the remaining settings are identical, the student is equivalent to F₀. All eight such runs also have identical test scores. Improvements in the dataset-level averages across seeds come from the students selected with α > 0 in other seeds. Results by selected α are provided in Table R2.1 under Q1 below.

W4. No-LLM ablation and SHAP baseline

Response. We added a control that matches the number of features, k, and combines SHAP-based feature selection with context construction from high-residual examples. RECAST wins against this SHAP control on 8/15 datasets overall and 3/4 datasets with n < 1000, p > 8. Thus, a higher win rate was observed in the subset with fewer samples and relatively more features (General Response, Table G3(a)). The lower regression performance of no-LLM relative to X-only also suggests that context suitability matters.

Q1. Results by selected alpha

Response. The student target is:

z = (1 − α)y + α t^OOF.

When α = 0, the student is equivalent to F₀ with the same training data, features, objective, hyperparameters, number of boosting rounds, and random seed. Of the 45 runs, all four regression runs and four classification runs with α = 0 match F₀. For α > 0, improvements occur in 16 of 17 regression runs and 20 of 20 classification runs (Table R2.1).

Table R2.1. Selected α: 45 runs (15 datasets, 3 seeds).

Taskα = 0 runsα > 0 runsImproved (α > 0)Mean change (α > 0)
Regression41716NRMSE −1.79%
Classification42020AUROC +2.60%

Relative mean changes apply to the α > 0 group; all eight α = 0 runs are identical to F₀.

Q2. When residual-state estimates are unhelpful

Response. Residual states may fail to provide useful correction information when the selected features and anchors do not adequately explain the error structure. RECAST already selects α on the validation set to control whether the teacher signal is used. All eight runs in which α = 0 was selected maintained the same performance as F₀ (Table R2.1).

Thank you for the detailed response and additional experiments. The new results, particularly the analysis of α=0\alpha = 0 and the matched controls, address several of my concerns regarding experimental reliability and consistency. However, I remain somewhat concerned about the necessity of the LLM component, as RECAST outperforms the SHAP-based control on only 8 out of 15 datasets. Overall, the rebuttal has improved my confidence in the empirical findings, although some concerns regarding the contribution of the LLM component remain. I am therefore inclined to raise my score slightly.

Thank you for your thoughtful reconsideration and constructive feedback. We appreciate your recognition of the additional analyses and matched controls. We also acknowledge your remaining concern regarding the incremental contribution of the LLM component. In the revised manuscript, we will more clearly distinguish the evidence supporting the overall RECAST pipeline from that specifically supporting LLM-guided context construction, retain the matched SHAP comparison prominently, and clarify the limitations of the current evidence.

Reviewer_tDJzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RECAST, a method to improve an existing tabular model. The proposed method uses an LLM to select a set of relevant features and a focus condition to guide in-context example selection. A tabular foundation model is used to estimate residual-state representations for new inputs and a teacher augments the features using these representations to predict residual corrections. A XGBoost student model is trained using the teacher's predictions and the ground truth. Once trained, only the student model is required at inference. Numerical results show marginal improved performance in both regression and classification tasks over the XGboost baseline and an ablation study show the benefit of residual learning. Overall, the contributions and the improvements over existing baselines demonstrated in the experiments seem marginal.

Critical strengths
  • The main strength lies in using residual representation to inform the teacher model. However, as the authors state, is not the first time it has been proposed in tabular learning tabular
Critical weaknesses
  • The use of LLMs for assisted tabular learning is not novel.
  • Overall the contributions and the empirical improvement over existing baselines seem marginal.
  • The proposed pipeline uses several components, such as an LLM, a teacher and a student model, which requires substantial computational effort for training.
Questions for authors
  • How much does the LLM help compared with simpler method for feature selection?
  • How does RECAST compare with XGBoost given the same budget (e.g., same validation and tuning)?

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Reviewer tDJz

We address your comments on fairness in validation-set access, the role of the LLM, and benchmark composition below. Shared experimental results are provided in the General Response (Tables G1–G5).

W1. Novelty of LLM-assisted tabular learning

Response. We agree that related work exists, including TabLLM, FeatLLM, and LLM-Select.

RECAST's core contribution is a training strategy that combines context augmentation based on the LLM's pretrained knowledge with residual-state-based knowledge distillation.

First, the LLM combines feature semantics and domain knowledge with the existing predictor's error information to propose suitable feature subsets and focus conditions for each residual state. This allows semantic and domain information to be used in residual-state estimation alongside statistical associations in the observed data.

Second, a TFM estimates residual states defined from the existing XGBoost model's OOF errors. Teacher prediction correction and distillation then transfer this information to a single XGBoost student that uses only the original inputs. The ablation results show better performance than direct distillation of the TFM's original target predictions with the same student architecture, supporting the usefulness of this residual-centered knowledge-transfer pathway.

In this evaluation, RECAST ranks second among ten models by average rank and is competitive with TabM and RealMLP, while OCTree and DeLTa rank sixth and tenth, respectively (General Response, Table G1). With the same student architecture, RECAST improves NRMSE by 0.66% and AUROC by 0.59% over direct distillation of TFM target predictions (Table G3(b)).

W2. Magnitude of empirical improvements

Response. RECAST improves over the original F₀ on all 50 datasets, reducing mean NRMSE by 1.43% and increasing mean AUROC by 1.16% (General Response, Tables G1–G2). The improvements in prior work (Table G5) and TabM's 1.47% RMSE reduction over XGBoost on the public TabArena benchmark provide context for interpreting this magnitude. RECAST achieves these improvements while retaining the inference structure of a single XGBoost model.

W3. Computational cost

Response. RECAST adds training computation relative to the original XGBoost model, but inference uses only a single student, with no LLM, TFM, or teacher. In the representative measurements from Table 17 of the submission, training time is 66.3% lower than TabM's for regression and 77.4% lower for classification, while inference time is 73.0% and 85.1% lower, respectively. Measured comparisons with RealMLP and the original XGBoost model are provided in the General Response, Table G4.

Q1. Contribution of the LLM

Response. We matched the number of features, k, for each seed, stage, and fold, selected features using SHAP, and constructed the context from examples with large residuals. RECAST wins against this SHAP control on 8/15 datasets overall, 5/9 with n < 1000, and 3/4 with n < 1000, p > 8 (General Response, Table G3(a)). A higher win rate was observed in the small-sample subset with more features. These results reflect the complete LLM context-construction strategy, including both feature selection and example selection.

Q2. Comparison under matched validation access

Response. We compared RECAST with XGBoost(+n_estimators), which keeps the other hyperparameters of F₀ fixed and selects n_estimators using the same external validation split as RECAST. With access to the validation data matched, RECAST still outperforms this control on 38 of 50 datasets, improving mean NRMSE by 1.33% and mean AUROC by 0.19% (General Response, Table G2).

AuthorsOpenReview ↗

RECAST: General Response

Thank you for your detailed reviews. We conducted additional experiments to address the points raised. RECAST improved over the original F₀ on all 50 datasets and outperformed XGBoost on 38 datasets under the matched validation setting. Inference uses only a single XGBoost student, with no LLM, TFM, or teacher.

XGBoost (F₀): The base XGBoost model tuned over 32 trials using an internal 80:20 split of the training data.

SHAP: The top k features are selected by SHAP importance, and examples with large residuals are used to construct the ICL context.

1. Competitive performance and improvements under matched validation conditions

We evaluated 50 datasets: 15 from OpenML, 15 from TabArena, and 20 from TALENT, comprising 20 regression and 30 classification tasks. Three tasks from the original evaluation (art_daily_nojump4, analcatdata_vineyard, and analcatdata_apnea1) were excluded because of time-series and data-leakage issues.

Best metric values are in bold. The RECAST row is also bold. Against RealMLP: 37/50 wins, mean NRMSE −1.41% and AUROC +1.31% (relative).

Table G1. Predictive performance on 50 datasets.

ModelRankAvg. rank ↓NRMSE ↓ (20)AUROC ↑ (30)
TabPFN-312.290.74680.8211
RECAST24.140.75060.8075
TabM34.530.75040.8099
LiMiX-2M44.830.76350.8103
CatBoost55.700.75310.8002
LightGBM66.040.76910.8026
OCTree66.040.76020.7996
RealMLP86.840.76130.7971
XGBoost96.940.76140.7983
DeLTa107.650.77120.7893

RECAST also improved over the XGBoost control using the same external validation set on 38 of 50 datasets. XGBoost(+n_estimators) keeps the other hyperparameters of F₀ fixed and additionally selects n_estimators on the validation set, as suggested in Reviewer bJWY's Q1.

W/T/L denotes dataset-level wins/ties/losses. All p-values are from two-sided Wilcoxon signed-rank tests.

Table G2. Task-wise comparisons against XGBoost baselines.

BaselineMetricBaseline → RECASTChangeW/T/LTwo-sided p
XGBoostNRMSE (20)0.7614 → 0.7506−1.43%20/0/01.9073 × 10⁻⁶
XGBoostAUROC (30)0.7983 → 0.8075+1.16%30/0/01.8626 × 10⁻⁹
XGBoost(+n_estimators)NRMSE (20)0.7607 → 0.7506−1.33%17/0/34.8256 × 10⁻⁴
XGBoost(+n_estimators)AUROC (30)0.8060 → 0.8075+0.19%21/0/90.0473

2. Contributions of LLM contexts and residual states

RECAST outperformed SHAP on 3 of 4 datasets in the small-sample subset with more features. The SHAP control matches the number of features, k, selects features using SHAP, and constructs the context from examples with large residuals.

(a) n: rows; p: features. The subsets are nested, and wins are based on final-student performance. Feature count k is matched for each seed, stage, and fold; the SHAP control uses high-residual examples. (b) Relative gains are measured with the same student architecture.

Table G3. LLM contexts and residual-state contributions.

(a) LLM wins (datasets)

Data regimeLLM vs. SHAP
All datasets8/15 (53.3%)
n < 10005/9 (55.6%)
n < 1000, p > 83/4 (75.0%)

(b) Relative improvements

Relative improvements are computed from the absolute performance values reported in Table 2 of the original submission. NRMSE reduction and AUROC increase are calculated relative to each ablation baseline.

ControlNRMSE reduction ↑AUROC increase ↑
Direct TFM target-prediction distillation0.66%0.59%
RECAST (X-only)0.78%0.76%

3. Inference-cost advantages

RECAST's inference time is 27.0% of TabM's for regression and 14.8% for classification. Training and inference times are the measurements reported in Table 17 of the submission.

Three representative datasets, three seeds. Full training/adaptation: mean ± SD; final-predictor inference: mean latency per sample.

Table G4. Training and inference times from submitted Table 17.

ModelRegression: train (s)Regression: infer. (ms)Classification: train (s)Classification: infer. (ms)
XGBoost4.69 ± 5.910.01466.81 ± 10.060.0065
RECAST154.62 ± 130.220.0144159.26 ± 160.260.0060
TabM459.06 ± 686.290.0533703.91 ± 818.180.0402
RealMLP403.12 ± 601.450.09851223.86 ± 1584.050.0609
RECAST vs. TabM (%)33.727.022.614.8
RECAST vs. RealMLP (%)38.414.613.09.9
AuthorsOpenReview ↗

RECAST: General Response (continued)

4. Magnitude of improvements compared with related work

TabM's RMSE reduction over XGBoost is 0.86–1.19% in the original paper. RECAST reduces mean NRMSE by 1.43% while retaining XGBoost's inference structure. Results from different evaluation settings are presented as context for interpreting the magnitude of the improvements.

TabM refers to TabM_mini†. Relative changes for TabM and TabDPT are calculated from the published scores. RealMLP reports an ablation-induced increase in error.

Table G5. Improvement magnitudes in related work.

ReferenceComparison/metricRelative difference
TabM (Table 2, ICLR 2025)vs. XGBoost / RMSE0.86%
TabDPT (Table 1, NeurIPS 2025)vs. TabPFN v2 / classification accuracy1.38%
RealMLP (Table B.1, NeurIPS 2024)Cosine LR vs. default (ablation); classification error1.10%
RECASTvs. XGBoost; NRMSE (20) / AUROC (30)1.43% NRMSE; 1.16% AUROC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.