acceptodds
Withdrawn from ICLR 2027

The Surprising Effectiveness of One-Step Boosting for Tabular Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tabular foundation models (TFMs) predict from labeled examples without updating their parameters, yet those examples can also support a small task-specific correction. We adjust class scores with a validation-selected class-prior shift, then fit ridge regression to cross-fitted probability residuals and add its output to the scores when it improves validation macro-F1. Because the residuals are negative gradients of multiclass cross-entropy, this is one step of supervised boosting. On datasets with uncorrected validation macro-F1 below ( for TabPFN/TabICL/TabFM), the combined adjustments raise mean test macro-F1 by points. The ridge step adds points beyond the shift and improves one-vs-rest score-ranking AUC by points. Even at validation macro-F1 of –, it adds macro-F1 and ranking-AUC points on average. On a low-performing TabPFN subset, the ridge gain is with the full labeled context and with one quarter of it. On one A100, four-fold cross-fitting and ridge fitting take a median s per TabPFN dataset beyond a shared s base pass; once backbone scores exist, applying both corrections including test-row featurization takes a separately measured median ms per rows. Thus even capable TFMs benefit from a small, lightweight supervised output update.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
tabular in-context learning, TabPFN, tabular foundation models, class imbalance, macro-F1, post-hoc correction, boosting

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_RRfbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of adapting tabular foundation model for any task using boosting. I would categorize this work as empirical and experimental. No novel approach has been presented - this is actually what the Tabular foundation model community is usually doing to reach better quality models and results. The approach is basically stacking boosting to the tabular foundation model - nothing new here - this is the foundation to make these models working better.

Sorry no surprising effect using this approach, which is altogether is not new, even applied to tabular foundation model.

Critical strengths
  • simple, cheap and practical but not new - and well known in the community for these models and problems. Results are trivial.
Critical weaknesses
  • limited datasets - but even a larger dataset won't help the paper - given no novelty is presented. Nice application, in-use paper. Fall short for ICLR.
  • key baselines are missing
  • scope is too narrow
  • readability: this starts from the Abstract - I have not asked but ask an LLM to assess the presentation - sure plenty of good practice will be shared. Specially on the numbers reported with no context.
Questions for authors
  • limited datasets: why?
  • key baselines are missing: why?
  • scope is too narrow: why?
Reviewer_wW2EOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a lightweight two-stage post-hoc adaptation method for frozen tabular foundation models (TFMs), TabPFN, TabICL, TabFM: (1) a validation-tuned class-prior logit shift that adjusts class decision boundaries without changing within-class row rankings, and (2) one step of cross-fitted ridge regression fitted to out-of-fold probability residuals on binned feature indicators. The authors evaluate the method across a large collection of tabular benchmarks and model families.

Critical strengths
  1. Simple and effective method: The proposed two-stage post-hoc correction is very simple to implement, requires no backbone fine-tuning, runs in seconds, and achieves large gains in macro-F1 (+6.5~8.2 points on low-performing tasks) and ranking AUC across TabPFN, TabICL, and TabFM.
  2. Comprehensive experiments on a huge set of benchmarks: The authors conduct an exhaustive empirical study across roughly 370 datasets (from PMLB, TabArena, TabZilla, and TP-BERTa) and 8 model families, with thorough ablations and diagnostic controls that clearly explain why both stages work.
Critical weaknesses
  1. Dense presentation and multiple dataset subsets: Because the 45-page paper tracks nine slightly different dataset subsets and two macro-F1 scoring conventions (Table 10), the text and appendix can be dense and somewhat hard to follow in places.
  2. Metric trade-off and split sensitivity of the class-prior shift: Tuning the class-prior shift for macro-F1 slightly lowers overall accuracy (by 1.8~2.2 points) and worsens log-loss/Brier score, and shows some sensitivity across alternate train/val/test splits on TabICL and TabFM (Table 29).
Questions for authors
  1. Since selecting τ\tau from the model's own validation posterior is more stable across splits than tuning on validation labels, should posterior selection be used as the default first stage?
Reviewer_R38NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a lightweight, post-hoc adaptation method for frozen tabular foundation models that improves their predictions on hard tabular classification tasks without changing any pretrained weights. Their method consists of a class-wide class-prior shift and residual boosting correction. The proposed method improves over three backbones.

Critical strengths
  1. The proposed method improves frozen tabular foundation models using a small supervised correction, without updating backbone parameters. This idea is interesting and timely.

  2. The empirical study is thorough, with experiments provided in diverse aspects. The provided interpretations for the results is also careful and valuable.

Critical weaknesses
  1. The idea can be seen as an engineering contribution, as its main components, class-prior adjustment, ridge regression, and boosting, are all well-established techniques. Therefore, the observations provided in this paper may not be surprising. The originality of this work appears to come from the combination itself. Therefore, the authors should clearly state what the real novelty of their work is, and how their observations are non-trivial compared to existing literature.

  2. The benefit of residual boosting is unclear, since a task-only correction and standard log-linear pooling achieve similar improvements in the reported comparisons. This makes me think whether simpler alternatives are enough without the level of complexity the proposed method has.

  3. The experiments focus on classification and mainly small datasets. Its usefulness for other downstream tasks, larger tables, and high-dimensional data is untested.

Questions for authors

Please refer to the weaknesses. Note that my expertise in tabular foundation models is limited. Therefore, this review is based primarily on my general knowledge in this area and my assessment of the methodology and experimental evidence.

Reviewer_CGakOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper observes that frozen tabular foundation models (TabPFN, TabICL, TabFM) under-select rare classes and leave some feature signal unused, and proposes to recover both from the labeled context without touching the backbone. Two corrections are applied to the model's class scores: a class-prior shift with one scalar tuned on validation macro-F1, which changes which class wins, and a one-step ridge regression fit to cross-fitted residuals on binned feature indicators, which changes within-class ranking and is kept only when it improves validation macro-F1 (Sec. 3). On each backbone's low-performing datasets (validation macro-F1 below 0.60, at most 8K rows, drawn from PMLB, TabArena, TabZilla and TP-BERTa) the two steps raise mean macro-F1 by roughly 6 to 8 points and the ridge step also improves ranking AUC (Table 1); the gain is about one point on stronger datasets and becomes small on strong ones (+0.23, Sec. 4.3). The main analytical finding is that the models give rare classes about the right total probability but spread it so the class rarely wins the argmax, and a synthetic-label experiment shows argmax is the wrong rule for macro-F1 even when the probabilities are correct (Sec. 4.2). A large set of controls covers recalibration, label shift, alternate splits, a newer checkpoint, tuned trees and fine-tuning (Appendices D to J). The study of how TFMs treat minority classes is the part I found most valuable, and it matters for practical adoption on imbalanced data. What is missing is evidence that the correction is consistent: it wins on 16 to 29% of datasets per backbone (Table 28), the dataset populations are undocumented and differ by backbone, and the ablations do not cover data types, cardinality or missingness.

Critical strengths
  1. Clean separation of two effects. A class-wide shift cannot change one-vs-rest ranking AUC (Fact 1, Appendix B, p. 15), so the AUC gains of +2.10/+3.58/+4.30 (Table 1) isolate a row-dependent effect. Test-set recentering confirms the ridge gain is not a hidden class offset (Sec. 4.1, p. 5; Table 34, p. 39).

  2. The diagnosis of rare-class under-selection is well supported. Minority classes receive about the right total probability (mass ratio 0.99) but are predicted at 0.37 to 0.53 of their observed rate, with 69 to 80% of minority mass on rows the class does not win against 48 to 53% for trees (Tables 16 and 17, p. 22). The synthetic-label null shows a correct posterior under argmax still leaves 70 to 81% of the shift gain available (Table 21, p. 26).

  3. Controls are extensive and internally consistent. The row-ordering gain survives recalibration (Table 22), DistPFN (Table 14), the 20K cap (Table 27), alternate splits (Table 29), balanced accuracy as the selection metric (Table 30) and TabPFN-3.5 (Table 26). Balanced fine-tuning does not raise the achievable macro-F1 above frozen plus shift (Appendix J, p. 41), and validation-tuned context resampling does not beat the shift either (Appendix H.2, p. 39), which is a useful result for anyone building these models.

  4. Reporting is thorough. Versions, grids, seeds and populations are documented (Appendix C; Table 10), costs are measured (Tables 4, 11, 12), and negative results are reported, including the failed probe positive control (Appendix I) and the head-to-head against trees that is Holm-significant only for TabICL (Table 15).

Critical weaknesses
  1. Dataset coverage and documentation of the evaluation. The primary populations are backbone-specific slices (76/77/74) of the 371 TabPFN datasets referenced in Appendix C (p. 16), selected on the metric the shift optimizes. No dataset is named, there is no per-dataset table, and the overlap between the three populations is not reported (52 datasets are known to be common to the three TFMs and both trees, Appendix E.3, p. 26); the Table 28 TabFM set is a third population of 86. Datasets above 20K rows are not evaluated, all-categorical tables and tables exceeding 4,000 indicator columns are filtered out (Appendix C, p. 16), the populations are 54 to 62% binary (Appendix G, p. 34), the task is classification only, and the primary numbers use one split and one inference seed. Feature composition (numeric versus categorical share, cardinality, missingness) is never analysed, although Appendix I (p. 41) shows the one-hot per-column basis carries the ridge effect. As a result, the claim that the effect holds across TFMs rests on three within-backbone means over different datasets and cannot be checked from the paper.

  2. The ridge gain is not consistent across datasets. Pooling Table 28 (p. 35): TabPFN wins on 16 of 76 datasets and loses on 6, TabICL 12 of 77 against 11, TabFM 25 of 86 against 9; the remaining 52 to 54 per backbone are ties because the validation gate (median 84 rows, Table 4 caption) switches the step off. The median gain is 0.00 in every row and the headline means (+2.41/+3.29/+5.16) average over these zeros. Sec. 5 (p. 9) reports the row correction harms 20 to 22% of datasets, which does not match the Table 28 counts (8 to 14% of all datasets, 27 to 48% of deployed ones). No gating rule better than validation selection was found (Appendix D, p. 21).

  3. The method is assembled from known operations and the paper's own ablations show the specific design does not matter. Eq. 1 is logit adjustment (Menon et al., 2021), as stated in Appendix K.1 (p. 44), and tabpfn 9.0.0 ships a tuner in the same family (Appendix F.4). Eq. 2 is stacking on out-of-fold predictions. Generic residual learners tie the ridge (Appendix H.2, p. 39) and a feature model that never sees the TFM recovers 63 to 114% of the gain (Table 32, p. 37). The title and abstract present a method result; the evidence supports an analysis result.

  4. The practical case is limited. The shift lowers accuracy by 1.8 to 2.2 points and worsens log-loss and Brier (Table 16). Against 20-trial tuned trees the corrected TFMs only edge or tie, with the tuned trees natively ahead by 2.1 to 6.1 points (Appendix D, p. 21), and the advantage over corrected default trees does not survive an oracle best-of-two tree (footnote 1, p. 20). Across all 264 eligible datasets the gains are about one point per step (Table 2, p. 7).

Questions for authors
  1. Can the authors run the full pipeline on a benchmark not used to develop the method (TabArena at full size, OpenML-CC18 or the full TabZilla suite), without the 8K cap or the featurization filters, with five random splits per dataset and the gate re-applied per split, and report (a) the fraction of datasets with a positive ridge gain in at least four of five splits, (b) win/loss/tie counts per backbone, and (c) on the datasets shared by all three TFMs, a paired per-dataset table of shift and ridge gains with the count of datasets where the ridge wins on three, two, one or none of the backbones? This would establish whether the effect is a property of datasets or of backbones, which the current evaluation cannot show.

  2. Can an appendix table list every dataset in the Table 10 populations with name, suite, rows, numeric and categorical column counts, maximum categorical cardinality, missingness rate, class count and imbalance ratio, and mark which backbones' populations it belongs to? How many datasets did each featurization filter remove, and what was their ridge gain where computable?

  3. Can the main text report deployment rate, win/loss/tie and the conditional mean gain per backbone, and the unconditional gain on the full pool? Which pipeline produced the 20 to 22% harm rate in Sec. 5, given the Table 28 counts?

  4. What is the ridge gain and win/loss split with numeric indicators only and with categorical indicators only, and binned by categorical share, cardinality and missingness? How are categorical missing values and test-time unseen levels encoded?

  5. Does selecting λ\lambda and the gate on the cross-fitted training residuals (about 500 rows) rather than the validation set (about 84 rows) improve agreement between the gate decision and the sign of the test gain? How do deployment rate, mean gain and loss rate move as the gate margin rises from 0 to 1 to 2 points? Under the identical gate, what are the win/loss/tie counts for XGBoost on the same residuals (Appendix H.2)?

  6. Given Appendix H.2 and Table 32, what does fitting the TFM's residuals add beyond adding any feature model in log space? Would the authors reframe the contribution as an analysis of frozen-TFM decision behaviour rather than a boosting method?

  7. Were any datasets above 20K rows evaluated? Does the approach extend to the regression heads of TabPFN and TabICL? Under what conditions would the authors recommend a corrected frozen TFM over a tuned tree on these datasets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.