acceptodds
Under review as a conference paper at ICLR 2027

GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models

Abstract

Tabular foundation models (TFMs) achieve strong performance through in-context learning, but context-dependent inference imposes substantial latency and memory costs, hindering large-scale deployment. We propose GEAR (Generative Expansion and Real Anchoring), a modular two-stage framework that distills TFMs into lightweight MLP or tree-based predictors that can be deployed on commodity CPUs. Stage 1 uses synthetic covariates solely as teacher-query locations and trains the student on soft TFM targets, expanding coverage beyond observed rows. Stage 2 re-anchors the student to the target distribution using real labels and out-of-fold teacher predictions, whitch avoids self-labeling leakage. We further derive a risk certificate characterizing the trade-off between generated-query volume and generator fidelity. Experiments on TALENT and TabArena demonstrate the broad applicability of GEAR. Two-stage MLPs outperform supervised MLPs by 1.81–2.00 AUC points on binary tasks and 1.19–1.35 points on multiclass tasks, with additional gains over real-data-only distillation of 1.76–2.19 and 2.09–2.40 points, respectively. On binary tasks, the gains also transfer to LightGBM and XGBoost, and all three student families outperform CatBoost, the strongest non-TFM baseline, in mean AUC. Ablations show gains beyond longer training or alternative warm starts, greater stability from staged than mixed optimization, and generator-dependent diminishing returns as query volume increases. Finally, GEAR reduces median inference time by 57–2866 and peak prediction memory by 1.9–3.3, while retaining higher AUC than matched supervised baselines.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Synthetic Data, Tabular Foundation Model, Model Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6z4YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Introduces a first stage of distillation where a synthetic data generator (TabPFGen, CTGAN, TabDiff) is used to derive new covariates for which soft labels are queried from the teacher model. This is followed by a second stage with real data similar to prior works on distillation of TFMs [1]. The authors test their framework with different combinations of teacher and student models on the TALENT benchmark, showing improvements relative to their respetive supervised baselines from distilling a Tabular Foundation Model.

[1] Tanna et al. Pocket Foundation Models: Distilling TFMs into CPU-Ready Gradient-Boosted Trees. 2026

Critical strengths
  • The writing is clear
  • The paper addresses a relevant problem with real practical implications and a clear motivation. Figure 1 in particular beautifully outlines the motivation and main results of the paper.
  • The main idea of the paper is conceptually simple and sensible (although adding significant complexity as it requires training and sampling from a synthetic generator).
Critical weaknesses
  • The authors claim to evaluate on both the TALENT benchmark and TabArena. However most results present are from the former and it is unclear to me exactly how TabArena was used.
  • Critically, and related to the previous point, the authors do not clearly identify the hyperparameter tuning protocol for the supervised baselines (and their distilled versions). TabArena defines its own perscriptive protocol. TALENT seems to support hyperparameter tuning but be less perscriptive and the authors do not specify how many trials were used and whether the baselines were optimized for AUC directly (instead of accuracy). This seems like a potentially crucial flaw and in need of clarification. I would rather see less results (in terms of teacher/student variations) but with more robust hyperparameter tuning.
  • Error bars are not provided for the main result figures. Looking at the ablation figures, these seem rather large, making the conclusions not as solid as one would hope, particularly given the previous point. To me the answers to the 4 research questions are much less clear than the authors make it seem. My main takeaway, when looking at the big picture, is that distillation from TFMs does seem to cover some of the gap from supervised baselines to TFMs but these improvements are still somewhat marginal, particularly for gradient boosting.
  • The theory section seems mostly useless in its current state. The main result is relegated to an appendix and only a corollary is given without proper contextualization or adequate explanation of all the terms. I would advise the authors to fully move this analysis to the appendix since it is impossible to understand without reading the companion appendix anyways. Regardless, the analysis doesn't seem particularly interesting to me as it seems to to provide very little actionable practical guidance on algorithmic choices and is otherwise rather obvious in how errors should scale.
  • The main novelty of the paper is introducing a first stage of distillation, relying on existing synthetic data generators for obtaining new covariates not present in the real data. While the authors do ablate the introduction of this second stage, I think this analysis should be more prominent and detailed in the paper (since it is testing the main claim to novelty). Namely, this stage adds significant complexity to the distillation process and I would be interested to know if simpler "generators" like SMOTE or simply adding noise to real covariates could achieve similar gains.
  • Figures are too small.
Questions for authors

Clarification requests:

  • Where was TabArena used and in what capacity? Did you evaluate the supervised baselines and distillation with the full hyperparameter tuning?
  • How is hyperparmeter tuning otherwise handled in the TALENT benchmark? Did you tune the baselines (and distilled version hyperparameters)? Did you simply evaluate the best existing hyperparameters for the baselines, keeping them fixed for the distilled models?
  • Can you provide the statistical confidence (e.g., p-value) on the answers to the following questions?
    • Does GEAR improve upon the supervised baselines (Q1)?
    • Does the introduction of the first stage (training on synthetic data) improve upon the results of prior approaches (second stage only)?

Requests for future work:

  • How would simpler generation baselines such as SMOTE or adding noise compare to the synthetic generators used?
Reviewer_SLYDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is on distillation of Tabular Foundation Models (TFMs). The goal is to distill the impressive capabilities of teacher TFMs into smaller student models (e.g., MLPs or tree-based models) to allow faster inference and/or inference on the CPU. This is done through a two-stage approach: first train a tabular generative model (e.g., TabPFGen, CTGAN, Copula, TabDiff) to generate synthetic data from a real dataset, and use a TFM to predict corresponding soft labels. The smaller student model is trained on these outputs. Then, in the second stage, the student models are “anchored” (i.e. fine-tuned) on the actual real dataset as well. The method is benchmarked on TabArena and TALENT classification datasets.

Critical strengths

Distilling TFMs is a relevant and timely research topic.

Critical weaknesses
  1. The 2-stage distillation approach is not well-justified. Training on synthetic data generated from a tabular generative model typically does not lead to much better prediction performance. Since, there is almost always a certain degree of distribution collapse. It is also a very costly approach, as it requires training a new tabular generative model when generalizing this approach to a new dataset. On a similar note, this approach still requires two inference stages using the large TFM: once on the synthetic data (stage 1) and once on the real data (stage 2). This is dataset-specific and will thus have to be done each time when generalizing to a new dataset. In this case, the purpose of distillation becomes unclear: if we have to do this 2-stage inference with the large TFMs, why do distillation at all, instead of just using the large TFM for the original prediction problem?

    A more sensible approach would distill the student models based on the original training data of the TFMs; this is also synthetic data, but from structural causal models. This sort of distillation would only have to be performed once, and the student model can then generalize to new datasets without requiring new inference of the large TFM. It seems these two types of synthetic data got mixed up in this work.

  2. There is no benchmarking at all against other relevant methods. Other distillation methods should have been included. There are already several works which aim to distill smaller student models from large TFMs. TabPFN even provides its own distillation engine to distill it into small MLPs or tree-based models. Additionally, there are already several small TFMs which provide very fast inference and can run easily on CPU, e.g., LightPFN and Kumo-Tabular-Small. The method should also have been compared against these.

  3. Regression datasets are not considered. It is unclear why, as the proposed method should be very easily extendable.

  4. There are several phrases used which are vague and uncommon in machine learning literature, for example “Risk certificate”, “making reusable representations difficult to learn and deploy”, “Abundant synthetic queries reduce the estimation burden”

  5. There are also phrases which are (partially) incorrect:

    “By learning across large collections of synthetic tabular tasks, these models achieve strong few-shot performance.” Few-shot means only having seen a few labeled examples in-context. However, TFMs generalize from the entire training dataset, which can have many examples.

    “Each prediction invokes the large teacher and requires access to the labeled table.” Recent TFMs use a KV cache and do not require access to the labeled table at test-time. You can “fit” the TFM on the labeled training data once, then deploy the model and do inference without access to the labeled training data.

  6. There are references used throughout the paper which do not directly support the claim that is being made. Some examples:

    “Tabular data are central to machine learning in finance, marketing, healthcare, and other highstakes domains (Grinsztajn et al., 2022).” This paper is on a comparison of tree-based models and deep learning models, not about any specific applications to ML in finance, marketing, healthcare, etc.

    “Practical tabular learning ... with more complex neural architectures (Van Breugel & Van Der Schaar, 2024).” This paper is on why TFMs should be a research priority, not on the fact that tree-based models remain competitive with complex deep learning methods.

Questions for authors

None

Reviewer_AoxUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GEAR, a two-stage framework that distills TFMs into lightweight MLP or tree-based students. In Stage 1, the student imitates the frozen TFM’s soft predictions on synthetic covariates from a tabular generator, expanding query coverage beyond the real rows. In Stage 2, the student is fine-tuned on real rows with a mix of hard labels and OOF teacher predictions to avoid self-labeling leakage. The authors also provide a risk certificate linking Stage 1 performance to generated-query volume and generator fidelity. On TALENT and TabArena with three TFM teachers, GEAR improves over supervised and real-only distilled students and greatly reduces inference time and memory compared with the teachers.

Critical strengths
  1. The paper tackles a practical deployment bottleneck of TFMs by distilling them into fast, context-free students, with a simple and modular design evaluated across multiple generators, teachers, and students.
  2. The ablations are informative, and the students achieve large speedups while outperforming matched supervised baselines.
Critical weaknesses
  1. Stage 1 of GEAR is closely related to FAST-DAD (Fakoor et al., NeurIPS 2020), which also distills a strong teacher into lightweight tabular students by labeling synthetic covariates from a learned generator with the teacher’s soft predictions. FAST-DAD also provides a similar bound, where the distillation error decomposes into a sampling term that decays with the number of synthetic samples and a mismatch term between the generated and real distributions.
  2. GEAR is compared only against supervised training and real-only distillation. It should be compared with existing distillation methods that use synthetic or augmented data, such as FAST-DAD and MUNGE (Bucilua et al., 2006), as well as with the TFM distillation methods discussed in the related work, such as TabDistill and MotherNet.
Questions for authors
  1. How faithfully does the student mimic the teacher? Could the authors report student–teacher agreement, e.g., top-1 agreement or KL divergence, for Stage 1, Stage 2, and real-only distillation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.