- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Introduces a first stage of distillation where a synthetic data generator (TabPFGen, CTGAN, TabDiff) is used to derive new covariates for which soft labels are queried from the teacher model. This is followed by a second stage with real data similar to prior works on distillation of TFMs [1]. The authors test their framework with different combinations of teacher and student models on the TALENT benchmark, showing improvements relative to their respetive supervised baselines from distilling a Tabular Foundation Model.
[1] Tanna et al. Pocket Foundation Models: Distilling TFMs into CPU-Ready Gradient-Boosted Trees. 2026
Critical strengths
- The writing is clear
- The paper addresses a relevant problem with real practical implications and a clear motivation. Figure 1 in particular beautifully outlines the motivation and main results of the paper.
- The main idea of the paper is conceptually simple and sensible (although adding significant complexity as it requires training and sampling from a synthetic generator).
Critical weaknesses
- The authors claim to evaluate on both the TALENT benchmark and TabArena. However most results present are from the former and it is unclear to me exactly how TabArena was used.
- Critically, and related to the previous point, the authors do not clearly identify the hyperparameter tuning protocol for the supervised baselines (and their distilled versions). TabArena defines its own perscriptive protocol. TALENT seems to support hyperparameter tuning but be less perscriptive and the authors do not specify how many trials were used and whether the baselines were optimized for AUC directly (instead of accuracy). This seems like a potentially crucial flaw and in need of clarification. I would rather see less results (in terms of teacher/student variations) but with more robust hyperparameter tuning.
- Error bars are not provided for the main result figures. Looking at the ablation figures, these seem rather large, making the conclusions not as solid as one would hope, particularly given the previous point. To me the answers to the 4 research questions are much less clear than the authors make it seem. My main takeaway, when looking at the big picture, is that distillation from TFMs does seem to cover some of the gap from supervised baselines to TFMs but these improvements are still somewhat marginal, particularly for gradient boosting.
- The theory section seems mostly useless in its current state. The main result is relegated to an appendix and only a corollary is given without proper contextualization or adequate explanation of all the terms. I would advise the authors to fully move this analysis to the appendix since it is impossible to understand without reading the companion appendix anyways. Regardless, the analysis doesn't seem particularly interesting to me as it seems to to provide very little actionable practical guidance on algorithmic choices and is otherwise rather obvious in how errors should scale.
- The main novelty of the paper is introducing a first stage of distillation, relying on existing synthetic data generators for obtaining new covariates not present in the real data. While the authors do ablate the introduction of this second stage, I think this analysis should be more prominent and detailed in the paper (since it is testing the main claim to novelty). Namely, this stage adds significant complexity to the distillation process and I would be interested to know if simpler "generators" like SMOTE or simply adding noise to real covariates could achieve similar gains.
- Figures are too small.
Questions for authors
Clarification requests:
- Where was TabArena used and in what capacity? Did you evaluate the supervised baselines and distillation with the full hyperparameter tuning?
- How is hyperparmeter tuning otherwise handled in the TALENT benchmark? Did you tune the baselines (and distilled version hyperparameters)? Did you simply evaluate the best existing hyperparameters for the baselines, keeping them fixed for the distilled models?
- Can you provide the statistical confidence (e.g., p-value) on the answers to the following questions?
- Does GEAR improve upon the supervised baselines (Q1)?
- Does the introduction of the first stage (training on synthetic data) improve upon the results of prior approaches (second stage only)?
Requests for future work:
- How would simpler generation baselines such as SMOTE or adding noise compare to the synthetic generators used?