- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how to transfer the predictive ability of tabular foundation models (TFMs) to smaller models that can make predictions faster. It mainly explores two questions: how to generate training targets from a teacher model and whether synthetic data can improve distillation or not. The authors test two teachers, TabICLv2 and TabPFN v3, with two student models, TabM and XGBoost, on 51 TabArena datasets. They find that using the full training data as context and training only on teacher predictions works best among the tested approaches. Adding synthetic queries further improves performance, with CutMix performing best. On TabArena, the distilled models outperform tuned and ensembled versions of the same models by 56.7–97.9 Elo points. On 300 TALENT datasets, they improve performance on 236–258 datasets, reducing median prediction error by 4.02–6.37%. They also achieve 3.0–21.6× faster inference than their teacher models, at the cost of an initial distillation step.
Critical strengths
-
The main accuracy results are supported by strong empirical evidence.
On 51 TabArena datasets, the distilled students outperform tuned-and-ensembled versions of the same model architectures by 56.7–97.9 Elo points, with paired bootstrap confidence intervals above zero and wins on 33–37 datasets (Table 1). The improvement also holds after removing the data splits used to select the recipe (Table 8, Appendix C). In addition, KD [TabPFN v3 → TabM] achieves 1510.7 Elo, outperforming RealMLP with tuning and ensembling (1489.9 Elo), which is the strongest non-TFM baseline reported in Table 2. This is notable because the distilled student achieves this result without student tuning, while the baseline uses a search over 200 configurations and ensembling. These results provide strong evidence that the proposed recipe can improve student performance without requiring an expensive tuning process.
-
The paper systematically compares design choices and tests the selected recipe on an external benchmark.
The paper compares full-context and out-of-fold teacher targets across four teacher–student pairs and six values of α. Full-context targets consistently perform better in these comparisons (Fig. 2), providing useful empirical guidance for choosing a distillation strategy. The authors then apply the selected recipe unchanged to 300 TALENT datasets. The distilled models improve over default students on 236–258 datasets, with positive median gains and confidence intervals above zero across all 12 teacher–student–task combinations (Table 6). This provides additional evidence that the technique is useful beyond the primary benchmark.
Critical weaknesses
- The main results may come mostly from label information, and the benefit of the proposed method over standard distillation is unclear.
In the best-performing setup, the teacher predicts each training example while that example and its label are already included in the context (Eq. 1). Since the teacher can see the correct label, its prediction may be close to the ground-truth label. This means the targets may behave more like smoothed labels than predictions that provide additional knowledge from the teacher. Fig. 2 supports this. Full-context targets perform best at α = 1, while OOF targets perform best at α = 0.6, where some direct label supervision is added back. The paper claims that teacher predictions can replace direct label supervision, but the labels are still available to the teacher through the context. The paper does not compare its method against label-smoothed targets with similar prediction entropy. It also does not compare teacher-generated targets for CutMix examples against CutMix examples trained using mixed labels without a teacher (Sec. 3.2). These experiments would help determine whether the improvements come from the teacher's knowledge or mainly from the label information and synthetic data generation.
Without these comparisons, it is difficult to tell what the proposed method adds beyond standard knowledge distillation (Bucilua et al., 2006; Hinton et al., 2015) combined with synthetic query generation, which has also been explored by FAST-DAD (Fakoor et al., 2020). The paper shows that the recipe works, but does not clearly explain why it works or which components are responsible for the improvements. Since the paper aims to study different design choices for distillation, understanding these factors is an important part of its contribution.
- The efficiency gains may be overstated because the teacher models are not evaluated with caching.
Appendix A.3 states that teacher inference is measured "without persistent KV caching." However, Sec. 2 notes that TabPFN-2.5 (the predecessor of the TabPFN v3 teacher) and TabICLv2 can cache context representations when making repeated predictions using the same training data. This is especially relevant to Sec. 4.3, where the paper studies repeated predictions using a fixed context. Caching can reduce the cost of processing the context for every request, potentially reducing the advantage of the distilled models.
This is important because the reported speedups of 3.0–21.6× (Table 4), their increase with context size (Table 5), and the break-even estimates (Eq. 4) all depend on inference latency. The paper also reports student latencies of 29–48 ms per 32-row request, which raises questions about the overhead of the student inference pipeline, particularly for models such as XGBoost and TabM. Also, all timing experiments are conducted on an H200 GPU host, so it is unclear how these results translate to CPU-based deployment, where faster inference could be particularly useful. Since reducing inference cost is a central motivation of the paper, a comparison against teacher models with caching enabled, along with latency measurements on relevant CPU hardware, would provide a more realistic picture of the practical benefits. Without these comparisons, the reported speedups do not fully establish how much faster or more cost-effective the proposed approach would be in a real deployment.
Questions for authors
-
The best-performing configuration uses full-context teacher predictions, where the queried training observation and its label are included in the teacher's context. How much do these predictions differ from the ground-truth labels in practice? Could you report teacher–label agreement and prediction statistics, and compare the proposed method against label-smoothed targets or a teacher-free baseline using the same synthetic data generation procedure?
-
The paper reports inference speedups of 3.0–21.6× over the teacher models. How would these speedups change if the teacher inference pipeline used persistent context or KV caching where applicable? This would help establish the practical efficiency benefits of the proposed approach.
-
On TALENT, KD is compared only with default students, while the stronger tuned-and-ensembled comparison exists only on TabArena, where the recipe was chosen. Can you report KD against tuned students on TALENT, or at least on the TALENT datasets that are not in TabArena? How many datasets do the two benchmarks share?
4 comments
W2, Q2. Uncached timings may overstate efficiency; CPU deployment and student overhead need evaluation.
Distillation retains a substantial speed advantage even when the teacher uses KV caching, especially for CPU deployment. The table below reports dataset-paired median speedups of 1.69–6.46× against GPU KV cache inference and 4.91–21.49× against CPU KV cache inference across all four teacher--student pairs.
| Model | Teacher (ms), GPU uncached / GPU KV / CPU KV | Student (ms), GPU / CPU | Median paired speedup, GPU uncached / GPU KV / CPU KV | Offline (s) [IQR] |
|---|---|---|---|---|
| TabICL v2 → TabM | 232.2 / 202.7 / 383.1 | 47.8 / 76.9 | 3.0× / 1.69× / 4.91× | 186.3 [93.1, 450.9] |
| TabICL v2 → XGBoost | 232.2 / 202.7 / 383.1 | 29.1 / 29.1 | 8.7× / 3.67× / 13.42× | 48.0 [20.0, 146.5] |
| TabPFN v3 → TabM | 566.9 / 183.2 / 571.2 | 46.7 / 76.9 | 11.1× / 3.69× / 7.53× | 192.4 [110.4, 563.4] |
| TabPFN v3 → XGBoost | 566.9 / 183.2 / 571.2 | 29.7 / 29.7 | 21.6× / 6.46× / 21.49× | 59.6 [30.6, 173.9] |
The table below reports speedups and break-even requests by context size.
| Teacher inference | Below 1K (n=10): Median paired speedup | Below 1K: Break-even requests | 1K to below 10K (n=26): Median paired speedup | 1K to below 10K: Break-even requests | ≥10K (n=15): Median paired speedup | ≥10K: Break-even requests |
|---|---|---|---|---|---|---|
| TabICL v2 → TabM | ||||||
| GPU uncached | 1.9× | 3,260.5 | 2.1× | 2,073.6 | 31.4× | 468.3 |
| GPU KV cache inference | 1.73× | 87,298.3 | 1.03× | 25,017,988.6 | 1.92× | 13,874.0 |
| CPU KV cache inference | 8.64× | 330.7 | 3.07× | 488.8 | 5.87× | 1,640.6 |
| TabICL v2 → XGBoost | ||||||
| GPU uncached | 3.0× | 393.0 | 6.8× | 211.2 | 23.9× | 160.1 |
| GPU KV cache inference | 1.45× | 1,670.2 | 4.15× | 564.8 | 3.76× | 3,832.6 |
| CPU KV cache inference | 16.08× | 51.0 | 15.60× | 81.4 | 11.75× | 357.5 |
| TabPFN v3 → TabM | ||||||
| GPU uncached | 10.2× | 503.2 | 6.8× | 388.9 | 53.7× | 266.5 |
| GPU KV cache inference | 6.49× | 915.3 | 3.16× | 1,236.8 | 4.35× | 4,733.2 |
| CPU KV cache inference | 6.54× | 504.1 | 3.77× | 469.9 | 45.85× | 195.1 |
| TabPFN v3 → XGBoost | ||||||
| GPU uncached | 13.1× | 97.5 | 20.3× | 97.6 | 32.3× | 100.8 |
| GPU KV cache inference | 9.17× | 151.5 | 8.09× | 244.2 | 3.87× | 2,497.8 |
| CPU KV cache inference | 13.55× | 96.7 | 18.18× | 97.1 | 68.54× | 60.0 |
The advantage persists across almost all context-size comparisons in the table above. Across four teacher--student pairs and three context-size groups, 11 of 12 comparisons against GPU KV cache inference retain median student speedups of 1.45–9.17×. The single near-parity setting is TabICL v2 → TabM in the 1K to below 10K group (1.03×). Against CPU KV cache inference, median student speedups exceed 3× in all 12 comparisons. These consistent CPU gains provide a practical benefit for repeated prediction.
We will add these GPU and CPU KV cache experiments to the revised draft and update the corresponding efficiency tables and break-even analysis.
Q3. TALENT lacks tuned baselines; how much does it overlap with TabArena?
Interim TALENT comparison against validation-tuned single-model baselines.
We regret that, given the limited rebuttal time, the full tuned-baseline evaluation is not yet complete. We therefore share interim results on the 137 completed datasets, averaged over five paired seeds. Each supervised baseline is the validation-selected single model from 50 random configurations plus the default. Brackets show paired dataset-bootstrap 95% intervals; W/T/L counts wins, ties, and losses.
Median KD gain (%)
| Model | All (n=137) | Binary (n=47) | Multiclass (n=32) | Regression (n=58) | W/T/L (n=137) |
|---|---|---|---|---|---|
| TabICL v2 → TabM | 1.03 [0.29, 2.37] | 3.59 [1.56, 6.27] | 1.27 [-0.18, 3.89] | 0.12 [-0.49, 0.78] | 86/0/51 |
| TabICL v2 → XGBoost | 2.15 [1.13, 4.06] | 9.00 [3.24, 11.18] | 4.28 [2.15, 8.77] | 0.61 [-0.02, 1.13] | 103/0/34 |
| TabPFN v3 → TabM | 1.03 [0.26, 2.13] | 3.60 [0.16, 5.75] | 2.06 [0.30, 4.09] | 0.18 [-0.32, 0.90] | 89/0/48 |
| TabPFN v3 → XGBoost | 2.44 [1.64, 3.45] | 7.23 [2.67, 12.15] | 4.60 [1.70, 9.72] | 0.54 [-0.01, 1.80] | 101/0/36 |
Our fixed recipe KD retains an advantage over validation-tuned supervised students across all four pairs, with all aggregate 95% confidence intervals above zero.
Overlap screening flags 59 TALENT entries as potentially related to 42 of the 51 TabArena tasks, leaving 241 entries. This metadata/source-family screen does not establish row-level disjointness. The table above uses the common subset of these 241 entries completed for both students as of November 16, selected by run completion rather than observed gains.
Experiments on the remaining screened datasets and the full 300-dataset set are ongoing. We will add the tuned-baseline TALENT experiments and overlap analysis to the revised draft, updating the table above with final results.
[1] Fakoor et al., Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation, NeurIPS 2020.
Thank you for the detailed response. The label-smoothing and teacher-free CutMix controls answer my first question, and the cached-teacher timings and interim tuned TALENT results are helpful. My remaining concern is the efficiency claim. With GPU KV caching, median speedups fall to 1.69–6.46×, and for TabICL v2 → TabM the median break-even point rises to about 87,000 requests below 1K context rows and about 25 million between 1K and 10K. The abstract and Section 4.3 still report the uncached 3.0–21.6×. I will hold my rating.
Thank you for your thoughtful review and constructive feedback. We appreciate your careful evaluation of our work.
W1, Q1. Full-context targets may mainly reproduce or smooth training labels.
Full-context KD uses training labels through the teacher context; teacher-only removes the separate student hard-label loss. As shown in the table below, KD improves held-out student error over fixed and mean-entropy-matched uniform label smoothing (LS) across all four pairs:
The tested LS controls underperform hard-label training in median relative error, whereas KD reduces error by 2.28–2.56% for TabM and 2.45–2.48% for XGBoost on the same 38 classification datasets. This supports useful teacher supervision beyond the tested uniform smoothing controls.
The table below quantifies how the targets differ from hard labels:
W1, Q1. A teacher-free CutMix comparison is needed to identify the source of gains.
The teacher supplies targets at recombined inputs without imposing a parent-label mixing rule. With the same generator, teacher-supervised TabM and XGBoost outperform their teacher-free controls:
We will add the label-smoothing and teacher-free CutMix experiments, target statistics, and their analysis to the revised draft.
W1. The contribution and sources of gain beyond standard KD with synthetic queries are unclear.
Our contribution is to establish two design principles for context-dependent TFM teachers: how to construct useful supervision and how to expand query coverage.
Supervision construction (RQ1) arises because teacher targets depend on the labeled context, even for fixed query rows. We jointly vary context construction and student label mixing, showing that full-context teacher-only supervision exceeds the best tested OOF mixture by 22.8–43.0 Elo across all four pairs (Section 3.1). The target statistics and no-augmentation smoothing controls above show that this supervision differs from label copying and improves over the tested fixed and entropy-matched smoothing baselines.
Query coverage (RQ2) varies a separate axis while holding the supervision recipe fixed. FAST-DAD [1] uses learned data generation for augmented distillation; we show that simple CutMix feature recombination already makes query expansion effective for TFMs, without training a generator (Figure 3). The same-generator teacher-free control above supports a benefit from teacher supervision in the complete recipe beyond augmentation alone.