acceptodds
Under review as a conference paper at ICLR 2027

A Simple Recipe for Distilling Tabular Foundation Models

Abstract

Tabular foundation models (TFMs) perform strong zero-shot predictions through in-context learning, but their dependence on labeled context makes repeated inference costly. Knowledge distillation offers a way to transfer this predictive strength to lightweight students. However, the context-dependent nature of TFM inference makes the distillation process non-trivial, raising two essential design questions about what constitutes the context and what constitutes the query. We study these questions across two TFMs and both neural and tree-based students, finding a simple and effective recipe for TFM distillation. On TabArena, our distilled students outperform supervised tuned-and-ensembled students by 57–98 Elo points. Moreover, the same distillation recipe improves students on 236–258 of 300 TALENT datasets and reduces median primary error by 4.0–6.4%. Finally, distilled students achieve median inference speedups of 3.0–21.6× over their teachers, making inference substantially cheaper. Code is available at https://anonymous.4open.science/r/TFM_Distillation-ED4A.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Knowledge Distillation, Tabular Foundation Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1UTVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to transfer the predictive ability of tabular foundation models (TFMs) to smaller models that can make predictions faster. It mainly explores two questions: how to generate training targets from a teacher model and whether synthetic data can improve distillation or not. The authors test two teachers, TabICLv2 and TabPFN v3, with two student models, TabM and XGBoost, on 51 TabArena datasets. They find that using the full training data as context and training only on teacher predictions works best among the tested approaches. Adding synthetic queries further improves performance, with CutMix performing best. On TabArena, the distilled models outperform tuned and ensembled versions of the same models by 56.7–97.9 Elo points. On 300 TALENT datasets, they improve performance on 236–258 datasets, reducing median prediction error by 4.02–6.37%. They also achieve 3.0–21.6× faster inference than their teacher models, at the cost of an initial distillation step.

Critical strengths
  1. The main accuracy results are supported by strong empirical evidence.

    On 51 TabArena datasets, the distilled students outperform tuned-and-ensembled versions of the same model architectures by 56.7–97.9 Elo points, with paired bootstrap confidence intervals above zero and wins on 33–37 datasets (Table 1). The improvement also holds after removing the data splits used to select the recipe (Table 8, Appendix C). In addition, KD [TabPFN v3 → TabM] achieves 1510.7 Elo, outperforming RealMLP with tuning and ensembling (1489.9 Elo), which is the strongest non-TFM baseline reported in Table 2. This is notable because the distilled student achieves this result without student tuning, while the baseline uses a search over 200 configurations and ensembling. These results provide strong evidence that the proposed recipe can improve student performance without requiring an expensive tuning process.

  2. The paper systematically compares design choices and tests the selected recipe on an external benchmark.

    The paper compares full-context and out-of-fold teacher targets across four teacher–student pairs and six values of α. Full-context targets consistently perform better in these comparisons (Fig. 2), providing useful empirical guidance for choosing a distillation strategy. The authors then apply the selected recipe unchanged to 300 TALENT datasets. The distilled models improve over default students on 236–258 datasets, with positive median gains and confidence intervals above zero across all 12 teacher–student–task combinations (Table 6). This provides additional evidence that the technique is useful beyond the primary benchmark.

Critical weaknesses
  1. The main results may come mostly from label information, and the benefit of the proposed method over standard distillation is unclear.

In the best-performing setup, the teacher predicts each training example while that example and its label are already included in the context (Eq. 1). Since the teacher can see the correct label, its prediction may be close to the ground-truth label. This means the targets may behave more like smoothed labels than predictions that provide additional knowledge from the teacher. Fig. 2 supports this. Full-context targets perform best at α = 1, while OOF targets perform best at α = 0.6, where some direct label supervision is added back. The paper claims that teacher predictions can replace direct label supervision, but the labels are still available to the teacher through the context. The paper does not compare its method against label-smoothed targets with similar prediction entropy. It also does not compare teacher-generated targets for CutMix examples against CutMix examples trained using mixed labels without a teacher (Sec. 3.2). These experiments would help determine whether the improvements come from the teacher's knowledge or mainly from the label information and synthetic data generation.

Without these comparisons, it is difficult to tell what the proposed method adds beyond standard knowledge distillation (Bucilua et al., 2006; Hinton et al., 2015) combined with synthetic query generation, which has also been explored by FAST-DAD (Fakoor et al., 2020). The paper shows that the recipe works, but does not clearly explain why it works or which components are responsible for the improvements. Since the paper aims to study different design choices for distillation, understanding these factors is an important part of its contribution.

  1. The efficiency gains may be overstated because the teacher models are not evaluated with caching.

Appendix A.3 states that teacher inference is measured "without persistent KV caching." However, Sec. 2 notes that TabPFN-2.5 (the predecessor of the TabPFN v3 teacher) and TabICLv2 can cache context representations when making repeated predictions using the same training data. This is especially relevant to Sec. 4.3, where the paper studies repeated predictions using a fixed context. Caching can reduce the cost of processing the context for every request, potentially reducing the advantage of the distilled models.

This is important because the reported speedups of 3.0–21.6× (Table 4), their increase with context size (Table 5), and the break-even estimates (Eq. 4) all depend on inference latency. The paper also reports student latencies of 29–48 ms per 32-row request, which raises questions about the overhead of the student inference pipeline, particularly for models such as XGBoost and TabM. Also, all timing experiments are conducted on an H200 GPU host, so it is unclear how these results translate to CPU-based deployment, where faster inference could be particularly useful. Since reducing inference cost is a central motivation of the paper, a comparison against teacher models with caching enabled, along with latency measurements on relevant CPU hardware, would provide a more realistic picture of the practical benefits. Without these comparisons, the reported speedups do not fully establish how much faster or more cost-effective the proposed approach would be in a real deployment.

Questions for authors
  1. The best-performing configuration uses full-context teacher predictions, where the queried training observation and its label are included in the teacher's context. How much do these predictions differ from the ground-truth labels in practice? Could you report teacher–label agreement and prediction statistics, and compare the proposed method against label-smoothed targets or a teacher-free baseline using the same synthetic data generation procedure?

  2. The paper reports inference speedups of 3.0–21.6× over the teacher models. How would these speedups change if the teacher inference pipeline used persistent context or KV caching where applicable? This would help establish the practical efficiency benefits of the proposed approach.

  3. On TALENT, KD is compared only with default students, while the stronger tuned-and-ensembled comparison exists only on TabArena, where the recipe was chosen. Can you report KD against tuned students on TALENT, or at least on the TALENT datasets that are not in TabArena? How many datasets do the two benchmarks share?

4 comments

Thank you for your thoughtful review and constructive feedback. We appreciate your careful evaluation of our work.

W1, Q1. Full-context targets may mainly reproduce or smooth training labels.

Full-context KD uses training labels through the teacher context; teacher-only removes the separate student hard-label loss. As shown in the table below, KD improves held-out student error over fixed and mean-entropy-matched uniform label smoothing (LS) across all four pairs:

Teacher → StudentMedian error reduction vs fixed LS (%) [95% CI]Median error reduction vs entropy-matched LS (%) [95% CI]
TabICL v2 → TabM3.59 [2.26, 7.55]2.31 [1.20, 3.88]
TabPFN v3 → TabM3.34 [2.15, 6.51]2.23 [1.54, 4.73]
TabICL v2 → XGBoost4.84 [3.32, 9.58]3.08 [0.85, 4.57]
TabPFN v3 → XGBoost4.95 [2.76, 8.79]2.90 [1.87, 5.34]

The tested LS controls underperform hard-label training in median relative error, whereas KD reduces error by 2.28–2.56% for TabM and 2.45–2.48% for XGBoost on the same 38 classification datasets. This supports useful teacher supervision beyond the tested uniform smoothing controls.

The table below quantifies how the targets differ from hard labels:

Full-context target statisticTabICL v2TabPFN v3
Teacher--label agreement (%)93.4193.25
Mean off-label probability mass (%)10.7810.86
Mean entropy / log(number of classes)0.27710.2713
Prediction--label NRMSE [95% CI], regression0.188 [0.105, 0.304]0.191 [0.103, 0.312]

W1, Q1. A teacher-free CutMix comparison is needed to identify the source of gains.

The teacher supplies targets at recombined inputs without imposing a parent-label mixing rule. With the same generator, teacher-supervised TabM and XGBoost outperform their teacher-free controls:

Teacher → StudentMedian error reduction vs teacher-free CutMix (%) [95% CI]Dataset wins
TabICL v2 → TabM7.76 [3.51, 10.03]47/51
TabPFN v3 → TabM6.95 [4.01, 10.46]47/51
TabICL v2 → XGBoost6.17 [3.27, 8.21]44/51
TabPFN v3 → XGBoost5.88 [2.75, 8.11]46/51

We will add the label-smoothing and teacher-free CutMix experiments, target statistics, and their analysis to the revised draft.

W1. The contribution and sources of gain beyond standard KD with synthetic queries are unclear.

Our contribution is to establish two design principles for context-dependent TFM teachers: how to construct useful supervision and how to expand query coverage.

Supervision construction (RQ1) arises because teacher targets depend on the labeled context, even for fixed query rows. We jointly vary context construction and student label mixing, showing that full-context teacher-only supervision exceeds the best tested OOF mixture by 22.8–43.0 Elo across all four pairs (Section 3.1). The target statistics and no-augmentation smoothing controls above show that this supervision differs from label copying and improves over the tested fixed and entropy-matched smoothing baselines.

Query coverage (RQ2) varies a separate axis while holding the supervision recipe fixed. FAST-DAD [1] uses learned data generation for augmented distillation; we show that simple CutMix feature recombination already makes query expansion effective for TFMs, without training a generator (Figure 3). The same-generator teacher-free control above supports a benefit from teacher supervision in the complete recipe beyond augmentation alone.

W2, Q2. Uncached timings may overstate efficiency; CPU deployment and student overhead need evaluation.

Distillation retains a substantial speed advantage even when the teacher uses KV caching, especially for CPU deployment. The table below reports dataset-paired median speedups of 1.69–6.46× against GPU KV cache inference and 4.91–21.49× against CPU KV cache inference across all four teacher--student pairs.

ModelTeacher (ms), GPU uncached / GPU KV / CPU KVStudent (ms), GPU / CPUMedian paired speedup, GPU uncached / GPU KV / CPU KVOffline (s) [IQR]
TabICL v2 → TabM232.2 / 202.7 / 383.147.8 / 76.93.0× / 1.69× / 4.91×186.3 [93.1, 450.9]
TabICL v2 → XGBoost232.2 / 202.7 / 383.129.1 / 29.18.7× / 3.67× / 13.42×48.0 [20.0, 146.5]
TabPFN v3 → TabM566.9 / 183.2 / 571.246.7 / 76.911.1× / 3.69× / 7.53×192.4 [110.4, 563.4]
TabPFN v3 → XGBoost566.9 / 183.2 / 571.229.7 / 29.721.6× / 6.46× / 21.49×59.6 [30.6, 173.9]

The table below reports speedups and break-even requests by context size.

Teacher inferenceBelow 1K (n=10): Median paired speedupBelow 1K: Break-even requests1K to below 10K (n=26): Median paired speedup1K to below 10K: Break-even requests≥10K (n=15): Median paired speedup≥10K: Break-even requests
TabICL v2 → TabM
GPU uncached1.9×3,260.52.1×2,073.631.4×468.3
GPU KV cache inference1.73×87,298.31.03×25,017,988.61.92×13,874.0
CPU KV cache inference8.64×330.73.07×488.85.87×1,640.6
TabICL v2 → XGBoost
GPU uncached3.0×393.06.8×211.223.9×160.1
GPU KV cache inference1.45×1,670.24.15×564.83.76×3,832.6
CPU KV cache inference16.08×51.015.60×81.411.75×357.5
TabPFN v3 → TabM
GPU uncached10.2×503.26.8×388.953.7×266.5
GPU KV cache inference6.49×915.33.16×1,236.84.35×4,733.2
CPU KV cache inference6.54×504.13.77×469.945.85×195.1
TabPFN v3 → XGBoost
GPU uncached13.1×97.520.3×97.632.3×100.8
GPU KV cache inference9.17×151.58.09×244.23.87×2,497.8
CPU KV cache inference13.55×96.718.18×97.168.54×60.0

The advantage persists across almost all context-size comparisons in the table above. Across four teacher--student pairs and three context-size groups, 11 of 12 comparisons against GPU KV cache inference retain median student speedups of 1.45–9.17×. The single near-parity setting is TabICL v2 → TabM in the 1K to below 10K group (1.03×). Against CPU KV cache inference, median student speedups exceed 3× in all 12 comparisons. These consistent CPU gains provide a practical benefit for repeated prediction.

We will add these GPU and CPU KV cache experiments to the revised draft and update the corresponding efficiency tables and break-even analysis.

Q3. TALENT lacks tuned baselines; how much does it overlap with TabArena?

Interim TALENT comparison against validation-tuned single-model baselines.

We regret that, given the limited rebuttal time, the full tuned-baseline evaluation is not yet complete. We therefore share interim results on the 137 completed datasets, averaged over five paired seeds. Each supervised baseline is the validation-selected single model from 50 random configurations plus the default. Brackets show paired dataset-bootstrap 95% intervals; W/T/L counts wins, ties, and losses.

Median KD gain (%) ↑\uparrow

ModelAll (n=137)Binary (n=47)Multiclass (n=32)Regression (n=58)W/T/L (n=137)
TabICL v2 → TabM1.03 [0.29, 2.37]3.59 [1.56, 6.27]1.27 [-0.18, 3.89]0.12 [-0.49, 0.78]86/0/51
TabICL v2 → XGBoost2.15 [1.13, 4.06]9.00 [3.24, 11.18]4.28 [2.15, 8.77]0.61 [-0.02, 1.13]103/0/34
TabPFN v3 → TabM1.03 [0.26, 2.13]3.60 [0.16, 5.75]2.06 [0.30, 4.09]0.18 [-0.32, 0.90]89/0/48
TabPFN v3 → XGBoost2.44 [1.64, 3.45]7.23 [2.67, 12.15]4.60 [1.70, 9.72]0.54 [-0.01, 1.80]101/0/36

Our fixed recipe KD retains an advantage over validation-tuned supervised students across all four pairs, with all aggregate 95% confidence intervals above zero.

Overlap screening flags 59 TALENT entries as potentially related to 42 of the 51 TabArena tasks, leaving 241 entries. This metadata/source-family screen does not establish row-level disjointness. The table above uses the common subset of these 241 entries completed for both students as of November 16, selected by run completion rather than observed gains.

Experiments on the remaining screened datasets and the full 300-dataset set are ongoing. We will add the tuned-baseline TALENT experiments and overlap analysis to the revised draft, updating the table above with final results.

[1] Fakoor et al., Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation, NeurIPS 2020.

Thank you for the detailed response. The label-smoothing and teacher-free CutMix controls answer my first question, and the cached-teacher timings and interim tuned TALENT results are helpful. My remaining concern is the efficiency claim. With GPU KV caching, median speedups fall to 1.69–6.46×, and for TabICL v2 → TabM the median break-even point rises to about 87,000 requests below 1K context rows and about 25 million between 1K and 10K. The abstract and Section 4.3 still report the uncached 3.0–21.6×. I will hold my rating.

Reviewer_twYoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Empirical study of how to distill tabular foundation models (TFMs) into inexpensive, dataset-specific predictors. The paper investigates whether a lightweight neural net or XGBoost student can learn to imitate context-dependent predictions from TFMs, eliminating the need to retain and process the teacher's context.

Problem

Tabular FMs are pretrained with a bunch of rows with labels available, and trained to predict missing labels, given tasks-specific tabular rows with labels. They learn to infer relationships in labels based on given context, and use that capability at inference time, but they always require a set of context rows with labels. (“in-context learners”)

The context processing (pre-fill) adds to the cost of inference. This is required every single, and hence if a model can be trained to perform inference directly without requiring context rows & labels, we can save the prefill cost.

Solution

  1. Initialize a student model, TabM (or other models like xgboost) with randomly initialized parameters.
  2. Give the pretrained TFM the labeled training dataset as context.
  3. Obtain teacher predictions for real and synthetic data points.
  4. Train TabM to imitate these teacher predictions, rather than directly training on ground-truth labels.
  5. Deploy TabM independently, without requiring the TFM or its context.

However other recent papers (particularly GEAR: https://arxiv.org/abs/2608.18849) have already proposed almost exactly the same approach and mechanics.

Process

Pretrained TabPFN / TabICL(Already learned to solve many tabular tasks) -> Specific real-world dataset (Used as the teacher's labeled context) -> Teacher predictions (On observed and synthetic feature rows) -> Train a lightweight NN or XGBoost (Dataset-specific student learned from teacher predictions)

Synthetic queries are generated by:

  1. CutMix: randomly pick individual features from 2 different rows
  2. Mixup: interpolation of numerical features & select categorical values from paired observations
  3. FullMix: sample features independently from randomly selected rows
  4. Perturb: add feature-scaled noise & randomly change categorical features

It’s evaluated across: TabArena: 51 datasets and TALENT: 300 datasets

Critical strengths
  1. Supervision construction (how to form teacher supervision) and query coverage (if & how expanding teacher queries helps with student quality) systematically studied
  2. Large set of experiments: >350 datasets total
  3. Clear metrics with confidence intervals
Critical weaknesses
  1. Mentions “pocket foundation model” and “GEAR” papers, but there’s no direct comparison of performance against those recent (& very similar) papers
  2. the novelty only is in finding a combination of teacher supervision and synthetic-query augmentation that works across use-cases, and doesn't discuss how they are different.
  3. full-context teacher predictions being better should not be a surprise. It’s a standard case of potential label leakage. Recent papers like GEAR stick to out-of-fold. This requires deeper analysis
  4. Unless I missed it, there’s no study to investigate why CutMix does consistently better than other strategies. I think that warrants some work and thought
  5. Appendix A.3 says that serving latency is measured without persistent KV caching - that is not really a fair comparison, since any reasonable serving infra would have that built in
Questions for authors
  1. Please compare and contrast with recent works in this area (see weaknesses section for details)
  2. Full-context needs to be investigated. Can you quantify the gains from leakage?
3 comments

Thank you for your thoughtful review and constructive feedback. We appreciate your careful evaluation of our work.

W1, Q1. Direct performance comparisons with Pocket and GEAR are missing.

Pocket Foundation Models [1] uses OOF + mixed supervision, a choice already covered by our context--mixing sweep. Within this common pipeline, our full-context + teacher-only design outperforms the best tested OOF + mixed-supervision setting by 22.8–43.0 Elo across all four teacher--student pairs (Section 3.1).

GEAR [2] first appeared on arXiv in August 2026; we could not locate an official implementation for a faithful head-to-head comparison. The core supervision difference concerns real rows: GEAR uses OOF targets mixed with hard labels, whereas our selected recipe uses full-context teacher-only targets.

For synthetic queries, GEAR evaluates generators including CTGAN, TabDiff, and TabPFGen alongside Copula, with synthetic pretraining followed by real-data anchoring. Simple CutMix improves over real-only KD across all four pairs at one synthetic query per real row (Figure 3). Our recipe trains jointly on real and synthetic teacher targets, without fitting a generator or adding a separate anchoring stage.

W2. The novelty appears limited to combining supervision and augmentation.

Our two RQs resolve distinct design questions: how to construct supervision for a context-dependent TFM, and where to query it.

Supervision construction (RQ1) follows from the teacher's context dependence. Even with teacher weights and query rows fixed, changing the labeled context changes the supervision. We therefore study context construction jointly with hard-label/teacher-target mixing, rather than treating teacher targets as fixed. The preferred mixture changes with context: OOF aggregate Elo peaks at α=0.6\alpha=0.6, whereas full-context Elo peaks at teacher-only supervision (α=1\alpha=1; Section 3.1). This establishes a supervision design principle specific to context-dependent teachers.

Query coverage (RQ2) is a separate axis: we fix the supervision recipe and expand the inputs queried. The TFM supplies targets for new feature combinations using the same task context, so expanding coverage requires no additional labeled examples or prescribed label interpolation. We show that simple CutMix feature recombination already makes this expansion effective for distillation, without fitting a generator (Figure 3).

W3, Q2. Could label leakage explain full-context gains, and how much?

Full-context targets use each bag member's fitting labels; its validation labels and held-out test labels are excluded (Appendix A.1). Teacher-only means that training labels enter through the teacher context, without a separate student hard-label loss.

As shown in the table below, the actual targets directly show that the teacher does not simply copy the labels: they sometimes favor a different class, retain off-label probability mass, and differ from the observed regression labels.

Full-context target statisticTabICL v2TabPFN v3
Teacher--label agreement (%)93.4193.25
Mean off-label probability mass (%)10.7810.86
Mean entropy / log(number of classes)0.27710.2713
Prediction--label NRMSE [95% CI], regression0.188 [0.105, 0.304]0.191 [0.103, 0.312]

Label smoothing (LS) provides a complementary generalization check. As shown in the table below, full-context KD outperforms fixed and mean-entropy-matched uniform LS across all four pairs:

Teacher → StudentMedian error reduction vs fixed LS (%) [95% CI]Median error reduction vs entropy-matched LS (%) [95% CI]
TabICL v2 → TabM3.59 [2.26, 7.55]2.31 [1.20, 3.88]
TabPFN v3 → TabM3.34 [2.15, 6.51]2.23 [1.54, 4.73]
TabICL v2 → XGBoost4.84 [3.32, 9.58]3.08 [0.85, 4.57]
TabPFN v3 → XGBoost4.95 [2.76, 8.79]2.90 [1.87, 5.34]

The tested LS controls underperform hard-label training in median relative error, whereas KD reduces error by 2.28–2.56% for TabM and 2.45–2.48% for XGBoost on the same 38 classification datasets. This supports useful teacher supervision beyond the tested uniform smoothing controls. We will add these label-smoothing experiments and target-statistic analyses to the revised draft.

W4. Why does CutMix consistently outperform other generators?

Our interpretation is that CutMix makes query expansion useful by balancing new feature combinations with partial preservation of observed structure. Each query takes feature values from two real rows: categorical states and numerical values remain observed, and features inherited from the same parent retain their joint configuration. Recombination adds diversity, although dependencies between features from different parents can change.

  • Perturb expands numerical coverage locally when noise is small; increasing noise or independently replacing categories can disrupt joint structure without preserving a second observed feature configuration.
  • FullMix samples features independently across rows. It preserves feature-wise values but removes their dependence, sacrificing more joint structure than CutMix's two-parent recombination.
  • Mixup interpolates numerical features, which can place queries in sparse gaps between observed modes. Even when categorical values are observed, their combinations with the interpolated numerics may have little support in the data.

W5. Uncached teacher inference makes the efficiency comparison unfair.

Distillation retains a substantial speed advantage even when the teacher uses KV caching, especially for CPU deployment. The table below reports dataset-paired median speedups of 1.69–6.46× against GPU KV cache inference and 4.91–21.49× against CPU KV cache inference across all four teacher--student pairs.

ModelTeacher (ms), GPU uncached / GPU KV / CPU KVStudent (ms), GPU / CPUMedian paired speedup, GPU uncached / GPU KV / CPU KVOffline (s) [IQR]
TabICL v2 → TabM232.2 / 202.7 / 383.147.8 / 76.93.0× / 1.69× / 4.91×186.3 [93.1, 450.9]
TabICL v2 → XGBoost232.2 / 202.7 / 383.129.1 / 29.18.7× / 3.67× / 13.42×48.0 [20.0, 146.5]
TabPFN v3 → TabM566.9 / 183.2 / 571.246.7 / 76.911.1× / 3.69× / 7.53×192.4 [110.4, 563.4]
TabPFN v3 → XGBoost566.9 / 183.2 / 571.229.7 / 29.721.6× / 6.46× / 21.49×59.6 [30.6, 173.9]

The table below reports speedups and break-even requests by context size.

Teacher inferenceBelow 1K (n=10): Median paired speedupBelow 1K: Break-even requests1K to below 10K (n=26): Median paired speedup1K to below 10K: Break-even requests≥10K (n=15): Median paired speedup≥10K: Break-even requests
TabICL v2 → TabM
GPU uncached1.9×3,260.52.1×2,073.631.4×468.3
GPU KV cache inference1.73×87,298.31.03×25,017,988.61.92×13,874.0
CPU KV cache inference8.64×330.73.07×488.85.87×1,640.6
TabICL v2 → XGBoost
GPU uncached3.0×393.06.8×211.223.9×160.1
GPU KV cache inference1.45×1,670.24.15×564.83.76×3,832.6
CPU KV cache inference16.08×51.015.60×81.411.75×357.5
TabPFN v3 → TabM
GPU uncached10.2×503.26.8×388.953.7×266.5
GPU KV cache inference6.49×915.33.16×1,236.84.35×4,733.2
CPU KV cache inference6.54×504.13.77×469.945.85×195.1
TabPFN v3 → XGBoost
GPU uncached13.1×97.520.3×97.632.3×100.8
GPU KV cache inference9.17×151.58.09×244.23.87×2,497.8
CPU KV cache inference13.55×96.718.18×97.168.54×60.0

The advantage persists across almost all context-size comparisons in the table above. Across four teacher--student pairs and three context-size groups, 11 of 12 comparisons against GPU KV cache inference retain median student speedups of 1.45–9.17×. The single near-parity setting is TabICL v2 → TabM in the 1K to below 10K group (1.03×). Against CPU KV cache inference, median student speedups exceed 3× in all 12 comparisons. These consistent CPU gains provide a practical benefit for repeated prediction.

We will add these GPU and CPU KV cache experiments to the revised draft and update the corresponding efficiency tables and break-even analysis.

[1] Tanna et al., Pocket Foundation Models: Distilling TFMs into CPU-Ready Gradient-Boosted Trees, 2026.

[2] Qin et al., GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models, 2026.

Thanks you for the detailed response. Label-smoothing and teacher-free CutMix baselines strengthen the evidence. The cached-teacher timing experiments also address my efficiency concern

Reservations around novelty and the lack of direct performance comparisons remain, but I'll revise my rating upward to weak acceptance.

Reviewer_SSTSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this work, the authors claim that the main problem of Tabular Foundation Models is that the inference cost is high. To deal with this problem, the authors explore solutions to provide a simple recipe to distill the foundation models into small student models trained for a particular context / dataset.

The authors further explore two things: how to provide the supervision labels, and whether we need to expand the query coverage. Through experimental results and empirical analysis, the authors provide a general and simple recipe to distill foundation models.

Based on this simple recipe, extensive experimental results demonstrate the effectiveness of the proposed method.


While I’m not an expert in this domain / field, my concerns come from the experimental settings and novelty (see the Critical Weaknesses).

Critical strengths
  1. The authors provide solid experimental results and detailed implementations.

  2. The experimental results show good improvements over baselines.

Critical weaknesses
  1. I feel like the comparison shown in Fig. 1 is not fair. TabICLv2 and TabPFN v3 are Tabular Foundation Models, which can help with different tabular contexts / datasets. In contrast, the distilled method in this work is context / dataset specific, losing the generalization ability to get better performance and speed. In terms of the trade-off, I don’t think the figure demonstrates a fair comparison.

  2. While the experiments are solid, I feel the novelty and contributions here are limited. First, distilling foundation models into context-specific student models has been well studied according to my search of related work, such as [1] [2]. Meanwhile, mixing teacher soft labels and GT has been a common practice for a long time.

  • [1] Pocket Foundation Models: Distilling TFMs into CPU-Ready Gradient-Boosted Trees
  • [2] MOTHERNET: FAST TRAINING AND INFERENCE VIA HYPER-NETWORK TRANSFORMERS
Questions for authors
  1. I would suggest the authors provide a simple definition of Tabular foundation models, like what model it is and what tasks it is for. Not all readers can understand what Tabular foundation models are, like me. Hence, it is a little bit hard for me to understand the task, problems, and settings in this paper. I didn’t even know whether it is a vision-language model or a language model before I searched outside the submission. The RELATED WORK section does not explain it well.

  2. The foundation model can be cached to speed up inference, with no need to process the context for each query. This is confirmed in Sec. A.3.

3 comments

Thank you for your thoughtful review and constructive feedback. We appreciate your careful evaluation of our work.

W1. The comparison with TFMs is unfair because students are dataset-specific.

Figure 1 shows a valid trade-off for our target problem: repeated inference on a fixed dataset. Both the TFM and the distilled student predict new examples for the same target task. General-purpose adaptability across datasets is outside this deployment objective, so retaining it is not a requirement for the student.

The relevant choice is whether to serve each request with the contextual teacher or incur an upfront distillation cost for a compact student. Figure 1 compares predictive quality and serving cost for these alternatives. Section 4.3 accounts for the upfront investment through offline-cost and break-even analysis.

W2. Dataset-specific distillation and label mixing have limited novelty.

Our core contribution is to establish how two routes of label supervision interact in TFM distillation. Labels enter through the teacher's context, which determines the distillation targets, and through an optional hard-label loss for the student. This calls for systematically varying both context construction and label mixing.

MotherNet [1] uses simple distill teacher as baseline, while Pocket Foundation Models [2] treats OOF labeling as necessary. Neither systematically compares OOF and full context across label-mixing choices.

In direct contrast to that claim, full-context teacher-only supervision exceeds the best tested OOF--label mixture by 22.8–43.0 Elo across all four teacher--student pairs. OOF aggregate Elo peaks at α=0.6\alpha=0.6, whereas full-context Elo peaks at α=1\alpha=1 (Section 3.1).

[1] Muller et al., MotherNet: Fast Training and Inference via Hyper-Network Transformers, ICLR 2025

[2] Tanna et al., Pocket Foundation Models: Distilling TFMs into CPU-Ready Gradient-Boosted Trees, arxiv preprint, 2026

Q1. Please define TFMs and the tasks they address.

The TFMs studied here are pretrained models for classification and regression on tables. TFMs predict new rows using labeled training rows as context, without updating their parameters. We will introduce this definition in the related work section.

Q2. Teacher caching can reduce inference cost.

Distillation retains a substantial speed advantage even when the teacher uses KV caching, especially for CPU deployment. The table below reports dataset-paired median speedups of 1.69–6.46× against GPU KV cache inference and 4.91–21.49× against CPU KV cache inference across all four teacher--student pairs.

ModelTeacher (ms), GPU uncached / GPU KV / CPU KVStudent (ms), GPU / CPUMedian paired speedup, GPU uncached / GPU KV / CPU KVOffline (s) [IQR]
TabICL v2 → TabM232.2 / 202.7 / 383.147.8 / 76.93.0× / 1.69× / 4.91×186.3 [93.1, 450.9]
TabICL v2 → XGBoost232.2 / 202.7 / 383.129.1 / 29.18.7× / 3.67× / 13.42×48.0 [20.0, 146.5]
TabPFN v3 → TabM566.9 / 183.2 / 571.246.7 / 76.911.1× / 3.69× / 7.53×192.4 [110.4, 563.4]
TabPFN v3 → XGBoost566.9 / 183.2 / 571.229.7 / 29.721.6× / 6.46× / 21.49×59.6 [30.6, 173.9]

The table below reports speedups and break-even requests by context size.

Teacher inferenceBelow 1K (n=10): Median paired speedupBelow 1K: Break-even requests1K to below 10K (n=26): Median paired speedup1K to below 10K: Break-even requests≥10K (n=15): Median paired speedup≥10K: Break-even requests
TabICL v2 → TabM
GPU uncached1.9×3,260.52.1×2,073.631.4×468.3
GPU KV cache inference1.73×87,298.31.03×25,017,988.61.92×13,874.0
CPU KV cache inference8.64×330.73.07×488.85.87×1,640.6
TabICL v2 → XGBoost
GPU uncached3.0×393.06.8×211.223.9×160.1
GPU KV cache inference1.45×1,670.24.15×564.83.76×3,832.6
CPU KV cache inference16.08×51.015.60×81.411.75×357.5
TabPFN v3 → TabM
GPU uncached10.2×503.26.8×388.953.7×266.5
GPU KV cache inference6.49×915.33.16×1,236.84.35×4,733.2
CPU KV cache inference6.54×504.13.77×469.945.85×195.1
TabPFN v3 → XGBoost
GPU uncached13.1×97.520.3×97.632.3×100.8
GPU KV cache inference9.17×151.58.09×244.23.87×2,497.8
CPU KV cache inference13.55×96.718.18×97.168.54×60.0

The advantage persists across almost all context-size comparisons in the table above. Across four teacher--student pairs and three context-size groups, 11 of 12 comparisons against GPU KV cache inference retain median student speedups of 1.45–9.17×. The single near-parity setting is TabICL v2 → TabM in the 1K to below 10K group (1.03×). Against CPU KV cache inference, median student speedups exceed 3× in all 12 comparisons. These consistent CPU gains provide a practical benefit for repeated prediction.

We will add these GPU and CPU KV cache experiments to the revised draft and update the corresponding efficiency tables and break-even analysis.

Thanks for the response and the KV-cache experiments.

My main concern is efficiency. The cached teacher is the natural baseline, but seems the headline speedups (3.0–21.6×) are against an uncached teacher. With GPU KV caching, it drops to 1.69–6.46×. I’d suggest authors report cached comparison in abstract and figure 1.

For the novelty, I realize full-context targets already contain each sample’s label, so a separate label loss being unnecessary seems somewhat expected. To me, this submission is more like a solid empirical recipe rather than a new insight.

I will keep my score, with limited confidence considering I'm not an expert in this specific domain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.