acceptodds
Under review as a conference paper at ICLR 2027

TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity

10 views

Abstract

Recent progress in tabular foundation models suggests that training on synthetic tasks can substantially improve in-context learning capabilities, with overall performance largely depending on how well models can infer task-specific predictive relationships from the available context during inference. In this paper, we introduce TAFFY, a tabular foundation model with an In-Context Diversity Prior and a Task-Conditioned Looped Transformer that strengthen this ability. Specifically, to construct each synthetic pretraining context, the In-Context Diversity Prior samples from multiple related environments derived via controlled interventions and distribution shifts on a shared causal process. This in-context diversity encourages the model to learn a more comprehensive and task-specific representation. Moreover, the Task-Conditioned Looped Transformer iteratively and selectively applies a shared group of Transformer blocks to refine contextual representations, with a task-conditioned gate modulating the final hidden-state update. This enables task-adaptive iterative refinement. Together, these components encourage the model to identify predictive relationships from contextual contrasts during pretraining and dynamically modulate context integration for each task. Across six classification and five regression benchmark datasets, TAFFY attains the lowest average rank.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular Foundation Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UygpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TAFFT, a tabular foundation model for in-context classification and regression. TAFFY builds on the tabular in-context learning paradigm with two core modifications. The first is its diversity prior, which comes from a synthetic causal model using distribution shifts and interventions to pretrain the model. The second is the task conditioned loop transformer which uses statistics from the labeled support data to produce a scalar that attenuates the final update. They evaluated the model across a broad suite of benchmarks and consistently outperformed common baselines on every benchmark.

Critical strengths
  1. Broad and strong empirical performance. They evaluate TAFFY on several benchmarks, and it outperforms strong competitors on them. This suggests something very real here independent of other critiques.

  2. The paper has a couple strong mechanistic ablations. The comparison of in context vs cross context diversity, task specific vs global gating, varying loop depth, and the frozen checkpoint intervention all add color to why TAFFY works to the extent that it does. Although they have some ablations I would like that are missing, it isn't reasonable to ask for every possible variation, and I appreciate the work put into these experiments.

Critical weaknesses
  1. The main results, which are the biggest strength of this paper, don't have uncertainty quantification of effect sizes. The headline results are reported primarily through average ranks across datasets, but I do not find confidence intervals, standard errors, repeated-training variability, or paired significance analyses for these main comparisons. For a primarily empirical paper, this feels like a glaring omission. Average rank also doesn't indicate how much stronger TAFFY is than its competitors. When it wins, what is the margin by which it does so, and when it loses, what is that margin? It is very difficult to understand the robustness of TAFFY without those kinds of metrics.

  2. The current ablations don't fully isolate why TAFFY is outperforming competitors. The in-context diversity experiment shows that diversity within a task works better than diversity between tasks, but it seems as though the relevant comparison would be whether causally related data is more important than just heterogeneous data. Similarly, the main ablation adds the looped Transformer and task-conditioned gate together, which the authors acknowledge doesn't isolate those two effects, and the looped models also use more computation. As a result, the experiments show that the complete TAFFY works well, but do not clearly establish whether the improvement is specifically due to the proposed causal structure and task adaptation, rather than more heterogeneous synthetic tasks and additional computation.

Questions for authors
  1. Any form of uncertainty quantification + effect size quantification would greatly improve this paper. This is particularly important because the paper does provide bootstrap confidence intervals for a secondary gate-analysis experiment, but not for the central results, which suggests it is plausible to do such analyses.

  2. Can you provide evidence that the specific causal intervention mechanism is important for the performance of TAFFY as opposed to just heterogeneous data?

  3. How much of the architectural gain remains under a compute-matched comparison? Taffy-4L requires substantially more training time per step than the single-pass backbone. A comparison against a non-looped model given a similar training/inference compute budget would materially affect whether I view the improvement as evidence for the recurrent/task-conditioned architecture specifically, rather than for additional computation more generally.

A quality answer to (1) alone will raise my score, as I think that's the most important omission given that this paper is mostly empirical.

9 comments
AuthorsOpenReview ↗

Response to Reviewer Uygp (Part 1/4)

We thank the reviewer for a precise and constructive review.

W1. The main results, which are the biggest strength of this paper, don't have uncertainty quantification of effect sizes. […] When it wins, what is the margin by which it does so, and when it loses, what is that margin? It is very difficult to understand the robustness of TAFFY without those kinds of metrics.

Thanks for identifying the most important omission of the submission. We have reported effect sizes with CIs, win/tie/loss counts with loss tails, and paired tests.

Table R1 — per suite (Appendix E.2). Mean accuracy (%) and average rank among 21 methods, with dataset-bootstrap 95% CIs. "Best other" is the baseline with the lowest average rank.

SuitenTAFFY accTAFFY rankBest otherits accits rank
BCCO10278.75 [75.50, 81.78]4.48 [3.73, 5.26]TabICLv278.48 [75.19, 81.57]5.48 [4.71, 6.32]
OpenML6290.04 [86.65, 92.91]4.77 [3.71, 5.91]TabICLv289.49 [86.15, 92.29]6.05 [4.94, 7.20]
PFN2987.52 [83.31, 91.37]5.12 [3.21, 7.21]TabICLv287.05 [82.94, 90.92]5.41 [4.15, 6.83]
TALENT17385.71 [83.75, 87.61]3.67 [3.19, 4.18]TabPFN385.38 [83.44, 87.27]4.46 [4.01, 4.94]
TabArena3288.12 [85.12, 90.76]4.17 [2.94, 5.59]TabPFN388.04 [85.14, 90.66]4.89 [3.53, 6.44]
TabZilla2789.95 [86.64, 92.72]5.22 [3.57, 7.15]TabICLv289.10 [85.93, 91.93]6.13 [4.57, 7.98]

Table R2 — paired, 281 unique classification datasets (Appendix E.2; 272 for AutoGluon and CatBoost, which lack results on nine). Δ = TAFFY minus competitor accuracy (pp) with 95% bootstrap CI. Loss columns describe the datasets where TAFFY loses. p = Wilcoxon signed-rank, Holm-corrected over the nine comparisons.

vs.W / T / Lmean Δ (pp)median winmedian loss90th pct lossmax lossp (Holm)
TabPFN3165 / 18 / 98+0.44 [+0.25, +0.65]0.450.221.696.67<1e-4
TabICLv2150 / 59 / 72+0.28 [+0.05, +0.49]0.510.451.5119.61<1e-4
LimiX-16M186 / 15 / 80+0.94 [+0.37, +1.62]0.800.493.0519.61<1e-4
TabPFN2198 / 16 / 67+0.97 [+0.67, +1.28]0.820.352.018.82<1e-4
LimiX-2M195 / 14 / 72+1.57 [+0.99, +2.23]0.840.442.2119.61<1e-4
TabICLv1186 / 48 / 47+1.67 [+1.18, +2.29]1.140.713.216.98<1e-4
Mitra218 / 10 / 53+2.91 [+2.30, +3.56]1.700.432.184.17<1e-4
AutoGluon220 / 5 / 47+2.10 [+1.66, +2.55]1.650.893.866.60<1e-4
CatBoost224 / 6 / 42+2.45 [+2.01, +2.97]1.600.383.249.41<1e-4
  • Pooled over datasets, every paired gain is significant. The gains over the two strongest models are small (+0.44 and +0.28 pp), and 90% of losses are under 1.7 pp. Large single losses occur: 19.6 pp against TabICLv2 on one small BCCO dataset (schizo).
  • The rank gap to the best baseline excludes zero on BCCO [0.17, 1.80], OpenML [0.44, 2.09] and TALENT [0.28, 1.29]. On PFN (29), TabArena (32) and TabZilla (27) it is positive, but these suites alone cannot resolve it.
AuthorsOpenReview ↗

Response to Reviewer Uygp (Part 2/4)

W1, continued: regression, sensitivity and pretraining-seed variance.

Table R3 — regression (Appendix E.2). The submitted regression model (step 22,175, the newest checkpoint available when the regression evaluation was run). Average rank with 95% CI; Δ = mean relative RMSE reduction (%) with 95% CI; p = Wilcoxon, Holm-corrected within suite.

SuitenTAFFY rankvs TabPFN3: W/L, Δvs TabICLv2: W/L, Δ
BCCO462.07 [1.52, 2.72]35/11, +3.2% [+0.8, +6.5], p <1e-439/7, +4.4% [+2.4, +7.0], p <1e-4
CTR23332.94 [1.85, 4.27]27/6, +11.2% [+4.6, +19.4], p <1e-425/8, +4.7% [-2.7, +12.2], p 0.003
PFN283.29 [2.25, 4.50]19/9, +8.4% [+1.4, +17.0], p 0.05618/10, +1.3% [-6.6, +9.4], p 0.19
TALENT892.44 [1.94, 3.04]67/22, +5.8% [+2.1, +9.6], p <1e-476/13, +2.4% [-3.0, +6.5], p <1e-4
TabArena131.31 [1.08, 1.62]11/2, +2.1% [+1.2, +3.1], p 0.00412/1, +3.8% [+1.8, +6.3], p 0.004
  • TAFFY has the lowest average-rank point estimate on all five suites and wins more datasets than it loses on each. The paired test is significant on 4/5 suites against each baseline; PFN (28 datasets) is not resolved. Against TabICLv2 on CTR23 and TALENT, the test is significant while the CI of the mean includes zero: TAFFY wins most datasets, and a few large losses widen the CI of the mean.

Table R4 — within-run sensitivity (Appendix E.4; each configuration inserted alone into the 20-baseline pool; rank among 21).

Configurationrank [95% CI]mean acc (%)
step 25,000 (final, reported)4.81 [4.11, 5.55]84.25
step 24,5004.77 [4.10, 5.50]84.25
step 24,0004.88 [4.19, 5.59]84.32
ensemble seed 1 (reported: seed 42)4.60 [3.95, 5.32]84.28
ensemble seed 24.91 [4.21, 5.65]84.25
row/column order 14.80 [4.17, 5.50]84.19
row/column order 24.74 [4.00, 5.47]84.28
  • TAFFY is first in all seven. Its rank has sd 0.10, against a gap of at least 1.01 ranks to TabPFN3.

Repeated-training variability. In foundation-model research, pretraining runs are rarely repeated because each one is expensive. One TAFFY-4L run takes 5,611 GPU-hours (3.65 days on 64 AMD MI210 GPUs; Table 5 of the paper), about USD 11,000 at an assumed cloud price of USD 2 per GPU-hour. Repeating our six-model ablation with two more seeds would take about 49,000 GPU-hours, roughly USD 100,000. TabPFN [1], TabICL [2], TabICLv2 [3] and LimiX [4], for example, each report a single pretraining run; the TabICL authors report 20 days of pretraining on three A100 GPUs for their own model [2]. We nevertheless quantified the variability we could:

  • Within one run, over checkpoints, ensemble seeds and row/column orders (Table R4), TAFFY's rank has sd 0.10 and stays first.
  • Across runs, we bounded it. If a different pretraining seed shifted TAFFY's average rank by a random amount with standard deviation s, the lead on the three suites where it is significant would lose significance only at s = 0.29 (BCCO), 0.50 (OpenML) and 0.31 (TALENT), that is 2.9–5.1 times the within-run sd.

References

  • [1] N. Hollmann, S. Müller, K. Eggensperger, F. Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. ICLR 2023.
  • [2] J. Qu, D. Holzmüller, G. Varoquaux, M. Le Morvan. TabICL: A tabular foundation model for in-context learning on large data. ICML 2025.
  • [3] J. Qu, D. Holzmüller, G. Varoquaux, M. Le Morvan. TabICLv2: A better, faster, scalable, and open tabular foundation model. ICML 2026.
  • [4] X. Zhang et al. LimiX: Unleashing structured-data modeling capability for generalist intelligence. arXiv:2509.03505, 2025.
AuthorsOpenReview ↗

Response to Reviewer Uygp (Part 3/4)

W2. The current ablations don't fully isolate why TAFFY is outperforming competitors. […] it seems as though the relevant comparison would be whether causally related data is more important than just heterogeneous data. Similarly, the main ablation adds the looped Transformer and task-conditioned gate together, which the authors acknowledge doesn't isolate those two effects, and the looped models also use more computation. […]

We have separated the loop, the gate and the extra computation, and stated what relates the environments of one context (Appendix E.1, E.5, E.6 and E.8).

  • The loop, with the same task gate. Rank 5.48 (2L), 5.30 (3L), 4.81 (4L); 4L vs. 2L p = 0.010, 4L vs. 3L +0.23 pp [+0.01, +0.45], p = 0.018. The rank improves with every added trained loop.
  • The gate, with the loops fixed. Within the trained four-loop model, the task dependence is used: removing it at inference (w = 0; α = tanh(a) = 0.34 for every dataset) costs 0.15 pp [0.07, 0.24] (45 / 70 / 19 wins / ties / losses for the unmodified model, Holm p < 0.001), and α = 1 costs 0.56 pp [0.30, 0.85] (p < 1e-4). Trained separately at two loops, task and global gates are close (+0.15 pp [−0.26, +0.72], p = 0.53).
  • The extra computation. Computation alone does not produce the gain: four untrained passes of a single-pass model's stack lower its accuracy by 0.88 pp, and TAFFY-4L is +1.16 pp [+0.70, +1.67] above them (p < 1e-6); 4× more ensemble members leave the single-pass model's rank unchanged (5.63 → 5.65).
  • Related versus heterogeneous data. The environments of one context are related by construction: each is the same generated table under a different small perturbation, and we have described them in the revision as related environments of one generating process. The runs with and without these contexts use the same generator and the same 25,000-step schedule; only what a context contains differs. Trained with these contexts, TAFFY-4L is +0.40 pp [+0.18, +0.68] above our single-pass model trained without them (p = 0.003). A control pretrained on mixtures of unrelated generating processes will be added to the final version.

Q1. Any form of uncertainty quantification + effect size quantification would greatly improve this paper. This is particularly important because the paper does provide bootstrap confidence intervals for a secondary gate-analysis experiment, but not for the central results, which suggests it is plausible to do such analyses.

We agree, and we have applied the same bootstrap to every central result (Appendix E.2–E.4):

  • Uncertainty: dataset-bootstrap 95% CIs (B = 2,000) for mean accuracy and average rank on all six classification suites, for average rank on all five regression suites, and for the rank gap to the best baseline.
  • Effect sizes: paired accuracy differences with 95% CIs against nine strong baselines, with win / tie / loss counts, median win and loss, and the 90th-percentile and largest loss.
  • Significance: Wilcoxon signed-rank tests, Holm-corrected; all nine classification comparisons give p < 1e-4.
  • Robustness: within-run sensitivity over checkpoints, ensemble seeds and row/column orders (rank sd 0.10), and the pretraining-seed variation the lead could absorb (2.9–5.1 times that sd).

In numbers: +0.44 pp [+0.25, +0.65] over TabPFN3 and +0.28 pp [+0.05, +0.49] over TabICLv2, and the rank gap to the best baseline excludes zero on BCCO, OpenML and TALENT.

Reviewer_p63SOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors present a tabular foundation model that introduces a diversity-promoting training pipeline and a looped Transformer component. The authors demonstrate competitive performance across a variety of regression and classification benchmarks.

Critical strengths

The authors study an interesting and timely topic. Their model also appears to perform competitively on an array of tabular benchmarks.

Critical weaknesses

I must first preface this by stating that I have no prior familiarity with tabular foundation models, so unusual choices made in this paper may be established conventions of this community that are simply unfamiliar to me. Please defer to the opinion of more experienced reviewers, where relevant.

A significant weakness appear to be the lack of accuracy metrics in this paper. Performance is compared purely based on average ranks against other models, with no statement about the actual performance attained by each model. The only gesture to accuracy I could find is Figure 1d, but the accuracy axis is unlabeled (!), casting substantial doubt on the true performance of these methods. While ranks offer an interesting window into how this model stacks against other models, it is minimally informative if, as a user, I would like to apply this method and gain a rough understanding of how it will perform. Even a brief table summarizing accuracy metrics in the Appendix would go a long ways towards restoring confidence in this approach. As is, an average rank of e.g. "4.67" reported on OpenML is a nebulous quantity, saying little about how the model performed in absolute terms.

The absence of accuracy metrics also makes it very difficult to judge whether the absolute gain attained by this method is authentic, or within noise. The authors don't appear to estimate confidence intervals to their measurements, and the lack of absolute accuracy makes it difficult to estimate whether a method is outperforming its competitors only slightly (and therefore subject to noise variations) or handedly defeating the opposition.

The other (perhaps more significant) weakness is in the quality of the authors' proposed methods. Their ablations (Figures 3 and 4) seem to indicate only small, fractional changes in rank when using their diversity prior and looped Transformer components, with respect to the baseline. Indeed, the baseline model already appears to attain very strong ranks, apparently outperforming the other tabular foundation models tested. Hence, the strength of their results appear to stem not from the authors' interventions, but simply because the base model is already very strong. I couldn't find details on the size or configuration of the base model. It appears the base model was also finetuned directly on the benchmarks under examination, and it seems that the other foundation models may not have been finetuned on these same sets of benchmarks, so the strong performance of the baseline could very well be from task-specific finetuning.

The proposed methods also seem to be limited in novelty and incremental in design. The importance of data diversity for in-context learning has been exhaustively studied and documented in prior work, and the authors do not appear to claim new findings about data diversity or looped Transformers. Rather, their intervention seems to be an incremental combination of these two ideas, and little new science seems to be reported.

Questions for authors
  1. Do you include accuracy metrics somewhere in your manuscript, or is it possible to include them during the rebuttal?
  2. How large are your base models? Is the comparison to other neural network based models controlled for size, compute, and training? Did the other models also get finetuned on your choice of benchmarks?
8 comments
AuthorsOpenReview ↗

Response to Reviewer p63S (Part 1/5)

We thank the reviewer for the review and for the concrete requests.

A factual correction first. The review states that our base model "was also finetuned directly on the benchmarks under examination". This is not the case. TAFFY, our single-pass baseline and the other foundation models in Table 1 are never fine-tuned on any benchmark. Every number is a frozen-parameter in-context prediction: the model reads a dataset's training split as context and predicts its test split in one forward pass. Appendix C.3 states this: "In-context foundation models predict query labels from the labeled support dataset without updating pretrained parameters." No model in the comparison has the advantage the review attributes to ours, and the strength of our baseline does not come from task-specific fine-tuning. As the soundness score appears to rest on this point, we ask the reviewer to reconsider it.

We now answer each point in order.

W1. A significant weakness appear to be the lack of accuracy metrics in this paper. […] Even a brief table summarizing accuracy metrics in the Appendix would go a long ways towards restoring confidence in this approach.

Average rank is the standard summary in this field. Tabular foundation models are compared across many datasets with relative metrics: TabPFN [1], TabICL [2], TabICLv2 [3], LimiX [4] and the TALENT benchmark study [5] all report average rank, and TabArena [6] uses the closely related Elo rating. Absolute accuracy differs widely between datasets, so its mean is dominated by a few hard datasets, whereas a rank counts every dataset equally. We followed this convention and hope the reviewer will understand our choice in this light.

We agree that absolute numbers also help, and we have added them to the revision.

Absolute accuracy with CIs (Appendix E.2; final checkpoint; dataset-bootstrap 95% CIs; "best other" = baseline with the lowest average rank):

SuitenTAFFY acc (%)best other: acc (%)TAFFY rankbest other rank
BCCO10278.75 [75.50, 81.78]TabICLv2: 78.48 [75.19, 81.57]4.48 [3.73, 5.26]5.48 [4.71, 6.32]
OpenML6290.04 [86.65, 92.91]TabICLv2: 89.49 [86.15, 92.29]4.77 [3.71, 5.91]6.05 [4.94, 7.20]
PFN2987.52 [83.31, 91.37]TabICLv2: 87.05 [82.94, 90.92]5.12 [3.21, 7.21]5.41 [4.15, 6.83]
TALENT17385.71 [83.75, 87.61]TabPFN3: 85.38 [83.44, 87.27]3.67 [3.19, 4.18]4.46 [4.01, 4.94]
TabArena3288.12 [85.12, 90.76]TabPFN3: 88.04 [85.14, 90.66]4.17 [2.94, 5.59]4.89 [3.53, 6.44]
TabZilla2789.95 [86.64, 92.72]TabICLv2: 89.10 [85.93, 91.93]5.22 [3.57, 7.15]6.13 [4.57, 7.98]

Fig. 1(d). You are right that it gave no accuracy values; we have added them to the revised figure. The axis spans 83.9–84.7%. TAFFY-4L's curve lies between 84.52% and 84.68%, and the curve of our own TabICLv2-architecture run peaks at 84.54% (its best single measurement, the dashed line, is 84.59%). The panel plots accuracy on the TALENT classification datasets evaluated during training, against the pretraining data budget (Appendix C.9).

References

  • [1] N. Hollmann, S. Müller, K. Eggensperger, F. Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. ICLR 2023.
  • [2] J. Qu, D. Holzmüller, G. Varoquaux, M. Le Morvan. TabICL: A tabular foundation model for in-context learning on large data. ICML 2025.
  • [3] J. Qu, D. Holzmüller, G. Varoquaux, M. Le Morvan. TabICLv2: A better, faster, scalable, and open tabular foundation model. ICML 2026.
  • [4] X. Zhang et al. LimiX: Unleashing structured-data modeling capability for generalist intelligence. arXiv:2509.03505, 2025.
  • [5] H.-J. Ye et al. A closer look at deep learning methods on tabular datasets. arXiv:2407.00956, 2024.
  • [6] N. Erickson et al. TabArena: A living benchmark for machine learning on tabular data. NeurIPS 2025.
AuthorsOpenReview ↗

Response to Reviewer p63S (Part 2/5)

W2. The absence of accuracy metrics also makes it very difficult to judge whether the absolute gain attained by this method is authentic, or within noise. The authors don't appear to estimate confidence intervals to their measurements […]

Thanks for this point; whether the gain is within noise is a paired question, and we answer it with paired statistics.

  1. Pooled over 281 datasets the gain is outside noise. TAFFY beats TabPFN3 on 165 datasets, ties on 18, loses on 98 (+0.44 pp [+0.25, +0.65]); against TabICLv2: 150 / 59 / 72 (+0.28 pp [+0.05, +0.49]). Holm-corrected Wilcoxon p < 1e-4 for these and the other seven baselines. Against these two strongest baselines, 90% of losses are under 1.7 pp.
  2. Against the other seven baselines the margin is larger: +0.94 pp (LimiX-16M), +0.97 (TabPFN2), +1.57 (LimiX-2M), +1.67 (TabICLv1), +2.10 (AutoGluon), +2.45 (CatBoost) and +2.91 (Mitra), each with a 95% CI above zero.
  3. Per suite, the 95% CI of the rank gap excludes zero on BCCO [0.17, 1.80], OpenML [0.44, 2.09] and TALENT [0.28, 1.29]. On the three smallest suites (27–32 datasets) the gap is positive, but these suites alone are too small to resolve it.
  4. Within-run variation is much smaller than the gap: over three neighbouring checkpoints, three ensemble seeds and two row/column orders, TAFFY's rank has sd 0.10, and it stays first in all seven configurations.

Average rank remains our headline summary; we have added absolute accuracy per suite to the revision, and we will release the per-dataset accuracies and RMSEs of all methods.

AuthorsOpenReview ↗

Response to Reviewer p63S (Part 3/5)

W3. […] Their ablations (Figures 3 and 4) seem to indicate only small, fractional changes in rank when using their diversity prior and looped Transformer components, with respect to the baseline. Indeed, the baseline model already appears to attain very strong ranks […] I couldn't find details on the size or configuration of the base model. It appears the base model was also finetuned directly on the benchmarks under examination, and it seems that the other foundation models may not have been finetuned on these same sets of benchmarks, so the strong performance of the baseline could very well be from task-specific finetuning.

The table below reports the ablation at final checkpoints (Appendix E.6); each variant is inserted alone into the 20-baseline pool. The last column is TAFFY-4L minus the row: paired accuracy difference (pp, 95% CI), wins / ties / losses, and Wilcoxon p (Holm over the five rows in parentheses).

Model (final checkpoint)rank [95% CI]mean acc (%)TAFFY-4L minus row
(a) single pass, no multi-env. contexts5.80 [5.05, 6.53]83.85+0.40 [+0.18, +0.68], 64/37/33, p 0.003 (0.013)
(b) single pass + multi-env. contexts5.63 [4.87, 6.34]83.97+0.28 [+0.06, +0.52], 60/39/35, p 0.008 (0.027)
(c) 2 loops, global gate5.37 [4.60, 6.14]83.92+0.33 [+0.04, +0.66], 60/44/30, p 0.007 (0.027)
(d) 2 loops, task gate (TAFFY-2L)5.48 [4.78, 6.19]84.07+0.18 [−0.23, +0.55], 58/44/32, p 0.010 (0.027)
(e) TAFFY-3L5.30 [4.57, 6.12]84.02+0.23 [+0.01, +0.45], 55/48/31, p 0.018 (0.027)
(f) TAFFY-4L4.81 [4.11, 5.55]84.25—
  • The full model is significantly above every ablated variant (Holm p ≤ 0.03). Each component adds a step, and together they add +0.40 pp and about one rank (5.80 → 4.81).
  • The strong baseline is our own. Variant (a) is the single-pass version of TAFFY, trained by us with the same pipeline. That it already matches the strongest external foundation model (5.80 vs. 5.82 for TabPFN3) shows that the pipeline is competitive; TAFFY's components then move the model clearly ahead, to first place on every classification suite (Appendix E.2).

Size and configuration. TAFFY and our single-pass baseline use the TabICLv2 architecture (§4.3–4.4: column and row blocks, a 12-block ICL Transformer of width 512) and have 27.57M parameters.

The baseline's strength does not come from fine-tuning. Neither TAFFY nor our single-pass baseline is fine-tuned on any benchmark; like the other foundation models in Table 1, both predict with frozen parameters.

The lead also holds under stricter protocols (Appendix E.2–E.3):

  • At the final checkpoint, without checkpoint selection, TAFFY is first on all six classification suites.
  • On datasets that share no name with the TALENT datasets whose column statistics our classification prior uses, TAFFY is first: 4.81 vs. 5.92 for TabPFN3 (gap 95% CI [0.34, 1.86]), with a paired gain of +0.63 pp [+0.27, +1.01]. Under a stricter, fuzzy name match: 4.89 vs. 6.02 (gap CI [0.25, 1.97]).
Reviewer_LUpMOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors train a tabular foundation model with an in-context diversity prior, and a looped transformer.

Critical strengths

The authors describe their results clearly and study an interesting question which has implications for future research

Critical weaknesses

I am unsure what we learn from this paper. To my understanding, the authors recapitulate known ideas from other settings in the tabular foundation model setting, and show that they improve performance.

Questions for authors

Could the authors clarify how much each part of the in-context diversity prior contributes to the final performance of the model? (Label shift, conditional shift, covariate shift, …, how much does each contribution help?)

Are the gains from the looped transformer just due to more compute? Or is there some connection between the model architecture and the data that make looped transformers a particularly effective choice in this setting?

6 comments
AuthorsOpenReview ↗

Response to Reviewer LUpM (Part 1/3)

We thank the reviewer for the review and the two focused questions.

Weakness. I am unsure what we learn from this paper. To my understanding, the authors recapitulate known ideas from other settings in the tabular foundation model setting, and show that they improve performance.

We respectfully disagree. The review states no substantive concern about soundness. The measurements below show that the known ideas do not transfer by themselves: depth added without training hurts, extra inference compute does not help, and the gains come from the specific design we propose.

What this paper contributes.

  1. A new form of pretraining context for tabular foundation models. Existing tabular priors draw one table per context. In TAFFY, each context holds several related environments of one generating process, without environment identifiers, so that the model learns from the contrasts within one context.
  2. A task-conditioned looped ICL stack. The ICL Transformer is applied repeatedly with shared weights, and each pass's update is scaled by a gate computed from the labeled support set. Depth grows without new parameters; the gate adds 52.
  3. A finding about tabular in-context learning: extra depth helps only when it is trained. Repeating a trained stack at inference lowers accuracy, more ensemble members do not help, and each added trained loop improves the rank.
  4. A new state of the art. TAFFY has the lowest average rank on all six classification and all five regression suites, against 20 baselines that include TabPFN-3, a model with about twice its parameters.

A contribution does not have to be a new building block. Much of the progress in deep learning came from showing how a known idea works in a new setting. ViT [1] applied a standard Transformer to images with as few changes as possible. MAE [2] carried BERT-style masked prediction to vision. TabPFN [3], the starting point of tabular foundation models, brought prior-data fitted networks [4] to tabular classification. Looped Transformers [5][6] took the weight sharing of Universal Transformers [7] to in-context learning and to reasoning. The value of these works lies not in a new component, but in showing what it takes for a known idea to work in a new setting and what it then delivers. This paper does that for tabular foundation models, and the review itself acknowledges that the result improves performance.

References

  • [1] A. Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021.
  • [2] K. He et al. Masked autoencoders are scalable vision learners. CVPR 2022.
  • [3] N. Hollmann et al. TabPFN: A transformer that solves small tabular classification problems in a second. ICLR 2023.
  • [4] S. Müller et al. Transformers can do Bayesian inference. ICLR 2022.
  • [5] L. Yang et al. Looped transformers are better at learning learning algorithms. ICLR 2024.
  • [6] N. Saunshi et al. Reasoning with latent thoughts: On the power of looped transformers. ICLR 2025.
  • [7] M. Dehghani et al. Universal Transformers. ICLR 2019.
AuthorsOpenReview ↗

Response to Reviewer LUpM (Part 2/3)

Weakness, continued: the evidence for each contribution.

Contribution 1: the pretraining context.

  • Diversity studies for in-context learning vary the number of distinct pretraining tasks [8] or how examples cluster within a sequence [9]. Drift-Resilient TabPFN [10] varies environments, but supplies ordered domain indices. TAFFY places several perturbed environments of one generating process in one context, without identifiers; the model has to integrate them from the context alone.
  • The prior and the loops are designed to work together. Trained on this prior, TAFFY-4L is +0.28 pp [+0.06, +0.52] above the single-pass model on the same prior (Holm p = 0.016), and +0.40 pp [+0.18, +0.68] above our single-pass model pretrained without it (64 / 37 / 33, p = 0.003).

Contributions 2 and 3: trained loops versus extra computation.

  • Applying a single-pass model's ICL stack four times at inference, the same number of passes as TAFFY-4L, lowers its accuracy, by 0.88 pp and 1.20 pp for our two single-pass models (p < 0.001 for both; Appendix E.8).
  • More inference compute does not help either: with 4× more ensemble members the single-pass model does not improve (rank 5.63 → 5.65) and still trails TAFFY-4L by 0.31 pp (Holm p = 0.016).
  • TAFFY-4L's trained loops are +1.16 pp [+0.70, +1.67] above the repeated stack (83 / 28 / 23 wins / ties / losses, p < 1e-6), and each added trained loop improves the rank: 5.48 (2L), 5.30 (3L), 4.81 (4L); 4L vs. 2L p = 0.010, 4L vs. 3L p = 0.018.
  • Looping is also more efficient per pretraining task: TAFFY-4L reaches the single-pass model's final rank after 16,000 of 25,000 steps.

Contribution 2: the trained model relies on its task-dependent gate.

  • On the frozen final TAFFY-4L, removing the gate's task dependence (w = 0) costs 0.15 pp [0.07, 0.24] (Holm p < 0.001; rank 4.81 → 5.06), and fixing α = 1 costs 0.56 pp [0.30, 0.85] (p < 1e-4).
  • The learned gate value tracks properties of the features, their categorical fraction and kurtosis, with bootstrap CIs and FDR control (Table 2 and Appendix C.11).

Contribution 4: the margin over the strongest models. On 281 unique classification datasets (272 for AutoGluon and CatBoost), TAFFY's paired gain over each of the seven foundation models, AutoGluon and CatBoost is significant (Holm p < 1e-4). It has 27.57M parameters, about half of TabPFN-3's 53M.

These are findings about tabular in-context learning that were not available before this paper: trained depth helps where repeated computation and extra ensembling do not, and the trained model adapts its update to each dataset through the gate. We ask the reviewer to reassess the paper's informativeness in light of them.

References

  • [8] A. Raventós, M. Paul, F. Chen, S. Ganguli. Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression. NeurIPS 2023.
  • [9] S. C. Y. Chan, A. Santoro, A. K. Lampinen, J. X. Wang, A. Singh, P. H. Richemond, J. McClelland, F. Hill. Data distributional properties drive emergent in-context learning in transformers. NeurIPS 2022.
  • [10] K. Helli, D. Schnurr, N. Hollmann, S. Müller, F. Hutter. Drift-resilient TabPFN: In-context learning temporal distribution shifts on tabular data. NeurIPS 2024.
AuthorsOpenReview ↗

Response to Reviewer LUpM (Part 3/3)

Q1. Could the authors clarify how much each part of the in-context diversity prior contributes to the final performance of the model? (Label shift, conditional shift, covariate shift, …, how much does each contribution help?)

The parts are five table-level operators in two groups (Appendix E.1):

  • Distribution shifts: class-prior shift (a label shift), label-conditional shift (a conditional shift of one feature given the label) and noise shift (feature corruption, closest to a covariate shift).
  • Dependence perturbations: hard or soft proxy-edge perturbation between neighbouring columns.

The operators work together. Within one context, every environment receives a different operator, so each context places several kinds of shift side by side. Their contribution is therefore best read as a whole:

  • Of the +0.40 pp [+0.18, +0.68] that TAFFY-4L gains over our single-pass model pretrained without multi-environment contexts (64 / 37 / 33 wins / ties / losses, p = 0.003), +0.12 pp comes from the prior at a single pass and +0.28 pp [+0.06, +0.52] from the trained loops on this prior (Holm p = 0.016).
  • We will add a per-operator breakdown to the final version.

Q2. Are the gains from the looped transformer just due to more compute? Or is there some connection between the model architecture and the data that make looped transformers a particularly effective choice in this setting?

No. The gain does not come from more parameters or more inference compute, and repeating computation without training it lowers accuracy. All comparisons are on the same prior (Appendix E.5 and E.8).

  • (a) Parameters. Looping adds none; the task gate adds 52 (27.57M in total).
  • (b) Inference compute. Giving the single-pass model 2× and 4× more ensemble members (with 4×: 1.92× TAFFY-4L's total inference time) does not improve it: rank 5.63 → 5.67 → 5.65. It still trails TAFFY-4L by 0.31 pp [0.08, 0.53] (Holm p = 0.016).
  • (c) The same passes, untrained. Running the single-pass model's stack four times at inference, as TAFFY-4L does, lowers its accuracy by 0.88 pp; TAFFY-4L is +1.16 pp [+0.70, +1.67] above it (83 / 28 / 23 wins / ties / losses, p < 1e-6). The gain comes from training the loops, not from the extra passes.
  • (d) Pretraining data. Looping also uses pretraining data more efficiently: TAFFY-4L reaches the single-pass model's final rank after 16,000 of 25,000 steps (5.60 vs. 5.63), that is, with 64% of the pretraining tasks, and leads it from 19,000 steps on.

Why looping fits this data. Each pretraining context holds several environments of one generating process, and the model has to work out from the context which relations hold across them. The looped stack is designed for this: each pass re-reads the support set and refines the representation, and the gate sets, for each dataset, how much each pass changes it. The measurements are consistent with this link:

  • Trained on this prior, TAFFY-4L is +0.28 pp [+0.06, +0.52] above the single-pass model on the same prior (Holm p = 0.016), and +0.40 pp [+0.18, +0.68] above the single-pass model pretrained without multi-environment contexts (64 / 37 / 33, p = 0.003).
  • The trained model adapts its update to each dataset: removing the gate's task dependence costs 0.15 pp [0.07, 0.24] (Holm p < 0.001), and fixing α = 1 costs 0.56 pp [0.30, 0.85] (p < 1e-4).
  • The learned gate value tracks properties of the data, their categorical fraction and kurtosis (Table 2).

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank all three reviewers. We address the three central concerns here. All new numbers come from frozen-parameter inference at the final checkpoint; the full tables are in the new Appendix E of the revised PDF.

1. Novelty: the known ideas do not transfer by themselves (LUpM, p63S).

  • Running a trained single-pass stack four times at inference lowers its accuracy by 0.88 pp; TAFFY's trained loops are +1.16 pp [+0.70, +1.67] above it (p < 1e-6), and more inference compute does not close the gap.
  • The full design is significantly better than each reduced version (Holm p ≤ 0.03), and the trained model relies on its task-dependent gate (removing it costs 0.15 pp, Holm p < 0.001).
  • TAFFY has the lowest average rank on all six classification and all five regression suites, with about half the parameters of TabPFN-3.

2. Soundness: TAFFY is not fine-tuned on any benchmark (p63S).

  • All foundation models in the paper, ours included, predict with frozen parameters by in-context inference (Appendix C.3).
  • The lead also holds under stricter protocols: at the final checkpoint, and on the datasets that share no name with the TALENT datasets used by our prior (4.81 vs. 5.92 for TabPFN3).

3. Uncertainty and effect sizes (Uygp).

  • Over 281 datasets, TAFFY's paired gain over each of nine strong baselines is significant (Holm p < 1e-4), including +0.44 pp [+0.25, +0.65] over TabPFN3 and +0.28 pp [+0.05, +0.49] over TabICLv2. We have reported CIs, win / tie / loss counts and loss tails for every comparison.
  • Across checkpoints, ensemble seeds and row/column orders, TAFFY's rank has sd 0.10 and it stays first. Repeated pretraining costs thousands of GPU-hours per run and is uncommon in foundation-model research; we have bounded its effect and will add seed replicates to the final version.
AuthorsOpenReview ↗

Comment to the Area Chair

Dear AC,

Thank you for handling our submission and for overseeing the review and discussion. We would like to draw your attention to three points in the reviews.

  • Reviewer LUpM. The review raises no concern about soundness; its main criticism is that the paper recapitulates known ideas. Our measurements show that these ideas do not transfer by themselves, and TAFFY ranks first on all eleven suites. We respectfully believe this assessment is not objective.
  • Reviewer p63S: a factual error. The review assumes that our model was fine-tuned on the benchmarks. This is incorrect: all foundation models in the paper, ours included, use frozen-parameter in-context inference (Appendix C.3). The soundness score of 1 rests on this error.

Thank you for your time.

AuthorsOpenReview ↗

Request for Reviewer Discussion

Dear AC,

Thank you for overseeing the discussion of our submission. As the author–reviewer discussion closes today, we would like to briefly summarize where it stands.

  • Reviewer Uygp has engaged in two rounds of follow-up and has stated that the score rises from 2 to 3.
  • Reviewer LUpM has not responded, despite a reminder. The review raises no concern about soundness; its rating rests only on the view that the paper recapitulates known ideas, which our new measurements answer directly. We respectfully believe this assessment is not objective.
  • Reviewer p63S has not yet replied to our response or to our follow-up.The review’s soundness score of 1 rests on the assumption that TAFFY is fine-tuned on the benchmarks. This is a factual error (Appendix C.3), and without a response, the score still rests on it.

Since two of the three reviewers have not yet engaged with our responses, we would be grateful if you could consider initiating a discussion among the reviewers, so that the final assessment reflects the evidence now available. We understand that this is a busy period, and we thank you again for your time and care.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.