- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TAFFT, a tabular foundation model for in-context classification and regression. TAFFY builds on the tabular in-context learning paradigm with two core modifications. The first is its diversity prior, which comes from a synthetic causal model using distribution shifts and interventions to pretrain the model. The second is the task conditioned loop transformer which uses statistics from the labeled support data to produce a scalar that attenuates the final update. They evaluated the model across a broad suite of benchmarks and consistently outperformed common baselines on every benchmark.
Critical strengths
-
Broad and strong empirical performance. They evaluate TAFFY on several benchmarks, and it outperforms strong competitors on them. This suggests something very real here independent of other critiques.
-
The paper has a couple strong mechanistic ablations. The comparison of in context vs cross context diversity, task specific vs global gating, varying loop depth, and the frozen checkpoint intervention all add color to why TAFFY works to the extent that it does. Although they have some ablations I would like that are missing, it isn't reasonable to ask for every possible variation, and I appreciate the work put into these experiments.
Critical weaknesses
-
The main results, which are the biggest strength of this paper, don't have uncertainty quantification of effect sizes. The headline results are reported primarily through average ranks across datasets, but I do not find confidence intervals, standard errors, repeated-training variability, or paired significance analyses for these main comparisons. For a primarily empirical paper, this feels like a glaring omission. Average rank also doesn't indicate how much stronger TAFFY is than its competitors. When it wins, what is the margin by which it does so, and when it loses, what is that margin? It is very difficult to understand the robustness of TAFFY without those kinds of metrics.
-
The current ablations don't fully isolate why TAFFY is outperforming competitors. The in-context diversity experiment shows that diversity within a task works better than diversity between tasks, but it seems as though the relevant comparison would be whether causally related data is more important than just heterogeneous data. Similarly, the main ablation adds the looped Transformer and task-conditioned gate together, which the authors acknowledge doesn't isolate those two effects, and the looped models also use more computation. As a result, the experiments show that the complete TAFFY works well, but do not clearly establish whether the improvement is specifically due to the proposed causal structure and task adaptation, rather than more heterogeneous synthetic tasks and additional computation.
Questions for authors
-
Any form of uncertainty quantification + effect size quantification would greatly improve this paper. This is particularly important because the paper does provide bootstrap confidence intervals for a secondary gate-analysis experiment, but not for the central results, which suggests it is plausible to do such analyses.
-
Can you provide evidence that the specific causal intervention mechanism is important for the performance of TAFFY as opposed to just heterogeneous data?
-
How much of the architectural gain remains under a compute-matched comparison? Taffy-4L requires substantially more training time per step than the single-pass backbone. A comparison against a non-looped model given a similar training/inference compute budget would materially affect whether I view the improvement as evidence for the recurrent/task-conditioned architecture specifically, rather than for additional computation more generally.
A quality answer to (1) alone will raise my score, as I think that's the most important omission given that this paper is mostly empirical.
9 comments
Response to Reviewer Uygp (Part 2/4)
W1, continued: regression, sensitivity and pretraining-seed variance.
Table R3 — regression (Appendix E.2). The submitted regression model (step 22,175, the newest checkpoint available when the regression evaluation was run). Average rank with 95% CI; Δ = mean relative RMSE reduction (%) with 95% CI; p = Wilcoxon, Holm-corrected within suite.
| Suite | n | TAFFY rank | vs TabPFN3: W/L, Δ | vs TabICLv2: W/L, Δ |
|---|---|---|---|---|
| BCCO | 46 | 2.07 [1.52, 2.72] | 35/11, +3.2% [+0.8, +6.5], p <1e-4 | 39/7, +4.4% [+2.4, +7.0], p <1e-4 |
| CTR23 | 33 | 2.94 [1.85, 4.27] | 27/6, +11.2% [+4.6, +19.4], p <1e-4 | 25/8, +4.7% [-2.7, +12.2], p 0.003 |
| PFN | 28 | 3.29 [2.25, 4.50] | 19/9, +8.4% [+1.4, +17.0], p 0.056 | 18/10, +1.3% [-6.6, +9.4], p 0.19 |
| TALENT | 89 | 2.44 [1.94, 3.04] | 67/22, +5.8% [+2.1, +9.6], p <1e-4 | 76/13, +2.4% [-3.0, +6.5], p <1e-4 |
| TabArena | 13 | 1.31 [1.08, 1.62] | 11/2, +2.1% [+1.2, +3.1], p 0.004 | 12/1, +3.8% [+1.8, +6.3], p 0.004 |
- TAFFY has the lowest average-rank point estimate on all five suites and wins more datasets than it loses on each. The paired test is significant on 4/5 suites against each baseline; PFN (28 datasets) is not resolved. Against TabICLv2 on CTR23 and TALENT, the test is significant while the CI of the mean includes zero: TAFFY wins most datasets, and a few large losses widen the CI of the mean.
Table R4 — within-run sensitivity (Appendix E.4; each configuration inserted alone into the 20-baseline pool; rank among 21).
| Configuration | rank [95% CI] | mean acc (%) |
|---|---|---|
| step 25,000 (final, reported) | 4.81 [4.11, 5.55] | 84.25 |
| step 24,500 | 4.77 [4.10, 5.50] | 84.25 |
| step 24,000 | 4.88 [4.19, 5.59] | 84.32 |
| ensemble seed 1 (reported: seed 42) | 4.60 [3.95, 5.32] | 84.28 |
| ensemble seed 2 | 4.91 [4.21, 5.65] | 84.25 |
| row/column order 1 | 4.80 [4.17, 5.50] | 84.19 |
| row/column order 2 | 4.74 [4.00, 5.47] | 84.28 |
- TAFFY is first in all seven. Its rank has sd 0.10, against a gap of at least 1.01 ranks to TabPFN3.
Repeated-training variability. In foundation-model research, pretraining runs are rarely repeated because each one is expensive. One TAFFY-4L run takes 5,611 GPU-hours (3.65 days on 64 AMD MI210 GPUs; Table 5 of the paper), about USD 11,000 at an assumed cloud price of USD 2 per GPU-hour. Repeating our six-model ablation with two more seeds would take about 49,000 GPU-hours, roughly USD 100,000. TabPFN [1], TabICL [2], TabICLv2 [3] and LimiX [4], for example, each report a single pretraining run; the TabICL authors report 20 days of pretraining on three A100 GPUs for their own model [2]. We nevertheless quantified the variability we could:
- Within one run, over checkpoints, ensemble seeds and row/column orders (Table R4), TAFFY's rank has sd 0.10 and stays first.
- Across runs, we bounded it. If a different pretraining seed shifted TAFFY's average rank by a random amount with standard deviation s, the lead on the three suites where it is significant would lose significance only at s = 0.29 (BCCO), 0.50 (OpenML) and 0.31 (TALENT), that is 2.9–5.1 times the within-run sd.
References
- [1] N. Hollmann, S. Müller, K. Eggensperger, F. Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. ICLR 2023.
- [2] J. Qu, D. Holzmüller, G. Varoquaux, M. Le Morvan. TabICL: A tabular foundation model for in-context learning on large data. ICML 2025.
- [3] J. Qu, D. Holzmüller, G. Varoquaux, M. Le Morvan. TabICLv2: A better, faster, scalable, and open tabular foundation model. ICML 2026.
- [4] X. Zhang et al. LimiX: Unleashing structured-data modeling capability for generalist intelligence. arXiv:2509.03505, 2025.
Response to Reviewer Uygp (Part 3/4)
W2. The current ablations don't fully isolate why TAFFY is outperforming competitors. […] it seems as though the relevant comparison would be whether causally related data is more important than just heterogeneous data. Similarly, the main ablation adds the looped Transformer and task-conditioned gate together, which the authors acknowledge doesn't isolate those two effects, and the looped models also use more computation. […]
We have separated the loop, the gate and the extra computation, and stated what relates the environments of one context (Appendix E.1, E.5, E.6 and E.8).
- The loop, with the same task gate. Rank 5.48 (2L), 5.30 (3L), 4.81 (4L); 4L vs. 2L p = 0.010, 4L vs. 3L +0.23 pp [+0.01, +0.45], p = 0.018. The rank improves with every added trained loop.
- The gate, with the loops fixed. Within the trained four-loop model, the task dependence is used: removing it at inference (w = 0; α = tanh(a) = 0.34 for every dataset) costs 0.15 pp [0.07, 0.24] (45 / 70 / 19 wins / ties / losses for the unmodified model, Holm p < 0.001), and α = 1 costs 0.56 pp [0.30, 0.85] (p < 1e-4). Trained separately at two loops, task and global gates are close (+0.15 pp [−0.26, +0.72], p = 0.53).
- The extra computation. Computation alone does not produce the gain: four untrained passes of a single-pass model's stack lower its accuracy by 0.88 pp, and TAFFY-4L is +1.16 pp [+0.70, +1.67] above them (p < 1e-6); 4× more ensemble members leave the single-pass model's rank unchanged (5.63 → 5.65).
- Related versus heterogeneous data. The environments of one context are related by construction: each is the same generated table under a different small perturbation, and we have described them in the revision as related environments of one generating process. The runs with and without these contexts use the same generator and the same 25,000-step schedule; only what a context contains differs. Trained with these contexts, TAFFY-4L is +0.40 pp [+0.18, +0.68] above our single-pass model trained without them (p = 0.003). A control pretrained on mixtures of unrelated generating processes will be added to the final version.
Q1. Any form of uncertainty quantification + effect size quantification would greatly improve this paper. This is particularly important because the paper does provide bootstrap confidence intervals for a secondary gate-analysis experiment, but not for the central results, which suggests it is plausible to do such analyses.
We agree, and we have applied the same bootstrap to every central result (Appendix E.2–E.4):
- Uncertainty: dataset-bootstrap 95% CIs (B = 2,000) for mean accuracy and average rank on all six classification suites, for average rank on all five regression suites, and for the rank gap to the best baseline.
- Effect sizes: paired accuracy differences with 95% CIs against nine strong baselines, with win / tie / loss counts, median win and loss, and the 90th-percentile and largest loss.
- Significance: Wilcoxon signed-rank tests, Holm-corrected; all nine classification comparisons give p < 1e-4.
- Robustness: within-run sensitivity over checkpoints, ensemble seeds and row/column orders (rank sd 0.10), and the pretraining-seed variation the lead could absorb (2.9–5.1 times that sd).
In numbers: +0.44 pp [+0.25, +0.65] over TabPFN3 and +0.28 pp [+0.05, +0.49] over TabICLv2, and the rank gap to the best baseline excludes zero on BCCO, OpenML and TALENT.
Response to Reviewer Uygp (Part 1/4)
We thank the reviewer for a precise and constructive review.
Thanks for identifying the most important omission of the submission. We have reported effect sizes with CIs, win/tie/loss counts with loss tails, and paired tests.
Table R1 — per suite (Appendix E.2). Mean accuracy (%) and average rank among 21 methods, with dataset-bootstrap 95% CIs. "Best other" is the baseline with the lowest average rank.
Table R2 — paired, 281 unique classification datasets (Appendix E.2; 272 for AutoGluon and CatBoost, which lack results on nine). Δ = TAFFY minus competitor accuracy (pp) with 95% bootstrap CI. Loss columns describe the datasets where TAFFY loses. p = Wilcoxon signed-rank, Holm-corrected over the nine comparisons.