- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors provide a substantial extension beyond the state of the literature on studying the layer-wise behavior of tabular foundation models. Their extensions include, among others, geometric and sample-wise extensions, and they re-evaluate claims from prior work. The study uses a large number of datasets from OpenML and TALENT as evidence.
Critical strengths
The two critical strengths are: study rigor and the extensiveness of new mechanistic analyses.
The study and its presentation seem rigorous, and the authors clearly put great effort into ensuring high-quality findings. The explanations are clear, the appendix provides all required additional details, and the figures directly represent the insights. The study is extremely well executed.
The large number of new ways to study the mechanistic layer-wise behavior of TFMs also shows significant creativity and understanding of the research question. Each new insight is well justified and worth sharing.
Critical weaknesses
The one critical weakness of the paper is the question of "so what?". Similar to many other studies on mechanistic interpretability, we are given a large amount of insight into an existing model and its existing behavior. But in the current manuscript, it remains unclear how to use these insights; the reader must determine why each insight might be useful. Similarly, the comparisons across models, which are only briefly mentioned in the main text, do not provide more insight into the generalization of the new insights. By the end of the paper, the reader is left with many unactionable insights and no leads for follow-up work from discussions or reflections. While the manuscript matches the study's scope, its value remains underexplored.
Questions for authors
How should researchers use your insights, or your insight-generation tools, when developing new tabular foundation models? Could these layer-wise measures test a valuable hypothesis?
What insights from TabPFN-3 help us generalize behavior across architectures, and what can it predict about future models like TabPFN-3.5? How should we interpret magnitude in this context?
4 comments
Author Response (2/2)
W2 and Q2. Cross-architecture generality and implications for future TFMs
We thank the reviewer for this question. Our paper has already examined the cross-model generality of our findings, although these results were not sufficiently emphasized in the main text. The insight that generalizes across architectures is not a fixed geometric trajectory or compression magnitude, but the separation among representation geometry, prediction commitment, and functional utility. We test this directly on TabPFN-3, TabPFN-v2, TabPFN-2.5, TabICL, and TabICLv2 by aligning each model at its own effective-dimensionality (ED) peak (Section 3.1, Lines 140–143; Table 11, Lines 1214–1223; Appendix C.4, Lines 1228–1256).
| Model | Uncommitted at ED peak | Peak-layer errors corrected | Correct predictions disrupted | Matched-readout gain |
|---|---|---|---|---|
| TabPFN-3 | 17.2% | 40.6% | 3.9% | +3.1 pp |
| TabPFN-v2 | 53.1% | 46.2% | 6.1% | +3.8 pp |
| TabPFN-2.5 | 68.1% | 53.0% | 5.4% | +4.6 pp |
| TabICL | 31.3% | 52.8% | 3.5% | +4.2 pp |
| TabICLv2 | 48.0% | 37.9% | 3.4% | +3.6 pp |
Across all five models, reaching the ED peak does not mean that inference is complete. At that point, 17.2%–68.1% of queries remain uncommitted. Subsequent layers correct 37.9%–53.0% of peak-layer errors while disrupting only 3.4%–6.1% of correct predictions, and matched-readout accuracy improves by 3.1–4.6 points (Table 11, Lines 1214–1223; Appendix C.4, Lines 1230–1256).
For future TFMs such as TabPFN-3.5, this yields a testable qualitative prediction: early readability or an ED peak may precede query-level inference completion. We do not predict identical commitment depths or compression ratios, because these magnitudes depend on architecture, scale, training, and decoding. Cross-model comparisons should instead use relative depth, each model's own ED peak, post-peak correction and disruption, and matched-readout gains.
Our analysis spans two distinct TFM architecture families. We select TabPFN-3 as our primary model because it integrates architectural advances from both the TabPFN and TabICL lineages, combining feature grouping, learned row compression, and deep row-level ICL. Its native decoder also enables consistent query-level prediction tracking across layers (Section 2, Lines 101–110; Section 3.2, Lines 158–165; Appendix A.1, Lines 709–795). Importantly, this architectural paradigm remains relevant to recent high-performing TFMs:TabFM [1] and Kumo-Tabular [2] employ similar row-compression and row-level ICL pipelines, while TabPFN-3.5 retains TabPFN-3's overall architectural design [3].
These findings provide concrete criteria for future model development: determine inference completion at the query level, avoid inferring redundancy from geometry alone, and use prediction trajectories to identify queries that should stop, receive additional refinement, or combine complementary information across depth. We have further demonstrated the feasibility of these directions through training-free early exit, targeted context augmentation, and layer-information fusion.
[1] Kong, W., et al. (2026). TabFM: A Zero-Shot Foundation Model for Tabular Data.
[2] NVIDIA (2026). NVIDIA Kumo Tabular Sets a New Accuracy-Efficiency Frontier for Tabular Prediction. NVIDIA Technical Blog.
[3] Jäger, B., et al. (2026). TabPFN-3.5: Technical Report.
Thank you for the replies and the updated paper. You addressed my questions and concerns. I have increased my score accordingly.
For clarity, I recommend referencing the subsection motivating the testable design decisions in Section 7. Right now it is hard to parse it from Section 7 by itself.
We sincerely thank the reviewer for the thoughtful evaluation, constructive feedback, and recognition of our revisions. We greatly appreciate your reconsideration of our work and your encouraging assessment.
We also appreciate your valuable suggestion regarding Section 7. In the revised manuscript, we will further strengthen the motivation for the proposed applications by explicitly connecting our mechanistic findings to the corresponding design decisions. Specifically, we will clarify how prediction stability motivates training-free early exit, prediction switching guides targeted context refinement, and cross-layer complementarity motivates information fusion. We will also add clearer references to the supporting analyses in the preceding sections.
Thank you again for your constructive suggestions, which have helped us substantially improve the clarity and practical relevance of our work.
Author Response (1/2)
We sincerely thank the reviewer for recognizing the rigor, creativity, and significance of our mechanistic study, as well as its clear presentation and comprehensive appendix. We also appreciate the constructive suggestions on practical implications. To demonstrate the practical value of our findings, we present three complementary applications: early exit for stable queries, targeted refinement for hesitant queries, and fusion of complementary layer information (Section 7, Lines 470–519; Appendix I.1–I.3, Lines 2390–2461).
W1 and Q1. From mechanistic findings to testable design decisions
We thank the reviewer for this question. To demonstrate the practical implications of our findings, we now provide experimental evidence for three complementary applications:
1. Training-free early exit using prediction consistency.
Cross-layer prediction agreement provides a label-free signal for adaptive-depth inference. If two designated layers predict the same class, the query exits at the later layer; otherwise, it continues to L23. This requires neither query labels nor an auxiliary decoder.
Across 272 datasets and 752,231 queries, the balanced policy reduces average executed depth by 22.0% with only a 0.67-point accuracy loss, demonstrating a controllable computation–accuracy trade-off (Section 7, Lines 476–484; Appendix I.1, Lines 2390–2409).
2. Identifying and refining hesitant queries using prediction switching.
We define prediction-switch frequency as the number of class changes between consecutive layers:
nch(x)=∑ℓ=1L−11[y^(ℓ)(x)=y^(ℓ−1)(x)].
This label-free measure reveals that queries with more prediction changes generally obtain larger gains from later computation:
The net L9–L23 gain increases from 0.0 pp for queries with no prediction switches to 19.2 pp for those with at least seven switches (Appendix I.2, Lines 2411–2432). We further use the fixed criterion nch≥2 to trigger targeted context augmentation, retrieving nearby training examples supporting the competing L9 and L13 predictions.
Without using query labels, this intervention improves accuracy by 2.52 points on triggered queries. As it uses a fixed threshold and requires retrieval and recomputation, we present it as a proof of concept rather than an optimized method (Section 7, Lines 502–513; Appendix I.2, Lines 2434–2442).
3. Exploiting complementary information across layers.
We further test whether different depths contain complementary predictive information by combining predictions from L9, L13, and L23. Three fusion coefficients are selected from {0.1,0.2,…,1.0}, with the best of 1,000 combinations chosen post hoc for each dataset.
Fusion improves mean accuracy by 0.98 points across 272 datasets, demonstrating that predictions from different depths contain exploitable complementary information. Because coefficients are selected post hoc using labels, this is an oracle analysis of functional complementarity, not a deployable label-free method (Section 7, Lines 492–499 and 515–519; Appendix I.3, Lines 2444–2461).
These results demonstrate three concrete uses of our mechanistic findings: stop stable queries, refine hesitant queries, and combine complementary predictions across depth. They establish practical value while leaving optimized, fully label-free query-adaptive policies for future work.