acceptodds
Under review as a conference paper at ICLR 2027

When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models

Abstract

Transformer-based tabular foundation models (TFMs) achieve state-of-the-art performance on small to medium-sized tabular prediction tasks while processing diverse inputs through a shared stack of layers. Recent layerwise analyses identify strong predictive performance in early TFM representations and substantial overlap in computation across layers. However, early predictive accessibility alone does not establish whether later layers are dispensable for every input, and aggregate accuracy can obscure which inputs benefit from additional computation. We study all 24 in-context-learning layers of TabPFN-3 across 272 classification datasets, combining representation geometry, prediction trajectories, truncation, and structural interventions, while also examining related representation dynamics in four additional TFMs. We uncover a pronounced expansion-compression trajectory, with stronger late dimensional contraction associated with larger increases in class-prototype separation. A targeted analysis of early-stabilizing samples shows that substantial representational changes persist after predicted classes stabilize, while locally ambiguous queries obtain larger accuracy gains from shallow to late layers than well-separated queries. Repeating a single pretrained block or perturbing layer order reduces predictive performance on average, constraining simple post-hoc substitutions in the frozen model. These findings distinguish predictive accessibility, class stability, and the benefits of continuing computation, supporting an input-dependent assessment of depthwise redundancy in TFMs.

Primary area:
interpretability and explainable AI
Keywords:
Tabular Foundation Models, Mechanistic Interpretability, Network Depth, Representation Dynamics

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_R8BUOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors provide a substantial extension beyond the state of the literature on studying the layer-wise behavior of tabular foundation models. Their extensions include, among others, geometric and sample-wise extensions, and they re-evaluate claims from prior work. The study uses a large number of datasets from OpenML and TALENT as evidence.

Critical strengths

The two critical strengths are: study rigor and the extensiveness of new mechanistic analyses.

The study and its presentation seem rigorous, and the authors clearly put great effort into ensuring high-quality findings. The explanations are clear, the appendix provides all required additional details, and the figures directly represent the insights. The study is extremely well executed.

The large number of new ways to study the mechanistic layer-wise behavior of TFMs also shows significant creativity and understanding of the research question. Each new insight is well justified and worth sharing.

Critical weaknesses

The one critical weakness of the paper is the question of "so what?". Similar to many other studies on mechanistic interpretability, we are given a large amount of insight into an existing model and its existing behavior. But in the current manuscript, it remains unclear how to use these insights; the reader must determine why each insight might be useful. Similarly, the comparisons across models, which are only briefly mentioned in the main text, do not provide more insight into the generalization of the new insights. By the end of the paper, the reader is left with many unactionable insights and no leads for follow-up work from discussions or reflections. While the manuscript matches the study's scope, its value remains underexplored.

Questions for authors

How should researchers use your insights, or your insight-generation tools, when developing new tabular foundation models? Could these layer-wise measures test a valuable hypothesis?

What insights from TabPFN-3 help us generalize behavior across architectures, and what can it predict about future models like TabPFN-3.5? How should we interpret magnitude in this context?

4 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely thank the reviewer for recognizing the rigor, creativity, and significance of our mechanistic study, as well as its clear presentation and comprehensive appendix. We also appreciate the constructive suggestions on practical implications. To demonstrate the practical value of our findings, we present three complementary applications: early exit for stable queries, targeted refinement for hesitant queries, and fusion of complementary layer information (Section 7, Lines 470–519; Appendix I.1–I.3, Lines 2390–2461).

W1 and Q1. From mechanistic findings to testable design decisions

We thank the reviewer for this question. To demonstrate the practical implications of our findings, we now provide experimental evidence for three complementary applications:

1. Training-free early exit using prediction consistency.

Cross-layer prediction agreement provides a label-free signal for adaptive-depth inference. If two designated layers predict the same class, the query exits at the later layer; otherwise, it continues to L23. This requires neither query labels nor an auxiliary decoder.

PolicyExit ruleDepth reductionAccuracyChange
Full depthExit at L230.0%84.80%—
ConservativeL9 = L21 → L217.3%84.56%−0.24 pp
BalancedL9 = L17 → L1722.0%84.13%−0.67 pp
AggressiveL9 = L13 → L1337.4%83.53%−1.27 pp

Across 272 datasets and 752,231 queries, the balanced policy reduces average executed depth by 22.0% with only a 0.67-point accuracy loss, demonstrating a controllable computation–accuracy trade-off (Section 7, Lines 476–484; Appendix I.1, Lines 2390–2409).

2. Identifying and refining hesitant queries using prediction switching.

We define prediction-switch frequency as the number of class changes between consecutive layers:

nch(x)=∑ℓ=1L−11[y^(ℓ)(x)≠y^(ℓ−1)(x)].n_{\mathrm{ch}}(x)=\sum_{\ell=1}^{L-1}\mathbf{1}\left[\hat y^{(\ell)}(x)\neq\hat y^{(\ell-1)}(x)\right].

This label-free measure reveals that queries with more prediction changes generally obtain larger gains from later computation:

Prediction changesQueriesL9 accuracyL23 accuracyCorrectionDisruptionNet gain
0111,71192.5%92.5%0.0%0.0%+0.0 pp
116,04976.8%82.6%11.5%5.8%+5.7 pp
216,01264.0%74.2%17.9%7.7%+10.2 pp
38,97554.2%66.7%23.0%10.5%+12.5 pp
47,94352.3%65.2%26.3%13.4%+12.9 pp
54,25145.6%62.1%27.1%10.6%+16.5 pp
63,62242.2%58.2%29.8%13.8%+16.0 pp
≥74,84533.3%52.5%30.4%11.2%+19.2 pp

The net L9–L23 gain increases from 0.0 pp for queries with no prediction switches to 19.2 pp for those with at least seven switches (Appendix I.2, Lines 2411–2432). We further use the fixed criterion nch≥2n_{\mathrm{ch}}\geq2 to trigger targeted context augmentation, retrieving nearby training examples supporting the competing L9 and L13 predictions.

TriggerQueriesBaselineAugmentedGain
nch≥2n_{\mathrm{ch}}\geq245,64872.41%74.93%+2.52 pp

Without using query labels, this intervention improves accuracy by 2.52 points on triggered queries. As it uses a fixed threshold and requires retrieval and recomputation, we present it as a proof of concept rather than an optimized method (Section 7, Lines 502–513; Appendix I.2, Lines 2434–2442).

3. Exploiting complementary information across layers.

We further test whether different depths contain complementary predictive information by combining predictions from L9, L13, and L23. Three fusion coefficients are selected from {0.1,0.2,…,1.0}\{0.1,0.2,\ldots,1.0\}, with the best of 1,000 combinations chosen post hoc for each dataset.

Dataset groupDatasetsL23 accuracyMean gainImproved
Easy (ES > 65%)14889.1%+0.88 pp59%
Medium (ES 35–65%)9374.2%+1.26 pp80%
Hard (ES < 35%)3159.9%+0.66 pp43%
All datasets272—+0.98 pp64%

Fusion improves mean accuracy by 0.98 points across 272 datasets, demonstrating that predictions from different depths contain exploitable complementary information. Because coefficients are selected post hoc using labels, this is an oracle analysis of functional complementarity, not a deployable label-free method (Section 7, Lines 492–499 and 515–519; Appendix I.3, Lines 2444–2461).

These results demonstrate three concrete uses of our mechanistic findings: stop stable queries, refine hesitant queries, and combine complementary predictions across depth. They establish practical value while leaving optimized, fully label-free query-adaptive policies for future work.

AuthorsOpenReview ↗

Author Response (2/2)

W2 and Q2. Cross-architecture generality and implications for future TFMs

We thank the reviewer for this question. Our paper has already examined the cross-model generality of our findings, although these results were not sufficiently emphasized in the main text. The insight that generalizes across architectures is not a fixed geometric trajectory or compression magnitude, but the separation among representation geometry, prediction commitment, and functional utility. We test this directly on TabPFN-3, TabPFN-v2, TabPFN-2.5, TabICL, and TabICLv2 by aligning each model at its own effective-dimensionality (ED) peak (Section 3.1, Lines 140–143; Table 11, Lines 1214–1223; Appendix C.4, Lines 1228–1256).

ModelUncommitted at ED peakPeak-layer errors correctedCorrect predictions disruptedMatched-readout gain
TabPFN-317.2%40.6%3.9%+3.1 pp
TabPFN-v253.1%46.2%6.1%+3.8 pp
TabPFN-2.568.1%53.0%5.4%+4.6 pp
TabICL31.3%52.8%3.5%+4.2 pp
TabICLv248.0%37.9%3.4%+3.6 pp

Across all five models, reaching the ED peak does not mean that inference is complete. At that point, 17.2%–68.1% of queries remain uncommitted. Subsequent layers correct 37.9%–53.0% of peak-layer errors while disrupting only 3.4%–6.1% of correct predictions, and matched-readout accuracy improves by 3.1–4.6 points (Table 11, Lines 1214–1223; Appendix C.4, Lines 1230–1256).

For future TFMs such as TabPFN-3.5, this yields a testable qualitative prediction: early readability or an ED peak may precede query-level inference completion. We do not predict identical commitment depths or compression ratios, because these magnitudes depend on architecture, scale, training, and decoding. Cross-model comparisons should instead use relative depth, each model's own ED peak, post-peak correction and disruption, and matched-readout gains.

Our analysis spans two distinct TFM architecture families. We select TabPFN-3 as our primary model because it integrates architectural advances from both the TabPFN and TabICL lineages, combining feature grouping, learned row compression, and deep row-level ICL. Its native decoder also enables consistent query-level prediction tracking across layers (Section 2, Lines 101–110; Section 3.2, Lines 158–165; Appendix A.1, Lines 709–795). Importantly, this architectural paradigm remains relevant to recent high-performing TFMs:TabFM [1] and Kumo-Tabular [2] employ similar row-compression and row-level ICL pipelines, while TabPFN-3.5 retains TabPFN-3's overall architectural design [3].

These findings provide concrete criteria for future model development: determine inference completion at the query level, avoid inferring redundancy from geometry alone, and use prediction trajectories to identify queries that should stop, receive additional refinement, or combine complementary information across depth. We have further demonstrated the feasibility of these directions through training-free early exit, targeted context augmentation, and layer-information fusion.

[1] Kong, W., et al. (2026). TabFM: A Zero-Shot Foundation Model for Tabular Data.

[2] NVIDIA (2026). NVIDIA Kumo Tabular Sets a New Accuracy-Efficiency Frontier for Tabular Prediction. NVIDIA Technical Blog.

[3] Jäger, B., et al. (2026). TabPFN-3.5: Technical Report.

Thank you for the replies and the updated paper. You addressed my questions and concerns. I have increased my score accordingly.

For clarity, I recommend referencing the subsection motivating the testable design decisions in Section 7. Right now it is hard to parse it from Section 7 by itself.

We sincerely thank the reviewer for the thoughtful evaluation, constructive feedback, and recognition of our revisions. We greatly appreciate your reconsideration of our work and your encouraging assessment.

We also appreciate your valuable suggestion regarding Section 7. In the revised manuscript, we will further strengthen the motivation for the proposed applications by explicitly connecting our mechanistic findings to the corresponding design decisions. Specifically, we will clarify how prediction stability motivates training-free early exit, prediction switching guides targeted context refinement, and cross-layer complementarity motivates information fusion. We will also add clearer references to the supporting analyses in the preceding sections.

Thank you again for your constructive suggestions, which have helped us substantially improve the clarity and practical relevance of our work.

Reviewer_2DRnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how predictions and representations evolve across the 24 in-context-learning layers of TabPFN-3 on 272 classification datasets, with additional representation analyses on four other TFMs. The paper reports an increase in effective dimensionality with a later drop, and stronger class separation in later layers. It also focuses on studying prediction trajectories at the sample level. It finds a majority of predictions stop changing early, while harder tasks and locally ambiguous samples tend to change longer. Prediction accuracy is further improved in later layers. The absolute gain is higher for locally ambiguous samples, and later-stabilizing tasks are more influenced by truncation. The paper also discusses repeated blocks and orderings of layers and claims that the depth of the network is not equally useful for all inputs.

Critical strengths
  1. The sample-level analysis is a useful and a new angle. The paper analyzes individual prediction performance across depth rather than only reporting dataset-level averages. It conducts the analysis from multiple complementary aspects, including stabilization depth, transition patterns, task difficulty, and local separation. This helps improve the understanding of TFM behaviors at a finer grain.

  2. The layer-matched readout head helps provide a fair comparison. Intermediate layers may look worse simply because the native decoder is trained to better match to the final representation. The paper fits a separate readout head at each layer, which makes the depthwise analysis and improvement conclusions more convincing.

Critical weaknesses
  1. The current analysis does not provide a test-time solution to “when does depth help”. The analysis around stabilization depth requires a full prediction trajectory, and local separation measures use the query label. They rely on information that is unavailable when deciding whether to exit early during test time. The analysis therefore only shows which samples benefited from more depth from a post-hoc view, but not how to identify them during inference. As such, the practical translation of the findings fall short.

  2. Some of the layerwise analyses are similar to those in prior TFM work. Pairwise layer similarity, layerwise readouts, and interventions based on repeating or reordering layers seem to have appeared in related studies, including Balef et al. (2025, 2026) and Küken et al. (2025), etc. It would make the paper’s contribution much clearer if the authors state explicitly on which experiments build on earlier work and which are newly added. Also, which of the findings are new vs observed elsewhere before, and which of the findings are most exciting and why.

References Balef, A. R., Koshil, M., and Eggensperger, K. (2025). Towards Understanding Layer Contributions in Tabular In-Context Learning Models. Workshop on AI for Tabular Data at EurIPS 2025. Balef Balef, A. R., Koshil, M., and Eggensperger, K. (2026). Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), PMLR 306:5864–5910. Proceedings of Machine Learning Research Küken, J., Purucker, L., and Hutter, F. (2025). Early Stopping Tabular In-Context Learning. 1st ICML Workshop on Foundation Models for Structured Data, 2025.

Questions for authors
  1. Is it possible to find a signal available during inference to predict whether a sample will benefit from more depth?

  2. The larger gains on ambiguous samples may be due to their starting point from lower accuracy at L9, and therefore have more room to improve. If the L9-to-L23 gain is normalized by the amount of error remaining at L9, do ambiguous samples still improve more?

  3. How were the expansion/plateau/compression boundaries chosen? From Fig 2, it seems that there is an upward trend from L0 to L20, and a downward trend from L20 to L23. It seems hard to tell whether a plateau period actually exists.

  4. In the repeated-block experiment, which block is repeated, and how sensitive are the results to this choice? A sensitivity analysis on the block chosen could be helpful, as prior work like Balef et al. (2026) showed that a trained looped single-layer TFM can achieve performance close to a standard model.

4 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely thank the reviewer for the careful evaluation and constructive suggestions. We particularly appreciate the recognition of our novel sample-level analysis and the rigor of our layer-matched readout experiments.

W1 & Q1. Can an inference-time signal predict whether a query will benefit from additional depth?

Yes. We considered this question in our query-level analysis and report the resulting evidence below. Cross-layer agreement and prediction-switch frequency provide directly observable, label-free signals of expected remaining depth utility. We have also incorporated these results into Section 7, “From Mechanistic Findings to Query-Level Decisions,” with complete protocols and additional results in Appendix I (Section 7, Lines 470–519; Appendix I, Lines 2384–2461).

1. Training-free early exit

If two designated layers predict the same class, the query exits at the later layer; otherwise, it continues to L23. This gate requires neither query labels nor an auxiliary model.

PolicyExit ruleAvg. depth reductionAccuracyAccuracy change
Full depthExit at L230.0%84.80%—
ConservativePred(L9) = Pred(L21) → exit at L217.3%84.56%−0.24 pp
BalancedPred(L9) = Pred(L17) → exit at L1722.0%84.13%−0.67 pp
AggressivePred(L9) = Pred(L13) → exit at L1337.4%83.53%−1.27 pp

Across 272 datasets and 752,231 queries, the balanced L9–L17 rule reduces average executed depth by 22.0% with a 0.67-point accuracy loss. The alternative operating points provide controllable compute–accuracy trade-offs.

2. Prediction switching identifies queries that benefit from depth

We define the number of prediction changes as

nch(x)=∑ℓ=1L−11[y^(ℓ)(x)≠y^(ℓ−1)(x)].n_{\mathrm{ch}}(x) = \sum_{\ell=1}^{L-1} \mathbf{1} \left[ \hat y^{(\ell)}(x) \neq \hat y^{(\ell-1)}(x) \right].

This signal uses only intermediate predictions and requires neither query labels nor an auxiliary model.

Prediction changes nchn_{\mathrm{ch}}QueriesL9 accuracyL23 accuracyCorrection rateCorruption rateNet gain
0111,71192.5%92.5%0.0%0.0%+0.0 pp
116,04976.8%82.6%11.5%5.8%+5.7 pp
216,01264.0%74.2%17.9%7.7%+10.2 pp
38,97554.2%66.7%23.0%10.5%+12.5 pp
47,94352.3%65.2%26.3%13.4%+12.9 pp
54,25145.6%62.1%27.1%10.6%+16.5 pp
63,62242.2%58.2%29.8%13.8%+16.0 pp
≥74,84533.3%52.5%30.4%11.2%+19.2 pp

The expected L9–L23 gain generally increases with prediction-switch frequency, from 0.0 points for queries with no switches to 19.2 points for queries with at least seven switches. We further use the fixed rule nch≥2n_{\mathrm{ch}}\geq2 to trigger targeted context augmentation.

TriggerEvaluated queriesBaseline accuracyContext-augmented accuracyAccuracy gain
nch≥2n_{\mathrm{ch}}\ge24564872.41%74.93%+2.52 pp

This intervention improves accuracy from 72.41% to 74.93% on 45,648 triggered queries without using their labels. The fixed threshold and second forward pass make this a scoped proof of concept rather than an optimized method.

3. Non-redundant depths contain complementary information

We combine predictions from L9, L13, and L23 to test whether information distributed across depth is functionally complementary.

Dataset group by Early-Stable ratioDatasetsL23 accuracyMean accuracy gainDatasets improved
Easy: ES > 65%14889.1%+0.88 pp59%
Medium: ES = 35–65%9374.2%+1.26 pp80%
Hard: ES < 35%3159.9%+0.66 pp43%
All datasets272—+0.98 pp64%

The dataset-level fusion improves mean accuracy by 0.98 points across 272 datasets, with improvements on 64% of datasets. Because the coefficients are selected post hoc using labels, this is an oracle test of complementarity rather than a deployable label-free policy.

These results provide three concrete uses of our layerwise analysis: stable queries can exit early, unstable queries can be identified for targeted refinement, and complementary predictions across layers can be fused. The signals predict expected benefit statistically rather than guaranteeing improvement for every query.

AuthorsOpenReview ↗

Author Response (2/2)

W2. What builds on prior work, and what is new?

We thank the reviewer for suggesting a clearer distinction between prior findings and our contributions.

Balef et al. (2025, 2026) :Early predictive accessibility and substantial aggregate depthwise redundancy, established through layerwise analyses and structural interventions. Küken et al. (2025) : Early exit using separately pretrained layerwise decoders and a dataset-level entropy threshold.

Our contributions are threefold:

  1. Distinguishing four query-level depth processes. We jointly analyze predictive accessibility, prediction commitment, geometric resolution, and functional utility through complete prediction trajectories, revealing when later computation remains useful beyond what aggregate similarity and readability can establish.

  2. Revealing the limits of early readability and stability. Even among Early-Stable queries, 69.2% of representation displacement occurs after L9, while independently fitted layer-matched accuracy rises from 82.2% at L2 to 85.6% at L23. Stable queries may remain incorrect, whereas repeatedly switching queries can eventually become correct.

  3. Characterizing and leveraging query-dependent depth utility. We distinguish corrections from disruptions and show that ambiguous queries retain larger depth gains after normalizing for L9 error headroom. These findings motivate training-free early exit, targeted refinement, and layer-information fusion.

Our central finding is that predictive accessibility, prediction commitment, geometric resolution, and functional utility need not coincide. Thus, depth is neither uniformly redundant nor uniformly necessary; its predictive value depends on the query.

Q2. Headroom-normalized gains

We thank the reviewer for this question. To account for the greater L9 error headroom of ambiguous samples, we compute the headroom-normalized net gain (NGR), considering both corrected errors and disrupted correct predictions.

Let eg=1−Acc9,ge_g=1-\mathrm{Acc}_{9,g} denote the L9 error rate, SgS_g the rescue rate, and RgR_g the retention rate. We define

NGRg=Acc23,g−Acc9,g1−Acc9,g=Sg−1−egeg(1−Rg).\mathrm{NGR}_g=\frac{\mathrm{Acc}_{23,g}-\mathrm{Acc}_{9,g}}{1-\mathrm{Acc}_{9,g}}=S_g-\frac{1-e_g}{e_g}(1-R_g).

Across ten local-separation bins, NGR remains significantly larger for more ambiguous samples (Spearman ρ=−0.721\rho=-0.721, p=0.019p=0.019): 18.4% in the most ambiguous bin versus 3.2% in the most separated bin.

Q3. Choice of the expansion, plateau, and compression boundaries

We thank the reviewer for this question. The three regimes are descriptive rather than fitted change points.

These metrics are independent of the L9/L17 boundaries. Appendix E.3 confirms robustness to alternative representative-layer pairs: (L8, L12), (L10, L14), (L9, L14), and (L8, L13) (Lines 1668–1672).

Importantly, plateau denotes relative geometric stabilization, not zero ED slope. Compared with rapid early expansion and pronounced late compression, intermediate layers maintain elevated ED, exhibit high CKA similarity, and show the lowest adjacent-layer displacement.

Q4. Repeated-block choice, trained weight sharing, and late-layer utility

We thank the reviewer for this question.

First, we repeat frozen Blocks L5, L13, and L20 for 24 steps, covering expansion, plateau, and compression. None reproduces the original expansion–compression trajectory or progressive class separation. Thus, the result is not specific to a particular Block: frozen Blocks cannot replace the learned sequence without retraining (Section 6.2, Lines 418–430; Appendix H.3).

Second, Balef et al. (2026) demonstrate that end-to-end trained weight sharing can support effective recurrent computation, which differs from repeating a pretrained frozen Block. As their looped-model checkpoints and exact training implementation are unavailable, we conduct a controlled nanoTabPFN comparison between six distinct Blocks and one Block shared across six steps, matching training data, architecture dimensions, computation depth, optimization budget, and 20 paired seeds.

Metric (L3 to L6)Distinct BlocksShared Block
Accuracy gain+2.144 pp+0.848 pp
Errors corrected6.20%3.96%
Correct predictions disrupted4.05%3.11%

Distinct Blocks outperform weight sharing in 19/20 seeds, yielding approximately 2.5× greater late-stage accuracy gain (Appendix D.3, Lines 1330–1384).

Third, interventions confirm the functional importance of distinct later Blocks: skipping L4/L5/L6 reduces accuracy by 0.380/0.367/0.906 pp, while swapping L4–L5 or reversing L4–L6 causes losses of 0.245/0.920 pp, respectively (Appendix D.3, Lines 1361–1392).

These findings complement rather than contradict Balef et al.

I thank the authors for their thorough response and additional experiments. Several of my questions have been directly addressed, particularly the headroom-normalized analysis and the repeated-block and shared-weight experiments.

However, some of my concerns still remain.

  1. While the agreement-based exit rule offers a useful label-free approach, a comparison with fixed-depth truncation at similar computational cost is needed to assess its effectiveness.
  2. The prediction-switch count relies on the full layer trajectory and thus supports post-hoc refinement rather than early-exit decisions.
  3. My concern about the plateau regime in Figure 2 also remains unresolved.

Nevertheless, the additional experiments strengthen the paper's main claims, and I have therefore raised my score to weak accept. I encourage the authors to incorporate the headroom-normalized analysis and the comparison with Balef et al. (2025) and Küken et al. (2025) into the revised manuscript, as these currently appear only in the response.

AuthorsOpenReview ↗

Round2 Author Response

We sincerely thank the reviewer for the thoughtful follow-up and for recognizing the novelty and value of our query-level analysis and layer-matched evaluation. The reviewer’s constructive suggestions have helped us strengthen the empirical evidence and clarify the presentation.

We have added the comparisons with Balef et al. (2025) and Küken et al. (2025) to the Related Work section (lines 127–132), included the headroom-normalized results in Appendix H.2 (lines 2308–2338), and added the compute-matched early-exit experiments to Appendix I.2 (lines 2495–2510). We address the three remaining concerns below.

1. Agreement-based routing versus compute-matched fixed truncation

We evaluated five agreement-based early-exit schedules on 72 datasets from OpenML-CC18. At each scheduled checkpoint, the online policy exits a query only if its predictions observed so far satisfy the consistency condition. Queries that reach the last checkpoint without exiting continue to the final layer. The gate uses only predictions already available at the current checkpoint and requires neither labels nor an auxiliary model.

For each schedule, we compared its accuracy with fixed-depth truncation at a matched mean number of executed Blocks. Mean executed depth is first averaged over queries within each dataset and then averaged across datasets. The mean depths of the adaptive and fixed policies differed by at most 0.04 Blocks.

Online scheduleMean executed BlocksAdaptive accuracyCompute-matched fixed accuracyGainDatasets improvedWilcoxon p
Single checkpoint L1317.2883.63%82.28%+1.36 pp73.6%1.6e-21
Single checkpoint L1720.0883.88%82.81%+1.07 pp69.4%1.9e-18
Single checkpoint L2122.7183.93%83.69%+0.23 pp51.4%2.4e-5
L9, L13, L1714.0883.13%81.61%+1.52 pp72.2%3.7e-20
L7, L11, L15, L19, L2112.2982.63%81.12%+1.51 pp75.0%4.1e-18

All five schedules outperform fixed truncation at comparable computation, with gains of 0.23–1.52 percentage points. All differences are statistically significant (p≤2.4×10−5p\leq2.4\times10^{-5}). For example, the L13 policy executes 17.28 of 24 Blocks on average and achieves 83.63% accuracy, 0.31 percentage points below the full-depth accuracy of 83.94%. The compute-matched fixed-depth baseline achieves 82.28%.

The comparison provides evidence that query-specific depth allocation improves accuracy at a matched average depth. The agreement gate allows queries with stable predictions to exit early and allocates later computation to queries whose predictions remain unstable. We present these results as a mechanism-guided proof of concept for training-free early exit.

2. Prediction switching is a label-free refinement signal

We agree with the reviewer’s interpretation that the complete prediction-switch trajectory can be used for post-hoc refinement. In our experiment, after the initial inference pass, we select queries with nch≥2n_{\mathrm{ch}}\geq2 for targeted context augmentation and recomputation without using their ground-truth labels.

The switch count nchn_{\mathrm{ch}} is cumulative and can be monitored during inference. The threshold nch≥2n_{\mathrm{ch}}\geq2 is met as soon as the second prediction change occurs and remains satisfied at all subsequent layers. A query below the threshold may still change its prediction later. We use prediction consistency to make early-exit decisions and prediction switching to identify queries for label-free second-stage refinement.

3. Clarification of the plateau regime

We use “plateau” to denote relative geometric stabilization, with effective dimensionality (ED) still increasing gradually over L9–L17. This interval lies between rapid early expansion and concentrated compression at L20–L23. It maintains a high effective dimension, forms a high-similarity CKA block, and exhibits the lowest adjacent-layer displacement. L9 and L17 serve as descriptive anchors for this regime. The main expansion and compression ratios depend only on the initial, maximum, and final ED values.

To prevent “plateau” from being interpreted as a mathematically flat interval, we will refer to it as a “relative plateau” or “high-dimensional middle regime” and clarify this definition in Figure 2 and its caption.

We thank the reviewer again for the constructive feedback and would appreciate hearing whether the additional experiments and clarifications help address the remaining concerns.

Reviewer_tJWnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how the computation is divided among the 24 in-context layers of a particular tabular foundation model, TabPFN-3, across 272 classification datasets from OpenML-CC18 and TALENT. Furthermore, the paper uses 4 more models for cross-model comparison, namely TabICLv1, TabICLv2, TabPFN-2.5, and TabPFNv2. The paper studies variant effects combining representation geometry, prediction trajectories, truncation, and structural interventions.

The authors show in their experiments that these TFMs exhibit an expansion-compression trajectory, with related representation dynamics observed in the other models. The authors also find that commitment to a class varies depending on task difficulty and local ambiguity.

Critical strengths
  1. The authors provide broad experimental coverage: The authors examine all 24 ICL layers of TabPFN-3 across a large number of datasets, with extra experiments on four other models
  2. The authors provide an insightful distinction between predictive accessibility, prediction stability, and the benefit of additional computation, which are often conflated when discussing redundancy in later layers. Furthermore, they also show that the differences in predictive performance persist even when each layer has its fitted readout, therefore establishing that these gains are not fully explained by a mismatched output layer.
  3. The authors provide an in-depth analysis of the early-stable cases. By focusing on samples whose predictions have already settled, the authors show that continued representational movement is not simply due to samples switching classes. This makes the analysis more informative than either examining prediction accuracy or representation geometry alone.
Critical weaknesses
  1. Limited novelty and missing mentions of relevant related works: The expansion-compression trajectory across depth is a phenomenon that has been observed before in large transformers [1,2]. The phenomenon of late growth in class separation relative to within-class spread (FDR, prototype margins) parallels progressive neural collapse [3,4]. The authors should discuss whether the relationship of their finding of late-depth resolution in the context of in-context learning to these prior works and why separation here concentrates late rather than accruing at a roughly constant rate across layers as in [4]. Furthermore, beyond these works, the contribution relative to [5] and [6] is incremental: [5] already shows early predictive accessibility and suggests depthwise redundancy in TFMs, and [6] already explores that ambiguous queries require deeper computation.
  2. Limited scope: The paper only discusses classification and completely ignores regression and focuses mainly on one model. While the authors provide extra analyses on four extra models in the appendix, their results don't exhibit exactly the same patterns found in the main results, so we cannot consider the findings on TabPFN-3 generalizable to all TFMs. And since the paper is mainly a study of these phenomena and does not attempt to explain them or offer a new methodology based on their findings, and is limited to one model and ignores regression, I consider the findings to be limited in scope and should be extended in depth to other models.
  3. The claim that ambiguous samples benefit more from depth could be a base-rate effect. Figure 7(b) shows that later layers actually rescue L9 errors more often for well-separated samples than for ambiguous ones. Ambiguous samples could be showing larger absolute gains simply because there are more such L9 errors to fix, while well-separated samples are already near their ceiling. Reporting relative error reduction, and separating fixed errors from newly broken predictions, would make this clearer.
  4. The evidence that the later layers add predictive information isn't convincing. With layer-matched readouts, NLL does not improve with depth, and accuracy fluctuates. The MLP probe improves on only 64% of datasets, and attn@V readability is highest at L2 and drops afterwards in 101 of 105 datasets. The big NLL improvement with the native decoder happens only in the last few layers. It's possible that the final layers adapt the representations to the decoder as additional computation. If so, the results support the redundancy view of [5] more than the paper suggests. 5.No direct test of middle-layer redundancy, and overstated intervention results. The CKA analysis shows L9–L17 as a highly similar block, but the interventions only repeat one frozen block or shuffle and reverse the entire stack. This puts the model far from its training setup and feeds blocks inputs they were never trained on, so performance should drop regardless of what the blocks compute. I suggest testing shuffling or skipping only the middle layers, which would directly test the redundancy question the paper discusses. The stronger claim in Section 6.2 and the conclusion, that later computation is “ordered rather than repetitive”, would require comparing against a model trained with the same compute and repeated layers that share weights (which is possible with the nanoTabPFN setup).

[1] Valeriani, L., Doimo, D., Cuturello, F., Laio, A., Ansuini, A., & Cazzaniga, A. (2023). The geometry of hidden representations of large transformer models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems (Vol. 36, pp. 51234–51252). doi:10.52202/075280-2230 [2] Cheng, E., Doimo, D., Kervadec, C., Macocco, I., Yu, L., Laio, A., & Baroni, M. (2025). Emergence of a High-Dimensional Abstraction Phase in Language Transformers. The Thirteenth International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=0fD3iIBhlV [3] Rangamani, A., Lindegaard, M., Galanti, T., & Poggio, T. A. (23--29 Jul 2023). Feature learning in deep classifiers through Intermediate Neural Collapse. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, & J. Scarlett (Eds.), Proceedings of the 40th International Conference on Machine Learning (pp. 28729–28745). Retrieved from https://proceedings.mlr.press/v202/rangamani23a.html [4] He, H., & Su, W. J. (2023). A Law of Data Separation in Deep Learning. arXiv [Cs.LG]. Retrieved from http://arxiv.org/abs/2210.17020 [5] Balef, A. R., Koshil, M., & Eggensperger, K. (06--11 Jul 2026). Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models. In T. Zhang, M. Dudik, M. Jaggi, A. Agarwal, S. Li, D. Schuurmans, … A. Bietti (Eds.), Proceedings of the 43rd International Conference on Machine Learning (pp. 5864–5910). Retrieved from https://proceedings.mlr.press/v306/balef26a.html [6] Liu, S.-Y., & Ye, H.-J. (2026). TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention. Forty-Third International Conference on Machine Learning. Retrieved from https://openreview.net/forum?id=ottLtChHp2

Questions for authors
  1. We see in Figure 10 that some models (such as TabPFNv2) compress less drastically than others in the deeper layers (Table 7). How does this behaviour translate into truncation and commitment behaviour, as we only see these analyses done for TabPFN-3? I think this would help the study establish whether the relationship of late compression to the value of later layers is generalizable, rather than being specific to TabPFN-3.
  2. In Table 14, the whitened expansion and compression ratios are identical to the raw ones. If whitening is applied independently at each layer, I find it unexpected that whitening would not affect the results at all. Could the authors clarify how they applied whitening, including any regularization or dimensionality reduction, and explain why the ratios remain unchanged?
12 comments
AuthorsOpenReview ↗

Author Response (1/3)

We sincerely thank the reviewer for the careful evaluation, positive feedback, and constructive suggestions.

W1. Novelty relative to prior geometric and TFM studies

We thank the reviewer; see revised Related Work (Section 2, Lines 122–125).

Early readability and average redundancy suggest later layers are expendable. But can aggregate evidence identify which queries still benefit from depth? We show that predictive accessibility, commitment, geometric resolution, and functional utility diverge: additional depth has query-dependent value (Sections 4.3–6.1, Lines 239–414). Averages cannot identify safely skippable computation.

[1,2] study intrinsic-dimensionality trajectories in language, image, and protein Transformers; [3,4] examine class statistics in fixed-task classifiers. Frozen TFMs instead solve support-defined new tasks. Neural collapse concerns within-class contraction, between-class variation, and classifier-weight alignment; our prototype margin captures query-specific separation of the two leading support-derived class prototypes. Class information is readable at L2, yet margins widen most near the output, alongside late displacement and dimensional contraction (Sections 4.1–4.3, Lines 176–284; Appendix F.3, Lines 1805–1828). Later layers thus reorganize readable information to resolve class competition, unlike the constant-rate separation in [4], which also does not hold for BERT.

Balef et al. [5], cited in our Introduction and Related Work, report early readability and substantial average depthwise redundancy. Yet averages can mask query-level differences in later computation's value. Even among Early-Stable queries, 69.2% of representation displacement occurs after L9 (Appendix G.5, Lines 2034–2095); independently fitted layer-matched readouts improve accuracy from 82.2% at L2 to 85.6% at L23 (Section 6.1, Lines 404–411; Appendix E.1, Lines 1499–1537). Readability, stability, and remaining utility are thus distinct.

TabSwift [6] trains per-query exits and observes later exits near class overlap. We instead assess net predictive benefit, counting both corrected errors and disrupted correct predictions. Trajectories, rescue/retention, and truncation reveal queries that remain stably wrong or switch classes repeatedly before becoming correct (Sections 5.1–6.1, Lines 285–414). Thus, ambiguity, late commitment, and benefit differ: neither late exits nor prediction changes alone establish utility.

W2 and Q1. Scope, model choice, and cross-model generality

We thank the reviewer for these comments.

1. Cross-model evidence and TabPFN-3 selection. To address Q1, we analyze commitment and truncation using the layerwise outputs in Figure 10 and Table 7. Independently fitted layer-matched readouts measure utility without native-decoder alignment confounds (revised Appendix C.4, Lines 1228–1256; Table 11, Lines 1214–1223).

ModelUncommitted at ED peakPeak errors correctedPeak correct disruptedReadout gain
TabPFN-317.2%40.6%3.9%+3.1 pp
TabPFN-v253.1%46.2%6.1%+3.8 pp
TabPFN-2.568.1%53.0%5.4%+4.6 pp
TabICL31.3%52.8%3.5%+4.2 pp
TabICLv248.0%37.9%3.4%+3.6 pp

At the ED peak, 17.2%–68.1% of queries remain uncommitted. Later layers correct 37.9%–53.0% of peak-layer errors, disrupt only 3.4%–6.1% of correct predictions, and improve matched-readout accuracy by 3.1–4.6 pp. Despite differing trajectories, all five models show that peak dimensionality does not imply completed inference or exhausted predictive utility. What generalizes is the dissociation among geometry, commitment, and functional value, not an identical compression curve or effect size.

These TFMs span cell-level attention and row-compression architectures. TabPFN-3 combines the TabPFN and TabICL lineages through feature grouping, learned row compression, and deep row-level ICL (Section 2, Lines 100–110; Appendix A.1, Lines 709–795). Its native decoder works across layers, permitting query trajectories without separate probes (Section 3.2, Lines 158–165). Recent state-of-the-art TFMs, including Kumo-Tabular and TabFM [7,8], employ similar designs.

2. Why classification? Classification is fundamental to machine learning and central to TFMs. Across 272 binary and multiclass datasets, discrete outputs precisely identify class switches, final-prediction commitment, stable errors, and later corrections without arbitrary thresholds, distinguishing readability, commitment, geometry, and utility. Regression needs a continuous commitment definition and remains future work.

[7] Kong, W., et al. (2026). TabFM: A Zero-Shot Foundation Model for Tabular Data.

[8] NVIDIA (2026). NVIDIA Kumo Tabular Sets a New Accuracy-Efficiency Frontier for Tabular Prediction. NVIDIA Technical Blog.

AuthorsOpenReview ↗

Author Response (2/3)

W3. The benefit for ambiguous samples is not explained by baseline headroom

We thank the reviewer for this suggestion. To test whether ambiguous samples benefit more merely because of their higher L9 error rate, we compute headroom-normalized net gain (NGR), equivalent to relative error reduction, accounting for both corrected and newly broken predictions.

Let eg=1−Acc9,ge_g=1-\mathrm{Acc}_{9,g} denote the L9 error rate, SgS_g the rescue rate, and RgR_g the retention rate. We define

NGRg=Acc23,g−Acc9,g1−Acc9,g=Sg−1−egeg(1−Rg).\mathrm{NGR}_g=\frac{\mathrm{Acc}_{23,g}-\mathrm{Acc}_{9,g}}{1-\mathrm{Acc}_{9,g}}=S_g-\frac{1-e_g}{e_g}(1-R_g).

Across ten local-separation bins, NGR remains significantly larger for more ambiguous samples (ρ=−0.721\rho=-0.721, p=0.019p=0.019): 18.4% in the most ambiguous bin versus 3.2% in the most separated bin.

Figure 7(b) is not contradictory: it measures the conditional rescue rate, whereas our claim concerns net predictive benefit after accounting for both corrections and disruptions. Thus, ambiguous samples benefit more from additional depth even after normalizing for their greater initial error headroom.

W4. Early predictive accessibility does not imply functional redundancy

We thank the reviewer for highlighting this distinction. We do not claim that later layers monotonically introduce new class information. The high attn@V readability at L2, followed by its decline, supports our central distinction: class information is accessible early, but predictive accessibility, prediction commitment, geometric resolution, and functional utility are not completed at the same depth.

Crucially, native-decoder adaptation cannot fully explain the later-layer benefit. With independently fitted layer-matched readouts, accuracy increases from 82.2% at L2 to 85.6% at L23, a gain of 3.4 percentage points (Appendix E.1, Lines 1432–1537; Figure 12, Lines 1470–1474). Probes independent of the native decoder also show positive mean late-versus-early gains in TabPFN-3, TabICL, and TabICLv2 ( Figure 13, Lines 1523–1530; Table 17, Lines 1566–1574). Thus, predictive gains remain after removing dependence on alignment with the native decoder.

The flat layer-matched NLL shows that log-probability quality does not improve alongside classification accuracy; it does not negate the significant accuracy gain. Similarly, improvement on 64% rather than all datasets supports our finding that depth utility is heterogeneous, not monotonically positive for every task or query. Early readability therefore does not justify treating later layers as functionally redundant.

W5-1. High CKA similarity does not imply post-hoc redundancy

We thank the reviewer for this suggestion. We perform two localized interventions on 72 CC18 datasets. First, skipping L10–L16 within the highly CKA-similar L9–L17 region decreases accuracy by 4.77 percentage points on average (median: −3.38 points), with a decrease on all 72 datasets. Second, swapping five adjacent Block pairs while preserving every Block, model depth, and computational cost reduces accuracy by 2.85 points and changes 6.9% of predictions. . More importantly, this effect is not uniform. Among originally correct predictions, 8.2% and 4.12% of the predictions in the low- and medium-confidence groups, respectively, become incorrect, whereas no correct prediction in the high-confidence group is corrupted (0%). These results show that high CKA similarity implies neither dispensability nor local interchangeability; the observed order sensitivity is not an artifact of global shuffling or reversal (revised Section 6.2, Lines 455–468).

AuthorsOpenReview ↗

Author Response (3/3)

W5-2. nanoTabPFN:Trained shared-weight control and late-layer interventions

We thank the reviewer for suggesting this control. A recurrently trained shared Block addresses a different question from repeating a frozen Block after training. Balef et al. (2026) show that a shared Block applied for six steps can approach the final performance of a six-layer nanoTabPFN.

However, similar final accuracy does not establish that the distinct later Blocks of a six-layer model are functionally redundant. Because the released repository does not provide the training implementation and configuration for this comparison, we independently train matched nanoTabPFN controls.

We compare a model with six independently parameterized Blocks against one that applies a shared Block for six steps. Both use the same architecture dimensions, training data, computation depth, optimization budget, and 20 paired random seeds. The distinct model achieves 76.143% accuracy, compared with 73.958% for the shared model, and performs better in 19 of 20 paired seeds (Appendix D.3, Lines 1330–1384).

MetricDistinct six-layer modelShared six-step model
L3 accuracy73.999%73.110%
L6 accuracy76.143%73.958%
Net gain from L3 to L6+2.144 pp+0.848 pp
Predictions corrected from L3 to L66.20%3.96%
Predictions corrupted from L3 to L64.05%3.11%
Predictions changed from L3 to L610.25%7.06%

The layerwise trajectories expose a distinction hidden by final accuracy: the distinct model gains 2.144 points from L3 to L6, approximately 2.5 times the gain from shared repetition. Direct interventions further reduce accuracy when L4, L5, or L6 is skipped, when L4 and L5 are swapped, or when L4–L6 is reversed (Appendix D.3, Lines 1361–1392).

InterventionAccuracyChange from full model
Full six-layer model76.143%—
Exit after L373.999%−2.144 pp
Skip L475.763%−0.380 pp
Skip L575.776%−0.367 pp
Skip L675.238%−0.906 pp
Swap L4 and L575.898%−0.245 pp
Reverse L4–L675.223%−0.920 pp

Thus, shared recurrence remains useful but saturates earlier. Even in a compact six-layer model, distinct later Blocks provide greater predictive refinement and are neither dispensable nor interchangeable. Similar representations or comparable final accuracy therefore do not establish functional redundancy.

Q2. Clarification of whitening

We thank the reviewer for identifying this labeling error. The reported operation was global normalization: each layer was centered and divided by a single layerwise standard deviation. Because ED depends on relative explained variance, this uniform rescaling leaves ED and its ratios exactly unchanged. We will rename the column “globally normalized.”

We also performed genuine layerwise covariance whitening using SVD, without shrinkage or dimensionality reduction. As expected, it flattened the spectrum, yielding an ED of 180 out of 200 dimensions and expansion/compression ratios of 1.00×/1.00×1.00\times/1.00\times. Per-dimension z-scoring retained the expansion–compression trajectory:

DatasetRawPer-dimension z-scored
segment4.20× / 2.10×3.43× / 1.60×
vehicle4.33× / 2.17×3.50× / 1.56×
balance-scale2.20× / 3.67×2.17× / 1.86×
adult2.38× / 2.21×2.06× / 1.50×

Thus, the identical values in Table 20 follow from scale invariance, while the z-scored results confirm that the trajectory is not caused solely by unequal marginal feature scales.

Reviewer_JFy6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the internal representations of table foundation models (in particular TabPFN-3) as a function of the layer depth. It focuses on the predictive readability at every layer, and characterizes how the geometry of these representations evolves. The main findings are that: 1) the representations follow an “expansion-compression” effective-dimensionality trajectory; 2) the class information is linearly readable early, but the late layers better separate the class candidates; 3) most samples commit to their final prediction early. Later commitment is associated with task difficulty and local input-space ambiguity; 4) repeating or reordering blocks degrades both the effective-dimensionality trajectory and accuracy.

Critical strengths
  1. The empirical characterization is broad and systematic. The analysis spans 272 datasets, and uses several complementary geometric and predictive measurements. The paper provides a useful descriptive overview of TabPFN-3’s internals.
Critical weaknesses
  1. Several central findings are close to tautological or expected a priori. The paper applies an extensive toolbox to TabPFN-3 but rarely turns the measurements into conclusions beyond “depth matters, and more so for harder inputs”.
  • 1.a. The truncation vs. stabilization depth argument (Fig. 7c) is circular. Stabilization depth is defined relative to the L23 prediction. A dataset with a later mean stabilization depth has therefore more samples whose L13 prediction differs from their L23 prediction, by definition.
  • 1.b. Task difficulty vs. commitment depth. Difficulty is defined from final-layer error. Wrongly classified, near-boundary samples are naturally the ones whose argmax flips across layers.
  • 1.c. The paper claims that "depth helps ambiguous samples more”, based on Fig. 1a/7a. However, this seems to be mostly a headroom effect. Indeed, Fig. 1a/7a shows that well-separated samples are already near 100% accuracy at L9, while Fig. 7(b) actually shows that both retention and rescue rates are lower for ambiguous samples. Therefore, it seems that, per-individual prediction, depth is less effective on ambiguous samples, and the larger aggregate gain on Fig. 1a/7a comes only from their larger initial error mass.
  • 1.d. Compression-resolution coupling. Late collapse toward class prototypes would mechanically produce both lower effective dimensionality and larger cosine margins, so their correlation is unsurprising.
  • 1.e. The effect of structural interventions is unsurprising. Repeating one block 24 times, or shuffling or reversing the layers of a frozen pretrained network, sends it far out of distribution, so degradation is expected for essentially any trained deep network.
  1. The contribution reduces to a descriptive study and is not actionable. The paper’s conclusion points to adaptive-depth inference, yet it proposes and evaluates no such method.
Questions for authors
  1. How much of the stabilization-depth / truncation-loss correlation remains after controlling for the fraction of samples whose L13 prediction differs from their L23 prediction?
  2. Can you report a headroom-normalized version of the depth gain, for example net gain as a fraction of L9 errors? Does it still favour ambiguous samples, given that Fig. 7(b) shows lower rescue rates for them?
4 comments
AuthorsOpenReview ↗

Author Response (1/3)

We appreciate your thorough review ! To ensure clarity and conciseness, we have grouped similar points raised in the weakness and question sections and addressed them as follows.

W1. The contribution is not simply that “depth matters more for harder inputs”

We thank the reviewer for this comment. Our contribution goes beyond the observation that harder inputs require more depth. We address a more fundamental and insufficiently explored question: when is inference truly complete for an individual query? Prior studies have primarily examined aggregate representation dynamics, predictive readability, and depthwise redundancy, without systematically resolving this question. Understanding this distinction is essential for determining when additional computation is redundant and when it remains predictively useful.

To address this question, we jointly analyze predictive accessibility, prediction commitment, geometric resolution, and functional utility (Section 3.2, Lines 152–172; Sections 4.3–6.1, Lines 239–414). Our central finding is that these four processes do not coincide and exhibit substantial query-level heterogeneity.

This separation reveals several counterintuitive results. Even among Early-Stable queries whose predictions no longer change after L9, 69.2% of representation displacement occurs after L9 (Appendix G.5, Lines 2034–2095). Meanwhile, independently fitted layer-matched readouts improve accuracy from 82.2% at L2 to 85.6% at L23 (Section 6.1, Lines 405–411; Appendix E.1). Furthermore, some queries remain stably incorrect from shallow layers, whereas others change predictions repeatedly before becoming correct. These observations demonstrate that early predictive readability, prediction stability, and continued representational computation are distinct from the functional benefit of additional depth.

Our contribution is therefore a query-level account of when and why additional depth remains useful, rather than another aggregate observation that harder inputs require deeper computation.

W1.a and Q1. Stabilization depth and truncation sensitivity

We thank the reviewer for this observation and clarify two points. First, stabilization depth is computed from the complete prediction trajectory, not merely L13–L23 disagreement. A query may predict the same class at L13 and L23 while switching at intermediate layers. Indeed, 36.2% of queries stabilizing after L13 have identical L13 and L23 predictions, so the relationship in Figure 7(c) is not circular by definition.

Second, controlling for the L13–L23 disagreement rate D13D_{13} changes the correlation between mean stabilization depth and truncation loss from ρ=0.551\rho=0.551 to ρ=−0.194\rho=-0.194, indicating that the original association primarily reflects the fraction of predictions affected by truncation. To distinguish prediction-change frequency from its utility, we decompose the accuracy gain as ΔAcc13=D13Q\Delta\mathrm{Acc}^{13}=D_{13}Q, where QQ is the net accuracy gain per changed prediction. Of 272 datasets, 254 (93.4%) exhibit positive gains.** Thus, later-stabilizing tasks are more sensitive to truncation mainly because more predictions change, and these changes yield positive net gains on most datasets.

W1.b. Task difficulty does not imply later prediction commitment

We thank the reviewer for this question. Task difficulty is defined by dataset-level final-layer error (Section 5.2, Lines 334–342), whereas stabilization depth is a query-level measure derived from the complete prediction trajectory (Section 5.1, Lines 290–316). A query may remain stably incorrect from shallow layers or change classes later and become correct; final error and late commitment are therefore distinct.

If their association merely reflected error prevalence, it should disappear after fixing the correct/incorrect mixture across datasets. Instead, it remains significant (ρ=0.568, p=8.2×10−21\rho=0.568,\ p=8.2\times10^{-21}), is stronger among final-layer-correct queries (ρ=0.716, p=6.0×10−37\rho=0.716,\ p=6.0\times10^{-37}), and is absent among incorrect queries (ρ=−0.014, p=0.83\rho=-0.014,\ p=0.83).

Moreover, lower local class separation is associated with later stabilization in 271 of 272 datasets, remaining significant even among queries correctly classified at L9, L13, and L23 (mean ρ=−0.236, p=1.1×10−68\rho=-0.236,\ p=1.1\times10^{-68}). These results confirm that later commitment is not simply an artifact of higher error rates.

AuthorsOpenReview ↗

Author Response (2/3)

W1.c and Q2. Headroom alone does not explain the larger gains for ambiguous samples

We thank the reviewer for this important question. However, headroom alone does not explain the larger depth gains for ambiguous samples. Our claim concerns aggregate net accuracy gain; rescue rate alone is not a complete measure of the value of additional depth. Later layers can both correct an incorrect prediction and disrupt a previously correct prediction.

Let eg=1−Acc9,ge_g=1-\mathrm{Acc}_{9,g} denote the L9 error rate, SgS_g the rescue rate, and RgR_g the retention rate. The exact headroom-normalized net gain is

NGRg=Acc23,g−Acc9,g1−Acc9,g=Sg−1−egeg(1−Rg).\mathrm{NGR}_g=\frac{\mathrm{Acc}_{23,g}-\mathrm{Acc}_{9,g}}{1-\mathrm{Acc}_{9,g}}=S_g-\frac{1-e_g}{e_g}(1-R_g).

This measure accounts for both corrected errors and disrupted correct predictions.

To directly answer Q2, we compute the headroom-normalized gain across ten local-separation bins. The gain remains significantly larger for more ambiguous samples (ρ=−0.721, p=0.019\rho=-0.721,\ p=0.019). The most ambiguous group achieves a normalized gain of 18.4%, compared with only 3.2% for the most separated group.

Thus, Figures 7(a) and 7(b) are not contradictory. Ambiguous samples have a lower conditional rescue rate, but they still obtain a larger net benefit from additional depth, even after accounting for their greater initial headroom.

W1.d. Dimensional compression does not necessarily imply class separation

We thank the reviewer for this observation. However, this interpretation equates dimensional compression with collapse toward class prototypes. Effective dimensionality measures how representation variance is distributed across directions (Section 4.1, Lines 178–185), whereas prototype margin measures the separation between competing class candidates (Section 4.3, Lines 239–247). Lower effective dimensionality therefore does not necessarily imply stronger class separation.

Our additional L20–L23 analysis supports this distinction. Among 105 datasets, 31 exhibit late effective-dimensional compression without an increase in prototype margin. If compression necessarily produced class separation, such cases should not occur. Moreover, class information is already highly readable at L2 (Appendix F.3, Lines 1805–1827), while prototype margins become strongest only near the output (Section 4.3, Lines 239–284).

Thus, our finding is not that compression causes class separation, but that predictive accessibility, dimensional compression, and geometric resolution are distinct properties with different trajectories across depth.

W1.e. Structural interventions test

Our experiments examine a more specific question: in a frozen pretrained model, is computation at later depths merely interchangeable repeated computation? If so, maintaining the same computational depth or retaining the same set of Blocks should approximately preserve the original representation trajectory and predictive behavior (Section 6.2, Lines 418–430 and 447–459).

To address the concern that the original interventions may be too strong, we additionally perform localized swaps between five adjacent Block pairs while preserving all Blocks and computation (revised Section 6.2, Lines 455–462). This intervention preserves all Blocks, parameters, model depth, and computational cost, while changing only the execution order of five pairs of adjacent Blocks.

Even so, average accuracy decreases by 2.85 percentage points, and 6.9% of samples change their predicted class. More importantly, this effect is not uniform. Among originally correct predictions, 8.2% and 4.12% of the predictions in the low- and medium-confidence groups, respectively, become incorrect, whereas no correct prediction in the high-confidence group is corrupted (0%). Therefore, average accuracy compresses this heterogeneous effect into a single net change and underestimates the sample-level functional impact caused by changing the learned Block order.

We also skip L10–L16, which lies within the highly CKA-similar L9–L17 region, while preserving all remaining Blocks and their original order (revised Section 6.2, Lines 459–463). Across 72 CC18 datasets, accuracy decreases by 4.77% on average, with a median decrease of 3.38%. The effect is negative on all 72 datasets.

These results show that, in the frozen pretrained model, later computation is not simply repetitive computation, and both Block identity and the learned execution order have practical significance.

AuthorsOpenReview ↗

Author Response (3/3)

W2. Adaptive-depth inference is proposed and evaluated

We thank the reviewer for raising this point. We have also been investigating the practical value of these findings and can now provide the following experimental results. Specifically, we have added a new section, Section 7, “From Mechanistic Findings to Query-Level Decisions” (Lines 470–519), to the revised manuscript, where we experimentally evaluate three concrete applications of our findings: training-free early exit, targeted refinement of unstable queries, and layer-information fusion. Full experimental protocols and supplementary results are provided in Appendix I.1–I.3 (Lines 2390–2461).

1. Training-free early exit using prediction consistency.

Cross-layer prediction agreement provides a label-free signal for adaptive-depth inference. If two designated layers predict the same class, the query exits at the later layer; otherwise, it continues to L23. This requires neither query labels nor an auxiliary decoder.

PolicyExit ruleDepth reductionAccuracyChange
Full depthExit at L230.0%84.80%—
ConservativeL9 = L21 → L217.3%84.56%−0.24 pp
BalancedL9 = L17 → L1722.0%84.13%−0.67 pp
AggressiveL9 = L13 → L1337.4%83.53%−1.27 pp

Across 272 datasets and 752,231 queries, the balanced policy reduces average executed depth by 22.0% with only a 0.67-point accuracy loss, demonstrating a controllable computation–accuracy trade-off (Section 7, Lines 476–484; Appendix I.1, Lines 2390–2409).

2. Identifying and refining hesitant queries using prediction switching.

We define prediction-switch frequency as the number of class changes between consecutive layers:

nch(x)=∑ℓ=1L−11[y^(ℓ)(x)≠y^(ℓ−1)(x)].n_{\mathrm{ch}}(x)=\sum_{\ell=1}^{L-1}\mathbf{1}\left[\hat y^{(\ell)}(x)\neq\hat y^{(\ell-1)}(x)\right].

This label-free measure reveals that queries with more prediction changes generally obtain larger gains from later computation:

Prediction changesQueriesL9 accuracyL23 accuracyCorrectionDisruptionNet gain
0111,71192.5%92.5%0.0%0.0%+0.0 pp
116,04976.8%82.6%11.5%5.8%+5.7 pp
216,01264.0%74.2%17.9%7.7%+10.2 pp
38,97554.2%66.7%23.0%10.5%+12.5 pp
47,94352.3%65.2%26.3%13.4%+12.9 pp
54,25145.6%62.1%27.1%10.6%+16.5 pp
63,62242.2%58.2%29.8%13.8%+16.0 pp
≥74,84533.3%52.5%30.4%11.2%+19.2 pp

The net L9–L23 gain increases from 0.0 pp for queries with no prediction switches to 19.2 pp for those with at least seven switches (Appendix I.2, Lines 2411–2432). We further use the fixed criterion nch≥2n_{\mathrm{ch}}\geq2 to trigger targeted context augmentation, retrieving nearby training examples supporting the competing L9 and L13 predictions.

TriggerQueriesBaselineAugmentedGain
nch≥2n_{\mathrm{ch}}\geq245,64872.41%74.93%+2.52 pp

Without using query labels, this intervention improves accuracy by 2.52 points on triggered queries. As it uses a fixed threshold and requires retrieval and recomputation, we present it as a proof of concept rather than an optimized method (Section 7, Lines 502–513; Appendix I.2, Lines 2434–2442).

These results demonstrate three concrete uses of our mechanistic findings: stop stable queries, refine hesitant queries, and combine complementary predictions across depth. They establish practical value while leaving optimized, fully label-free query-adaptive policies for future work.

AuthorsOpenReview ↗

Kind Reminder from Authors

Thank you once again for your thoughtful feedback and active engagement during the discussion phase! With one day left in the discussion period, we wanted to highlight the revisions it prompted. We (1) clarified the novelty of distinguishing early readability, prediction stability, and depth utility, (2) distinguished full-trajectory stabilization from agreement between two layers, (3) added controls for prediction correctness and initial error headroom, (4) clarified how compression and class separation can diverge, (5) added localized Block interventions, and (6) evaluated training-free early exit against compute-matched fixed-depth truncation.

We hope these updates address your questions and provide a clearer picture of the contributions and significance of our work. If there are any remaining concerns or additional points for discussion, we would be more than happy to engage further before the deadline. In light of these clarifications and improvements, we kindly request your consideration of a higher assessment for our submission. Your feedback has been instrumental in refining this work, and we are grateful for your thoughtful review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Author Response

Dear Reviewers,

We sincerely thank you for your careful reading, valuable feedback, and constructive suggestions. We have substantially revised the manuscript in response, with the new material highlighted in blue.

We first thank the reviewers for highlighting the strengths of our work.

  • The study addresses a meaningful research question and offers a useful query-level perspective on TFM inference. (JFy6, tJWn, 2DRn, R8BU)
  • The empirical evaluation is broad and systematic. (JFy6, tJWn, R8BU)
  • The analysis is comprehensive and draws on complementary perspectives to examine how inference unfolds across layers. (JFy6, tJWn, R8BU)
  • The study reveals a counterintuitive separation among several depthwise processes that are often conflated. (tJWn, 2DRn)
  • The methodology is rigorous, and carefully designed controls strengthen confidence in the conclusions. (tJWn, 2DRn, R8BU)
  • The paper is clearly presented (tJWn, R8BU), and the newly added label-free early-exit analysis strengthens its practical relevance (2DRn, tJWn,R8BU).

We have addressed each reviewer’s questions individually in our detailed responses. We invite the reviewers to consult those responses and welcome any requests for further clarification. In response to the reviewers’ suggestions, we added controls, clarified the scope of our conclusions, expanded the cross-model analyses, and demonstrated how our findings can inform decisions for individual queries.

We have made the following key updates:

  • [Section 2] To clarify what the paper contributes beyond prior work, we sharpened the description of its query-level contribution and expanded the discussion of related studies. (tJWn, 2DRn)
  • [Section 6] To examine Block redundancy and sensitivity to execution order, we tested localized Block swaps and skips in the frozen model. (tJWn)
  • [Section 7] To show how the mechanistic findings can guide inference, we evaluated three query-level applications: training-free early exit, targeted context refinement, and cross-layer fusion. (R8BU)
  • [Appendix C.4] To assess whether the query-level findings generalize across models, we deepened our existing five-model analysis by examining prediction dynamics and depth gains under layer-matched readouts. (tJWn)
  • [Appendix D.3] To test whether shared Blocks can reproduce the gains of distinct Blocks, we compared trained nanoTabPFN models with distinct or shared Blocks and tested interventions on later Blocks. (tJWn)
  • [Appendix E] To clarify the whitening control, we corrected the label for global normalization and added covariance whitening and per-dimension z-scoring analyses. (tJWn)
  • [Appendix H.2] To test whether greater initial error headroom explains the larger depth gains for ambiguous queries, we compared headroom-normalized gains across levels of query ambiguity. (JFy6, tJWn, 2DRn)
  • [Appendix I.2] To assess the value of allocating depth adaptively, we compared agreement-based early exit with fixed-depth truncation at matched average computation on 72 OpenML-CC18 datasets. (2DRn, tJWn)
  • [Figure 2 and accompanying discussion] To address questions about the plateau and its boundaries, we described the middle interval as a relative plateau and explained why the expansion and compression ratios are independent of its boundaries. (2DRn)

In closing, we thank the reviewers again for your time and valuable feedback. If you have further concerns, please let us know, and we will be happy to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.