acceptodds
Under review as a conference paper at ICLR 2027

Thinking in Depth: Retrospective Inference for Tabular Foundation Models

Abstract

Tabular foundation models (TFMs) are pretrained across diverse tabular tasks and make predictions on a new table at inference time using its labeled examples as context. Most recent TFMs perform such in-context prediction with stacked Transformer layers, repeatedly transforming how examples are represented and compared. By tracing individual queries through several strong TFMs, we find that predictive refinement is highly uneven across depth and is often concentrated in later layers. This uneven refinement motivates us to reconsider how intermediate representations are constructed and reused throughout the network. We introduce Retro, a tabular foundation model based on retrospective inference, where later stages can explicitly revisit and recombine intermediate information produced earlier in the network. Retro organizes this process around two complementary operations: which intermediate information to revisit, and how the resulting contextual update should be shaped for each query. Attention Residuals address the former by adaptively reweighting contributions from different depths, while query-conditioned Gated Attention addresses the latter by modulating the attention output element-wise across representation dimensions. Our analysis shows that Retro shifts predictive refinement earlier and more broadly across depth, with different stages revising different subsets of queries in a pattern suggestive of multi-view refinement. Across TabArena, TALENT, and RelArena, Retro ranks among the top three and lies on the Pareto frontier. These results indicate that directly reusing intermediate representations provides a practical way to better exploit depth in TFMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular Foundation Models, Tabular Deep Learning, Structured Data

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_s2ZVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how sample-level predictions evolve across the depth of tabular foundation models and proposes RETRO, a TabICLv2-based architecture for retrospective reuse of intermediate computation. The motivating analysis applies a frozen linear readout, fitted on final-layer cross-fitted support representations, to intermediate representations from several TFMs. Under this diagnostic, predictive changes are unevenly distributed across depth and are often concentrated in later layers. RETRO modifies the row-level in-context learning stack through two components: block Attention Residuals, which selectively aggregate historical groups of residual updates, and query-conditioned Gated Attention, which modulates attention outputs across representation dimensions. The paper reports earlier and more distributed prediction changes in retrospective variants, alongside competitive native-head performance: RETRO ranks third, second, and first by overall Elo on TabArena, TALENT, and RelArena, respectively, within the evaluated comparison pools.

Critical strengths

1.The paper studies a meaningful question beyond aggregate predictive performance.

2.The architectural design is coherent and practically motivated.The architectural design is coherent and practically motivated.

3.The evaluation covers multiple task settings and includes additional mechanistic experiments.

Critical weaknesses

W1.The connection between the observed depth dynamics and downstream performance remains insufficiently established. Section 4 measures predictions under a readout trained exclusively on final-layer representations and then frozen across depth. Consequently, the diagnostic measures compatibility with a particular final-layer decoder, rather than the intrinsic predictive information in each layer or the usefulness of that information to subsequent computation. Changes in these trajectories could reflect changes in readout alignment without demonstrating that intermediate representations have become more informative or more useful for the final task. The authors acknowledge this limitation in Appendix A.5, but the analysis still leaves a substantial gap between the motivating observation and the proposed explanation for the model’s performance.

W2.The component ablation and its training controls are insufficiently specified. Figure 6 is briefly discussed in the “Component ablation” paragraph of Section 6, but the discussion primarily summarizes the Elo point estimates. I could not determine from this description whether all four configurations were independently pretrained with their respective components, obtained through inference-time removal of components from a trained model, or produced through some other procedure. Appendix D.3 explicitly describes separately trained gated and ungated models, but it does not clearly establish how that comparison maps onto the complete four-way ablation in Figure 6.

W3.The Elo results do not sufficiently resolve statistical robustness or dataset-level improvements.

Questions for authors

Q1.Does the architectural benefit generalize beyond the TabICLv2 backbone? Attention Residuals and Gated Attention are adopted from prior work, and the paper also notes an existing application of Attention Residuals in tabular modeling. Therefore, evidence that this combination transfers beyond the TabICLv2-based implementation would strengthen the TFM-specific contribution. Could the authors evaluate the components on at least one additional backbone, such as TabPFN-3, TabFM, or LimiX-2? These are proposed extensions, not architectures on which the submission has demonstrated the benefit of the modifications. In particular, LimiX-2 offers an additional architecture to consider outside the models modified in this paper.

Q2. Can the authors provide a matched, independently retrained TabICLv2-vanilla baseline? Could RETRO and an unmodified TabICLv2 baseline be trained under the same synthetic-data generation procedure, training budget, optimizer, curriculum, and checkpoint-selection protocol, preferably with multiple seeds? Please also report the released TabICLv2 checkpoint separately, so that practical competitiveness and controlled architectural improvement can be assessed independently. In my own reproduction attempt, retrained TabICLv2 underperformed the released checkpoint. This observation does not establish a problem with the authors’ training, but it makes the distinction between a released-checkpoint comparison and a matched-training comparison particularly important. Does the “Neither” configuration in Figure 6 already provide this control? If so, please document its training protocol and its relationship to the TabICLv2 baseline in the main benchmark tables.

Q3. Can the authors provide runnable code and the actual model checkpoints for verification?

3 comments

Thank you for your specific comments on the problem formulation, architectural organization, and evaluation coverage. Our code and model weights are now available at https://anonymous.4open.science/r/Retro/, where we have also added several new visualizations.

W1: Section 4 measures predictions under a readout trained exclusively on final-layer representations and then frozen across depth.

We acknowledge that this is an important issue. Please refer to our response to Reviewer 4AEs W1.

W2/Q2: The component ablation and its training controls are insufficiently specified. Can the authors provide a matched, independently retrained TabICLv2-vanilla baseline? Please also report the released TabICLv2 checkpoint separately.

We have added detailed explanations and further results for the ablation study. To clarify, Fig. 6 in the paper uses Stage 2 model weights obtained under the same training configuration for the ablation study. We have further added Retro Stage 3, our reproduction of TabICLv2 Stage 3, and the official TabICLv2 checkpoint to this comparison, and report Elo scores, confidence intervals, and pairwise win rates.

IDModelELO95%
ARetro (Stage 3 final)1113.51057.8–1164.1
BRetro (Both) (Stage 2)1057.81026.2–1091.1
CTabICLv2 (Default)1046.31005.5–1081.8
DTabICLv2 (Ours stage 3)1037.7996.1–1076.6
EAttnRes only (Stage 2)1018.4966.5–1064.3
FGated only (Stage 2)912.0870.6–954.7
GNeither (Our TabICLv2 stage 2)814.2745.7–875.6
ABCDEFG
A—59.5961.2160.4463.4275.6982.22
B40.41—52.6655.7652.7272.0578.76
C38.7947.34—54.0254.2771.0776.17
D39.5644.2445.98—54.9771.9676.92
E36.5847.2845.7345.03—64.9975.83
F24.3127.9528.9328.0435.01—72.87
G17.7821.2423.8323.0824.1727.13—

W3: The Elo results do not sufficiently resolve statistical robustness or dataset-level improvements.

The Elo results and their corresponding confidence intervals provide some indication of statistical robustness. TabArena uses 200 task-level bootstrap replicates to compute Elo, RelArena uses 100 task-level bootstrap replicates, and TALENT uses 30 random permutations of dataset order. We further provide win rates between Retro and mainstream methods on each benchmark in Fig. G, Fig. H, and Fig. I in the anonymous repository. We also provide a per-dataset comparison with TabICLv2 on TabArena classification tasks in Fig. J.

Q1: Does the architectural benefit generalize beyond the TabICLv2 backbone?

This is an important question and one we would like to investigate further. However, the rebuttal period is insufficient to fully train a TFM with a new architecture, and we currently have no experience reproducing architectures other than TabICLv2. We therefore plan to investigate this question in future work.

Q3: Can the authors provide runnable code and the actual model checkpoints for verification?

Our model weights and inference code are available in the anonymous repository at https://anonymous.4open.science/r/Retro/.

I appreciate the authors’ responses and clarifications, which have addressed my concerns. Based on the rebuttal, I have decided to raise my score from 2 to 3.

Thank you very much for carefully considering our clarification and updating your assessment. We sincerely appreciate your time.

Reviewer_vxizOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how tabular foundation models (TFMs) refine their predictions across transformer depth. Through layer-wise analysis of TabICLv2, TabPFN-3, and EXAONE-Tabular, the authors observe that prediction changes are unevenly distributed across layers and often concentrated toward the end of the network.

Motivated by this observation, the authors propose retrospective inference, an architectural approach that allows later layers to revisit and selectively combine intermediate representations from earlier computation stages rather than relying exclusively on the most recently accumulated hidden state. The proposed model, RETRO, incorporates two mechanisms: Attention Residuals, which dynamically aggregate contributions from different depths using input-dependent weights, and Gated Attention, which applies query-conditioned, channel-wise modulation to contextual attention outputs. RETRO builds upon the feature-to-row encoding and in-context learning architecture of TabICLv2. RETRO obtains competitive rankings across these benchmarks, including third place on TabArena, second on TALENT, and first on RelArena in overall Elo among the evaluated methods.

Critical strengths
  1. Detailed investigation of inference dynamics across transformer depth. The paper goes beyond aggregate predictive performance and investigates how individual predictions evolve across layers in several strong TFMs. The analyses include changes in prediction margins, correctness transitions, depth-wise representation visualizations etc.

  2. Broad empirical evaluation and useful component-level interventions. RETRO is evaluated across three benchmark suites covering classification, regression, and relational tasks, with competitive performance against several contemporary foundation models. The paper additionally evaluates the contributions of Attention Residuals and Gated Attention and provides targeted interventions investigating the geometry of contextual updates. In particular, replacing the learned update direction with one obtained from an averaged gate produces substantially greater degradation than replacing only its magnitude, suggesting that channel-wise gating influences how contextual information is incorporated. These experiments provide useful evidence that the proposed components affect the trained model's computational behaviour. The relatively small additional parameter cost and reported inference-time competitiveness further support the practical relevance of the approach.

Critical weaknesses
  1. Although Figure 6 includes four component configurations, the results do not fully isolate whether adaptive historical selection is necessary compared with simpler residual aggregation approaches, such as fixed weighted skips, uniform aggregation of intermediate representations, or other parameter-matched dense residual connections. More importantly, the combined model achieves a TabArena classification Elo of 1115.6 compared with 1093.1 for Attention Residuals alone in the component ablation. This relatively modest incremental improvement warrants a more direct statistical assessment of whether Gated Attention provides consistent additional benefits when retrospective aggregation is already present.

  2. The manuscript would be strengthened by controlled comparisons against simpler historical aggregation strategies under matched parameter counts and training budgets. Such experiments are important for distinguishing the value of the proposed retrospective inference formulation from the benefits of combining already established architectural components.

Questions for authors
  1. can the authors compare RETRO with simpler alternatives such as fixed learned depth weights, or parameter-matched dense residual connections? In particular, do input-dependent weights provide a consistent advantage over static weighting? This would help establish whether adaptive retrospective selection is necessary for the reported improvements.
3 comments

Thank you for your careful assessment. Our code and model weights are now available at https://anonymous.4open.science/r/Retro/, where we have also added several new visualizations.

W1(a): The results do not fully isolate whether adaptive historical selection is necessary compared with simpler residual aggregation approaches, such as fixed weighted skips, uniform aggregation of intermediate representations, or other parameter-matched dense residual connections.

We agree that the existing comparisons mainly assess the contributions of Attention Residuals and gating. Comparisons with fixed learnable depth weights, uniform aggregation, or other dense residual connections would help further clarify the benefits of adaptive historical selection. However, our current computational resources are insufficient to complete these comparisons within the rebuttal period. We therefore leave these comparisons for future work.

Fig. F in the anonymous repository shows the mean and variance of the trained model’s weights for each historical source. In this analysis, the model exhibits clear preferences associated with the read site and depth, often placing greater weight on the initial ICL encoding and the current update group. This shows that the model does use non-uniform historical reads. We chose Attention Residuals because it is already a relatively established technique and provides an explicit, learnable path for historical access with few additional parameters, which closely matches our intended idea of retrospection.

W1(b): Whether Gated Attention provides consistent additional benefits when retrospective aggregation is already present?

In Fig. 6 of the original paper, adding Gated Attention to a model that already uses Attention Residuals increases Elo by only about 20 points, with a corresponding win rate of 52.72%. This incremental improvement is indeed limited, but it is likely underestimated when more comparison methods are considered. As discussed in our response to Reviewer s2ZV W2, a model using only Attention Residuals does not yet outperform TabICLv2 on TabArena classification at Stage 2, whereas adding Gated Attention allows it to do so. In this comparison, the Elo gap between the two models increases to around 40 points. Meanwhile, as analyzed in the paper, we view retrospective inference as having two parts: how to revisit historical information and how to incorporate historical information, corresponding to Attention Residuals and Gated Attention, respectively. We therefore consider this design both empirically supported and consistent with our design intuition, particularly given its very small additional overhead.

W2/Q1: The manuscript would be strengthened by controlled comparisons against simpler historical aggregation strategies under matched parameter counts and training budgets.

We agree that these controls are valuable for isolating the mechanism. We plan to add these comparative experiments in a subsequent update and include them in the appendix.

I have read the author's rebuttal. I will keep the score.

Thank you for taking the time to read our rebuttal and for your thoughtful feedback on our work!

Reviewer_4AEsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how retrospection (a concept well studied for LLM) impacts tabular foundation models. For that they implement residual attention (the model can look at representations of earlier layers) and a gating mechanism (the model can learn how to combine earlier representations). Empirical analysis shows that both (residual attention and gating) improved performance in ablation and that the final model perform en-par with TabICLv2.

Critical strengths

I really enjoyed reading the paper, as it is well-written and (mostly) human-readable. It introduces a novel concept for TFMs and, in doing so, pushes our understanding of how to handle tabular predictions with TFMs.

Critical weaknesses

I have a few concerns that require discussion and justify my current score:

W1: Method used to test the main assumption. The main motivation builds on the assumption that (in current TFMs) predictions form in the last layers (and not continuously throughout the forward pass as in RETRO). However, evaluating a probe (trained on the last layer) on all layers does study whether representations are predictive and align with the final representation space. Prior work (Balef et al. (2026)) showed that representations are predictive early on, but do not align with the final layer. --> The main effect of the approach may be that the model now operates in the same space throughout the forward pass (as shown in Figure 2) rather than actual performance gains from retrospection.

W2: Justification for the approach. Retrospection and the gate mechanism are well studied for LLMs, where we know that layer specialization is important. For TFMs, this is the first (as far as I am aware) work studying these approaches, so this itself is novel enough and warrants exploration. However, to justify this as a beneficial mechanism for tabular predictions, some evidence that layer specialization emerges in TFMs would be a convincing motivation (e.g., do the mechanisms illustrated in Figure 2 actually emerge) --> I am not convinced that this approach fixes a critical issue in existing models.

W3: Ablation and Comparison to TabICLv2. RETRO performs competitively and even slightly better than other models for regression tasks. Since RETRO uses the same backbone, prior and training as TabICLv2, the comparison to TabICLv2 is the most interesting. In the ablation in Figure 6, both features (gating and retrospection) increase performance over the "neither" baseline. However, this performance difference does not generalize to an improvement over TabICLv2 (e.g., ELO scores on TabArena classification: RETRO performs similarly to TabICLv2 (Table 3 in App)). To me, it is unclear whether the results show that this architecture solves tabular tasks as well as other architectures or whether this approach increases efficiency in some dimension (data, weights, memory, inference costs, etc.). --> When would one prefer to use RETRO over "neither" or TabICLv2?

Questions for authors

Q1/W1: (see above) Did I overlook anything?

Q2/W2: (see above) Did I overlook anything? Measuring the differences across tasks would be interesting and provide further evidence (and also raw results for the Ablation in Figure 6; see next question)

Q3/W3: (see above) I want to emphasize that this is not about state-of-the-art performance; I want to understand what we learn here. Can you report the raw scores comparing "neither" with TabICLv2 (since the methods in the comparison affect ELO)? Is this difference significant? Are there other confounding factors in comparing "neither" and TabICLv2 that I have not considered?

4 comments

Thank you for carefully reading our paper and recognizing our investigation into the internal prediction process of tabular foundation models. Our code and model weights are now available at https://anonymous.4open.science/r/Retro/, where we have also added several new visualizations.

W1/Q1: The main effect of the approach may be that the model now operates in the same space throughout the forward pass rather than actual performance gains from retrospection.

We acknowledge that this is one of the most important issues raised in the reviews. We find that our current experimental design is indeed affected, to some extent, by representation alignment across layers. We have added alternative visualizations and plan to revise some statements in the main text.

Let us first explain why we designed the experiments this way and address your concern.

  1. We consider both layer-specific probes and a shared probe to be ways of presenting the results, but with different perspectives. Our perspective is that the residual update at each layer performs refinement, so a shared classification head can continuously show how representations are refined.
  2. Beyond revealing representation separability, we believe that layer-specific probes are strongly affected by regularization, sample size, and generalization. Fig. B in the anonymous repository shows the results using layer-specific probes. Across 38 classification datasets, TabICLv2 achieves an average accuracy of 86.99% at the first layer—as you noted, separability is already present in early layers—and 87.28% at the final layer. Taking randomness into account, the variance would be larger than the changes across layers. We therefore consider layer-specific probes suitable for analyzing early representation separability, but less suitable for analyzing layer dynamics.
  3. Balef et al. (2026) analyzed TabPFN 2.5 and LimiX-16M, both of which alternate row-wise and column-wise attention, and found that TFM representations are not necessarily aligned with the original decoder. We find that models such as TabICLv2, which first extract row embeddings and then perform only row-wise ICL, exhibit much better representation alignment. Applying TabICLv2’s orginal pretrained classification head to its first-layer representations yields an average accuracy of 75.70% across 38 datasets. Compared with its final average accuracy of 88.66%, we consider these representations already reasonably aligned and capable of directly producing effective predictions. Fig. E in the anonymous repository shows the layer-wise results of TabICLv2 and our model using their original prediction heads, which are more consistent and stable than those shown in Balef et al. (2026).
  4. We believe that the tendency toward representation alignment in these TFMs mainly comes from the original residual connections, rather than from our Attention Residuals further separating the information from each layer. When our model in this comparison also directly uses its original prediction head, its accuracy is 84.05% at the first layer and 88.71% at the final layer. Fig. C and Fig. D in the anonymous repository visualize these results. Our model still exhibits greater differentiation across layers than TabICLv2; please see our response to W2 below for details. One possible reason why our model’s first layer is better aligned with its original classification head is that later layers assign greater retrieval weights to the first layer, which contains the most information, as shown in Fig. F.
  5. We found that the above visualizations were still not ideal, because the original prediction head already performs well at shallow layers. Based on our understanding of representation alignment, we fitted a shared linear layer on the final-layer representations as a regularized output head. This does show more gradual changes in performance, but it is also affected by representation misalignment. We plan to explicitly present this figure as a pilot study instead of a formal evidence.

We recognize the importance of this issue and welcome your further thoughts. In the revision, we will emphasize the rationale for choosing a shared probe rather than layer-specific probes, add the visualization using the original pretrained prediction head (Fig. C), and explicitly identify the current Fig. 3 as an illustrative pilot study.

W2/Q2: How to prove layer specialization?

We have added quantitative analysis results. Independently of the prediction head used for probing, we follow Balef et al. (2026) and use CKA to analyze similarity across layers. The average difference between adjacent layers (1−CKA) is 0.0282 for TabICLv2 and 0.0830 for our model, indicating greater differences between adjacent layers in our model. The average difference across all distinct layer pairs is 0.1207 for TabICLv2 and 0.1991 for our model, indicating that the differences extend beyond adjacent layers to layer pairs across the network. When considering only the early and middle layers, L1–L9, the average difference decreases to 0.0248 for TabICLv2 but increases to 0.0915 for our model. This indicates that representation changes in our model are more active in the early and middle layers, whereas TabICLv2 shows the opposite pattern, consistent with the conclusions in our paper.

We also compute a statistic based on singular-value spectral entropy. The mean effective rank across layers is 159.46 for TabICLv2 and 134.52 for our model. Taken together, our model does not necessarily span more effective dimensions at each layer, but its representation structure changes more noticeably across layers, with less repetition across layer representations. This is similar to each layer of our model focusing on a smaller local region. We will include these quantitative results in the revision.

We also originally intended Fig. 3, now to be presented as a pilot study, to illustrate layer differentiation more specifically: on the right, our model improves the margins of different classes at L3/L6/L9, respectively. This is similar to the illustration in Fig. 2, where our model seeks to focus on and make better predictions in different neighborhoods at different layers.

W3/Q3: What can we learn from Retro?

We have added further explanations and results for the ablation study; please see our response to Reviewer s2ZV W2. We have also provided an efficiency analysis; please see our response to Reviewer Nap4 Q4. Some characteristics of Retro can be summarized as follows:

  1. After completing only the first two training stages, Retro can outperform TabICLv2, which completes all three stages, on the TabArena classification benchmark. We believe this demonstrates its potential to achieve better performance and parameter efficiency with less data and computation. This may be because Attention Residuals improve the network’s learnability while introducing almost no additional parameters.
  2. Retro’s peak GPU memory usage during inference remains completely unchanged relative to TabICLv2, while its inference time without any optimization is approximately 1.14 times that of TabICLv2. This makes the proposed architecture a low-cost improvement over TabICLv2.
  3. Retro is notably more competitive on regression tasks, suggesting that this architecture and the idea of retrospective inference may be better suited to regression tasks with continuous targets.

Thank you for your effort to resolve concerns about the main assumption in your paper.


Some comments on your response to my concern about the lack of justification that "refinement concentrated in late layers" actually happens and is an issue worth solving for TFM inference.

  1. Ok.
  2. A linear probe on TabICLv2 shows that representations are separable after the first ICL layer. This suggests that the representation is already rich enough to make predictions and that later layers contribute negligibly to separability. This also suggests that the ICL layers are not effective (or not crucial to solve the task), rather than supporting the main claim.
  3. Regarding the comparison to Balef et al (2026), I'd compare to the results on TabICLv1 (rather than TabPFN and Limix), which has a similar architecture and for which the final-layer decoder performs more similarly and steadily across layers (see Figure 6 in Balef et al.), actually supporting your findings.
  4. Ok.
  5. Ok.

Overall, I agree that studying layer alignment and decodability requires a careful experimental setup and interpretation. For a future revision, I also recommend studying fine-tuning the final decoder per layer (tabular logit lens) to compare the performance of a decoder with the same expressivity as the original one. While all of the experiments are valid, I don't see how they further clarify my main concern.

Q2: This is very interesting, and I would agree to add this to the paper. However, I'd like to note that this (similarly to the illustration in Figure 2) seems highly speculative.

Q3: My question was about comparing "neither" and TabICLv2 (which I expect should perform similarly) to estimate the expected performance gain of RETRO over TabICLv2 (or to establish that we do not expect a performance gain). Additionally, I agree it is interesting that RETRO performs better for regression, but without an explanation or intuition for why this should be the case, I am not convinced that this performance gain comes from the method (and not other confounding factors).


Having thought about the paper more, I'd like to emphasize that I do see value in continuing research in this direction. Also, the rebuttal contains several interesting starting points for further explanation and intuition, but also requires substantial further experimental analysis (beyond the scope of a short rebuttal and light edits). Most importantly, the authors wrote that they "plan to revise some statements in the main text.", which is too vague to assess whether a revised version is sufficient for publication.

While I think the rebuttal led to progress in understanding how this method works and how it impacts inference, overall, the paper in its current stage (taking into account that the promised updates are substantial) is still below conference standards, so my overall recommendation, unfortunately, remains negative.

Thank you for your careful review and for clarifying your concerns about the central assumption and the ablation comparisons. We plan to make the following revisions to the paper:

  1. In the revised main text, we will emphasize why we chose a shared probe over the more commonly used layer-specific probes. In the appendix, we will discuss the related work by Balef et al. and include evidence that the ICL representations in TabICLv2 are already reasonably well aligned with the original prediction head (Fig. E). We will frame the motivation for Retro as a directly testable question: Can later layers improve the final prediction by revisiting representations from earlier layers and selectively incorporating them?
  2. We will add the visualization using the original pretrained prediction head (Fig. C) to the appendix. In the main text, we will clarify that the current Fig. 3 uses a linear layer refitted on the final-layer representations for illustration and serves primarily as a pilot study. We appreciate your suggestion to fine-tune the original decoder at each layer and agree that this direction is worth exploring.
  3. In our response to Reviewer s2ZV's W2, we explained that our ablation study used Stage 2 model weights. We also compared the ablation results with those of Stage 3 Retro, our reproduction of TabICLv2, and the official TabICLv2 checkpoint, and clarified that Neither is the Stage 2 checkpoint of our TabICLv2 reproduction. The pairwise win rates of our final Retro model against our reproduction of TabICLv2 and the official TabICLv2 checkpoint are 60.44% and 61.21%, respectively. In the revision, we will update the full ablation study to use Stage 3 model weights and report Elo scores and win rates for the individual comparisons, as well as Elo scores computed across a common set of methods.
  4. We will include the quantitative results from our response to W2 in the main text as evidence of differences between layers and clarify that layer specialization is one possible interpretation.
  5. In the appendix, we will report how training time, inference time, and peak GPU memory usage differ between Retro and TabICLv2. We will also include per-dataset performance comparisons and identify which datasets account for the gains on classification and regression tasks, respectively.

Thank you again for your thoughtful feedback. We’d be happy to address any further concerns.

Reviewer_Nap4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RETRO, a TabICLv2-based model which incorporates two changes from the LLM literature: 1) block attention residuals (Kimi Team, 2026); and 2) query-conditioned sigmoid gated attention (Qiu et al., 2025). These changes are motivated by a layerwise analysis showing that in current table foundation models, predictive refinement is uneven across depth and often concentrated in late layers. Among the evaluated models, RETRO ranks 3rd on TabArena, 2nd on TALENT and 1st on RelArena by Elo, with inference time close to TabICLv2.

Critical strengths
  1. RETRO is competitive across three established benchmarks, at moderate compute cost, which is practically valuable to the community, especially if released as planned.

  2. The modifications brought to TabICLv2 are simple and easy to implement. The component ablation (Fig. 6) suggests that attention residuals carry most of the gain at almost no parameter cost. This makes the work a good starting point for follow-ups on depth-wise information flow in TFMs.

Critical weaknesses

1. The novelty of the contributions is limited.

  • Both modifications brought to TabICLv2 are borrowed from the LLM literature: Attention Residuals come from Kimi Team (2026) and gated attention from Qiu et al. (2025). The rest of the model retains TabICLv2’s architecture, prior and optimization setup.

2. The analysis does not convincingly show that RETRO’s mechanism explains its performance gains.

  • The layerwise diagnostic is biased by construction. Both the readout and the PCA basis are fitted on final-layer representations. In a standard residual stack, intermediate states are naturally less aligned with this final-layer geometry. So “refinement concentrated in late layers” may be an artifact of the experimental design.

  • The gating analysis does not test whether gating helps. The direction/magnitude interventions (Fig. 5) replace gates in a model trained with gating, which pushes it out of distribution.

  • Figs. 3 and 4 show that RETRO switches predicted classes much more often across depth than TabICLv2. The paper provides no evidence that this volatility/instability is actually useful.

  • The comparison with TabICLv2 is not training-controlled. The authors themselves state that “shared pretraining data, optimization, and initialization are not established” between their variant and TabICLv2 (App. A.1).

  • The RelArena protocol is undocumented. RETRO is a single-table model, but the paper does not explain how it was applied to relational data: which flattening or relational encoder, which context construction? TabICLv2 is also absent from the RelArena results. Without TabICLv2 under the same relational pipeline, the first place on RelArena cannot be attributed to RETRO’s architecture rather than to the pipeline.

Questions for authors
  1. How was RETRO applied to RelArena: which relational encoder or flattening, context sampling and inference configuration? Why is TabICLv2 not reported under the same pipeline? Which base TFM does the RDBLearn entry use?

  2. Can you report a TabICLv2-architecture model retrained with exactly RETRO’s pretraining pipeline and budget?

  3. Do the conclusions of Sec. 4 and Figs. 3-4 hold when using layer-specific probes instead of a final-layer readout and PCA basis?

  4. What is RETRO’s peak GPU memory at inference and during training, relative to TabICLv2, for large tables (e.g., 60k rows), given the retained sources and FP32 reads?

1 comment

Thank you for specifically identifying the shortcomings in the mechanistic explanation, training controls, and evaluation on relational data. Our code and model weights are now available at https://anonymous.4open.science/r/Retro/, where we have also added several new visualizations.

W1: The novelty of the contributions is limited.

Thank you for recognizing the practical value and simplicity of the implementation. We agree that Attention Residuals and Gated Attention themselves come from prior work. The paper does not claim to have invented these two modules and already discusses the related application in Xiaomi-TabLDM. The question we seek to answer is how explicitly retaining and selecting historical updates affects query-level prediction trajectories in tabular in-context prediction, and whether this organization yields reproducible gains in the performance–cost trade-off.

W2/Q3: "Refinement concentrated in late layers" may be an artifact of the experimental design. Do the conclusions hold when using layer-specific probes instead of a final-layer readout and PCA basis?

We acknowledge that this is an important issue. Please refer to our response to Reviewer 4AEs W1.

W3: The gating analysis does not test whether gating helps.

We acknowledge that Fig. 5 tests a trained model’s dependence on learned gated updates, rather than the benefit of including gates from the start of training. §5.2 and D.1 already specify this scope, along with the limitations concerning unmatched perturbation magnitudes and normalization. We will distinguish these intervention results more clearly from the architectural ablation. The additional benefit of gating is addressed by comparing the performance of Both and Attention Residuals only under the same training conditions.

W4: Figs. 3 and 4 show that RETRO switches predicted classes much more often across depth than TabICLv2. The paper provides no evidence that this volatility/instability is actually useful.

We agree that more frequent switching is not inherently more useful, so we will not use “more switching” to directly explain the final performance gains. Our narrative centers on the idea that “TFMs need to revisit information” and that “revisiting information and layer specialization inform each other.” Please also refer to our response to Reviewer 4AEs W2.

W5/Q2: The comparison with TabICLv2 is not training-controlled.

We have provided detailed explanations and additional ablation results, including the official TabICLv2 checkpoint and the results of our full reproduction of TabICLv2. Please see our response to Reviewer s2ZV W2 for details.

W6/Q1: The RelArena protocol is undocumented.

We evaluate on RelArena using exactly the same protocol as TabPFN-Rel. We will clarify this in the paper.

Q4: What is RETRO’s peak GPU memory at inference and during training, relative to TabICLv2, for large tables (e.g., 60k rows), given the retained sources and FP32 reads?

We conducted the tests on a single A800 GPU with 80 GiB of memory. For inference, we analyzed APSFailure, the largest task in TabArena, which contains 76,000 rows and 170 features. Retro’s peak GPU memory usage during inference does not increase at all relative to TabICLv2, remaining at 46.09 GiB. This is because the inference memory bottleneck lies in the Column Encoder, where Retro introduces no additional overhead. Meanwhile, without any optimization, Retro’s inference time is approximately 1.14 times that of TabICLv2.

Inference stageTabICLv2 peak GPU memoryRetro peak GPU memory
Column Encoder46.08 GiB46.09 GiB
Row Encoder / Aggregation33.39 GiB33.40 GiB
ICL1.33 GiB5.48 GiB

For training, we tested on training data with 60,000 rows and 100 numerical features. TabICLv2’s peak GPU memory usage during training is 43.90 GiB, while Retro’s is 48.47 GiB, an increase of 10.4%. TabICLv2 takes 5.57 s per step, while Retro takes 6.40 s, approximately 1.15 times as long. Taken together, these results make the two lightweight architectural designs proposed in Retro a low-cost improvement over TabICLv2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.