- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how sample-level predictions evolve across the depth of tabular foundation models and proposes RETRO, a TabICLv2-based architecture for retrospective reuse of intermediate computation. The motivating analysis applies a frozen linear readout, fitted on final-layer cross-fitted support representations, to intermediate representations from several TFMs. Under this diagnostic, predictive changes are unevenly distributed across depth and are often concentrated in later layers. RETRO modifies the row-level in-context learning stack through two components: block Attention Residuals, which selectively aggregate historical groups of residual updates, and query-conditioned Gated Attention, which modulates attention outputs across representation dimensions. The paper reports earlier and more distributed prediction changes in retrospective variants, alongside competitive native-head performance: RETRO ranks third, second, and first by overall Elo on TabArena, TALENT, and RelArena, respectively, within the evaluated comparison pools.
Critical strengths
1.The paper studies a meaningful question beyond aggregate predictive performance.
2.The architectural design is coherent and practically motivated.The architectural design is coherent and practically motivated.
3.The evaluation covers multiple task settings and includes additional mechanistic experiments.
Critical weaknesses
W1.The connection between the observed depth dynamics and downstream performance remains insufficiently established. Section 4 measures predictions under a readout trained exclusively on final-layer representations and then frozen across depth. Consequently, the diagnostic measures compatibility with a particular final-layer decoder, rather than the intrinsic predictive information in each layer or the usefulness of that information to subsequent computation. Changes in these trajectories could reflect changes in readout alignment without demonstrating that intermediate representations have become more informative or more useful for the final task. The authors acknowledge this limitation in Appendix A.5, but the analysis still leaves a substantial gap between the motivating observation and the proposed explanation for the model’s performance.
W2.The component ablation and its training controls are insufficiently specified. Figure 6 is briefly discussed in the “Component ablation” paragraph of Section 6, but the discussion primarily summarizes the Elo point estimates. I could not determine from this description whether all four configurations were independently pretrained with their respective components, obtained through inference-time removal of components from a trained model, or produced through some other procedure. Appendix D.3 explicitly describes separately trained gated and ungated models, but it does not clearly establish how that comparison maps onto the complete four-way ablation in Figure 6.
W3.The Elo results do not sufficiently resolve statistical robustness or dataset-level improvements.
Questions for authors
Q1.Does the architectural benefit generalize beyond the TabICLv2 backbone? Attention Residuals and Gated Attention are adopted from prior work, and the paper also notes an existing application of Attention Residuals in tabular modeling. Therefore, evidence that this combination transfers beyond the TabICLv2-based implementation would strengthen the TFM-specific contribution. Could the authors evaluate the components on at least one additional backbone, such as TabPFN-3, TabFM, or LimiX-2? These are proposed extensions, not architectures on which the submission has demonstrated the benefit of the modifications. In particular, LimiX-2 offers an additional architecture to consider outside the models modified in this paper.
Q2. Can the authors provide a matched, independently retrained TabICLv2-vanilla baseline? Could RETRO and an unmodified TabICLv2 baseline be trained under the same synthetic-data generation procedure, training budget, optimizer, curriculum, and checkpoint-selection protocol, preferably with multiple seeds? Please also report the released TabICLv2 checkpoint separately, so that practical competitiveness and controlled architectural improvement can be assessed independently. In my own reproduction attempt, retrained TabICLv2 underperformed the released checkpoint. This observation does not establish a problem with the authors’ training, but it makes the distinction between a released-checkpoint comparison and a matched-training comparison particularly important. Does the “Neither” configuration in Figure 6 already provide this control? If so, please document its training protocol and its relationship to the TabICLv2 baseline in the main benchmark tables.
Q3. Can the authors provide runnable code and the actual model checkpoints for verification?
3 comments
I appreciate the authors’ responses and clarifications, which have addressed my concerns. Based on the rebuttal, I have decided to raise my score from 2 to 3.
Thank you very much for carefully considering our clarification and updating your assessment. We sincerely appreciate your time.
Thank you for your specific comments on the problem formulation, architectural organization, and evaluation coverage. Our code and model weights are now available at https://anonymous.4open.science/r/Retro/, where we have also added several new visualizations.
We acknowledge that this is an important issue. Please refer to our response to Reviewer 4AEs W1.
We have added detailed explanations and further results for the ablation study. To clarify, Fig. 6 in the paper uses Stage 2 model weights obtained under the same training configuration for the ablation study. We have further added Retro Stage 3, our reproduction of TabICLv2 Stage 3, and the official TabICLv2 checkpoint to this comparison, and report Elo scores, confidence intervals, and pairwise win rates.
The Elo results and their corresponding confidence intervals provide some indication of statistical robustness. TabArena uses 200 task-level bootstrap replicates to compute Elo, RelArena uses 100 task-level bootstrap replicates, and TALENT uses 30 random permutations of dataset order. We further provide win rates between Retro and mainstream methods on each benchmark in Fig. G, Fig. H, and Fig. I in the anonymous repository. We also provide a per-dataset comparison with TabICLv2 on TabArena classification tasks in Fig. J.
This is an important question and one we would like to investigate further. However, the rebuttal period is insufficient to fully train a TFM with a new architecture, and we currently have no experience reproducing architectures other than TabICLv2. We therefore plan to investigate this question in future work.
Our model weights and inference code are available in the anonymous repository at https://anonymous.4open.science/r/Retro/.