- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper adds recursive reasoning to tabular foundation models (such as TabPFN), in the style of looped Transformers and the Tiny Recursive model. They make two contributions.
- Experimentally, they show that at matched pretraining budget, this marginally improves accuracy in regression tasks.
- The more highlighted contribution of the paper is a Bayesian interpretation of how their recursive TFMs refine their predictions as the model loops over the test examples. They approximate model prediction with kernel regression where the kernel compares the attention patterns of the labeled context of two test samples. They validate this approximation by showing that it predicts model predictions more accurately than kernels that measure test example similarity via raw input features or initial representations only. They interpret these kernel predictors as Gaussian-process posterior means, which allows them to compute the negative log marginal likelihood (NLML). They find that while the recursive TFMs loop over the data, NLML decreases, and how the data fit and model complexity terms change depends on the type of recursive model (looped Transformer-based or TRM). However, a vanilla TFM is also found to decrease NLML when logging its evolution across layer depth. They interpret the decrease in NLML as implicit Bayesian model selection.
Critical strengths
I found the paper's starting question and idea quite interesting: TFMs are essentially trained for ICL, ICL is often interpreted as implicit Bayesian inference. What does adding recursive depth mean in that Bayesian interpretation? The paper is not merely bringing in a popular technique (scaling recursive depth) into yet another type of model, but chooses a setting where it's especially interesting to add that component, and attempts to understand and explain how this changes the model's internal workings.
Critical weaknesses
I find the implicit Bayesian model selection argument somewhat weak. While the kernel ridge regression approximation with the attention-based kernel is well motivated, it only matches the model's predictions, and does not guarantee that the model is itself performing this learning algorithm. The model selection argument holds only for this surrogate model, not the actual recursive TFM. In addition, the NLML decreases for the vanilla TFM as well, making it unclear why the model selection argument is claimed to hold more for the recursive models than the vanilla TFM. I find this an important weakness, since the implicit Bayesian model selection argument is positioned as the paper's main contribution, and is made specifically for the recursive models.
The benchmark evaluations attempt fair comparison settings, but fairness raises questions. The authors use equalised pretraining budget (in terms of wall-clock time) between the vanilla TFM and recursive variants. Inference time cost is not discussed (which are higher for recursive models). Due to a small computational budget, the authors' reproduction of the original, vanilla TFM models used a smaller pretraining budget. This makes it unclear whether recursive reasoning compensates for the models being undertrained, or would benefit any model. This concern is relevant as its answer impacts the significance of the benchmark results.
Questions for authors
Questions
- The 'null-baseline' enc_x_rbf kernel is quite competitive with the authors' proposed attention kernel. Do you have any intuition on why this is the case?
- The Bayesian model selection argument does not seem specific to Tabular foundation models, it seems to potentially generalise to PFNs or any model that does in-context learning. How do you think your results and interpretations might transfer to those? And why did you consider Tabular foundation models in particular, and not a wider class of models?
Clarification requests / small changes
- Table 2 is unclear and confusing at first, I think it should be clarified more what an endpoint change means (in the table and caption).
- Figure 2 has too small legends
Suggestions:
- The clarity of the paper should be improved. Especially section 6 is unclear: some tables are difficult to interpret, and the discussion is hard to follow. I think isolating and clearly highlighting each finding would be helpful.
- Evaluate benchmark performance at equalised inference-time budget, for example, comparing 36-layer standard Transformer with a 6-layer Transformer looped six times. This would test whether inference-time computation is better spent on recursive depth or a larger architecture.
- Could you plot benchmark performance as a function of the pretraining budget, so that we can see how much benefit recursive depth brings at different pretraining stages?
6 comments
We address the reviewer's questions below.
Why is the enc_x_rbf kernel competitive.
The strong performance of enc_x_rbf is largely due to the effectiveness of RBF similarity in capturing local, nonlinear relationships, rather than solely the quality of the encoded representations. As shown in Table 1, replacing RBF with linear or cosine similarity substantially reduces for both enc_x and raw_x kernels. Moreover, RBF achieves high even when applied directly to raw input features raw_x, suggesting that the similarity function itself plays an important role.
To further support this point, we evaluate our attention-induced kernel, attn, using all three similarity functions and compare its performance with that of the two null kernels. The results below report median across recursive steps for TRM-TabDPT using seed 0 and fold 0. Therefore, they differ slightly from those in Table 1, which aggregates results over five seeds and two folds.
| Kernel | lin | cos | rbf |
|---|---|---|---|
attn | 0.98 | 0.98 | 0.97 |
raw_x | 0.79 | 0.72 | 0.93 |
enc_x | 0.76 | 0.76 | 0.94 |
The attention-induced kernel achieves consistently high prediction agreement across all three similarity functions (linear, cosine, and RBF), with ranging from 0.97 to 0.98, whereas the input-based kernels are much more sensitive to the choice of similarity function. Even with RBF similarity, our attention kernel outperforms both null kernels, further supporting the effectiveness of our approach. We use cosine similarity for the attention kernel primarily because it provides a direct interpretation in terms of attention-profile alignment and yields a fixed kernel trace, simplifying the GP analysis (line 317).
Generalization beyond TFMs
TFMs are closely related to PFNs, as they share the same in-context learning formulation and Transformer-based architectures, and many TFMs, such as TabPFN, are themselves PFNs. Our analysis could therefore naturally extend to other PFN-style models with numerical inputs and scalar regression outputs. The key requirements are that queries attend to individual labeled context samples and that the outputs are scalar, allowing us to construct model-induced kernels and analyze their GP marginal likelihoods. In contrast, extending our analysis to other forms of in-context learning or recursive reasoning, such as text-based ICL with LLMs or TRM-style reasoning on structured puzzles like Sudoku and ARC-AGI, is less straightforward because context elements are text tokens or grid cells rather than numerical labeled samples.
We focus on TFMs for two main reasons. First, as described in Introduction, while scaling test-time computation through techniques such as chain-of-thought reasoning in LLMs or recursive reasoning of TRM has proven effective, it remains largely unexplored for TFMs, despite their growing importance (lines 36-43). Second, TFMs provide a natural setting for our Bayesian analysis: the context consists of labeled samples, and regression admits a direct interpretation through kernel predictors and GP marginal likelihood.
Clarification Requests
Thank you for pointing out these clarification issues. Regarding Table 2, an endpoint change refers to the difference between the final and initial values over the evaluated interval: recursive steps to for TRM-TabDPT, recursive steps to for Looped-TabDPT, and layers to for Standard TabDPT. We will carefully clarify this definition in Table 2 and improve the readability of Figure 2's legends in the revised version.
We thank the reviewer for the valuable suggestions and address them below:
Equalized Inference-Time Budget.
To address this suggestion, we are considering training a 24-layer Standard TabDPT for comparison with the 6-step Looped-TabDPT and will report the results in our rebuttal if the experiments are completed in time (24 rather than 36 because Looped-TabDPT propagates gradients through only the last four of its six recursive steps; so, only a 24-layer standard model could be trained within our computational budget without changing the pre-training setting or incorporating additional techniques such as gradient checkpointing). However, importantly, the point is that recursive reasoning frameworks can achieve greater effective depth while maintaining a smaller parameter count and lower training memory requirements by reusing the same backbone and limiting gradient propagation through recursion (the last four steps for Looped-TabDPT and deep supervision for TRM-TabDPT; see Appendix A). For example, each of TRM-TabDPT's six outer steps applies the 6-layer backbone four times (Figure 4b), resulting in effective layer passes (see lines 599-600). Training a standard Transformer of comparable depth would be impractical under our computational budget, whereas TRM framework enables substantially greater effective depth with manageable training resources.
Performance Across Pre-training Budgets.
Based on our observations during pre-training, Standard TabDPT improves relatively quickly and reaches a performance plateau earlier, whereas recursive models require more pre-training to reach their best performance. Our six-day pre-training budget was intended to allow both standard and recursive models to reach stable performance, and our observations suggest that their performance had largely saturated by this point. Therefore, the improvements achieved by recursive models are not simply by insufficient pre-training of Standard TabDPT.
We sincerely thank the reviewer for their valuable feedback and constructive suggestions.
Below, we first address the raised weaknesses.
NLML also decreases for the vanilla TFM
The NLML is the sum of a data-fit term D and a complexity term C (Equation 8). Bayesian model selection (Occam's razor) balances these two terms and can prefer a simpler hypothesis even if it fits the data slightly worse. Therefore, a decrease in NLML alone does not distinguish Bayesian model selection, which considers both terms, from simply improving data fit. Table 2 reveals a key difference between Standard TabDPT (vanilla TFM) and the recursive TFMs. For Standard TabDPT, NLML decreases only through improved data fit: the data-fit term decreases on all 35/35 datasets, while the complexity term never decreases (0/35 datasets; see Table 2, row 3). Thus, wherever NLML decreases, the improvement comes from only better data fit. This pattern is consistent with improved data fitting rather than an considering both terms data-fit and complexity together (and possibly their trade-off). In contrast, recursion reduces complexity on 28/35 datasets for TRM-TabDPT and 18/35 for Looped-TabDPT. More importantly, on 13/35 and 6/35 datasets, respectively, NLML decreases despite worse data fit, as the reduction in complexity outweighs the loss in fit. Thus, our claim specific to recursion is not merely that NLML decreases, but that it can decrease through complexity reduction, even at the expense of data fit. This is the signature of Bayesian (Occam’s razor) model selection, and the vanilla TFM never exhibits it.
Bayesian model selection applies only to the surrogate model
We agree that matching the TFM's predictions with a kernel predictor does not prove that the recursive TFM explicitly performs Bayesian model selection or optimizes NLML. As stated in the paper (e.g., lines 21, 91, 358), our claim is about empirical evidence of Bayesian model-selection behavior, not a proof of the underlying mechanism.
Importantly, however, the kernel is not constructed independently of the TFM. It is derived directly from the TFM's internal attention patterns at each recursive step, and its predictions closely approximate the recursive TFM's actual predictions (median R2=0.98 for both recursive models; Table 1). Prior work has also motivated viewing TFMs as similarity-based predictors over the context (see lines 206–208). Therefore, the observed NLML trajectories characterize the evolution of the TFM's own learned similarity structure, providing empirical evidence consistent with Bayesian model-selection behavior during recursive refinement.
Pretraining budget and inference cost
Our goal was to study the benefits of recursive reasoning under a controlled, equal pretraining compute budget. For this purpose, we believe that six A100 GPU-days per model represents a reasonable computational budget for a controlled architectural study, although it is smaller than the resources used to train large-scale TFMs (e.g., TabDPT v1.1 was trained using eight GPUs, and TabICL-v1 required 60 A100 GPU-days, and, some TFMs, such as the TabPFN family and TabICL-v2, do not provide open-source pretraining pipelines, making controlled reproduction difficult). We acknowledge that our results do not establish whether the benefits of recursion would persist under substantially larger pretraining budgets. However, they demonstrate that recursion improves predictive performance under matched pretraining budget we could provide. Importantly, our objective was not to develop a TFM that outperforms existing large-scale models, but rather to investigate how recursive reasoning can benefit TFMs.
Regarding inference time, we agree that recursive models incur additional computational costs, and we will acknowledge this more explicitly in the revised paper. However, the central motivation of test-time compute scaling is precisely to allocate additional inference computation to improve predictive performance, as in other approaches such as chain-of-thought reasoning for LLMs and the original TRM paper. Our experiments demonstrate that recursive reasoning can effectively leverage this additional computation to improve predictive performance, particularly on regression tasks.