acceptodds
Under review as a conference paper at ICLR 2027

Recursive Tabular Foundation Models Exhibit Bayesian Model-Selection Behavior

Abstract

Tabular foundation models (TFMs) have recently achieved strong performance on new prediction tasks through in-context learning without dataset-specific training. However, most existing TFMs perform inference in one computational pass, which may limit their ability to capture complex relationships in the context. We study latent recursive reasoning to scale test-time computation, developing recursive TFMs that repeatedly revisit the context and progressively refine predictions. On regression tasks, we characterize this refinement with attention-induced kernel predictors that closely reproduce the networks' predictions and analyze how the corresponding Gaussian-process (GP) marginal likelihood evolves across recursive steps. We find that recursive refinement typically improves marginal likelihood by changing the balance between data fit and model complexity. These findings provide empirical evidence for Bayesian model-selection behavior in recursive TFMs, although neither predictive pre-training nor the kernel predictors' prediction-matching calibration explicitly optimizes marginal likelihood.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular foundation models, recursive reasoning, recursive models, looped Transformers, test-time computation, in-context learning, Bayesian model selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_taeeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adds recursive reasoning to tabular foundation models (such as TabPFN), in the style of looped Transformers and the Tiny Recursive model. They make two contributions.

  • Experimentally, they show that at matched pretraining budget, this marginally improves accuracy in regression tasks.
  • The more highlighted contribution of the paper is a Bayesian interpretation of how their recursive TFMs refine their predictions as the model loops over the test examples. They approximate model prediction with kernel regression where the kernel compares the attention patterns of the labeled context of two test samples. They validate this approximation by showing that it predicts model predictions more accurately than kernels that measure test example similarity via raw input features or initial representations only. They interpret these kernel predictors as Gaussian-process posterior means, which allows them to compute the negative log marginal likelihood (NLML). They find that while the recursive TFMs loop over the data, NLML decreases, and how the data fit and model complexity terms change depends on the type of recursive model (looped Transformer-based or TRM). However, a vanilla TFM is also found to decrease NLML when logging its evolution across layer depth. They interpret the decrease in NLML as implicit Bayesian model selection.
Critical strengths

I found the paper's starting question and idea quite interesting: TFMs are essentially trained for ICL, ICL is often interpreted as implicit Bayesian inference. What does adding recursive depth mean in that Bayesian interpretation? The paper is not merely bringing in a popular technique (scaling recursive depth) into yet another type of model, but chooses a setting where it's especially interesting to add that component, and attempts to understand and explain how this changes the model's internal workings.

Critical weaknesses

I find the implicit Bayesian model selection argument somewhat weak. While the kernel ridge regression approximation with the attention-based kernel is well motivated, it only matches the model's predictions, and does not guarantee that the model is itself performing this learning algorithm. The model selection argument holds only for this surrogate model, not the actual recursive TFM. In addition, the NLML decreases for the vanilla TFM as well, making it unclear why the model selection argument is claimed to hold more for the recursive models than the vanilla TFM. I find this an important weakness, since the implicit Bayesian model selection argument is positioned as the paper's main contribution, and is made specifically for the recursive models.

The benchmark evaluations attempt fair comparison settings, but fairness raises questions. The authors use equalised pretraining budget (in terms of wall-clock time) between the vanilla TFM and recursive variants. Inference time cost is not discussed (which are higher for recursive models). Due to a small computational budget, the authors' reproduction of the original, vanilla TFM models used a smaller pretraining budget. This makes it unclear whether recursive reasoning compensates for the models being undertrained, or would benefit any model. This concern is relevant as its answer impacts the significance of the benchmark results.

Questions for authors

Questions

  • The 'null-baseline' enc_x_rbf kernel is quite competitive with the authors' proposed attention kernel. Do you have any intuition on why this is the case?
  • The Bayesian model selection argument does not seem specific to Tabular foundation models, it seems to potentially generalise to PFNs or any model that does in-context learning. How do you think your results and interpretations might transfer to those? And why did you consider Tabular foundation models in particular, and not a wider class of models?

Clarification requests / small changes

  • Table 2 is unclear and confusing at first, I think it should be clarified more what an endpoint change means (in the table and caption).
  • Figure 2 has too small legends

Suggestions:

  • The clarity of the paper should be improved. Especially section 6 is unclear: some tables are difficult to interpret, and the discussion is hard to follow. I think isolating and clearly highlighting each finding would be helpful.
  • Evaluate benchmark performance at equalised inference-time budget, for example, comparing 36-layer standard Transformer with a 6-layer Transformer looped six times. This would test whether inference-time computation is better spent on recursive depth or a larger architecture.
  • Could you plot benchmark performance as a function of the pretraining budget, so that we can see how much benefit recursive depth brings at different pretraining stages?
6 comments

We sincerely thank the reviewer for their valuable feedback and constructive suggestions.

Below, we first address the raised weaknesses.

NLML also decreases for the vanilla TFM

The NLML is the sum of a data-fit term DD and a complexity term CC (Equation 8). Bayesian model selection (Occam's razor) balances these two terms and can prefer a simpler hypothesis even if it fits the data slightly worse. Therefore, a decrease in NLML alone does not distinguish Bayesian model selection, which considers both terms, from simply improving data fit. Table 2 reveals a key difference between Standard TabDPT (vanilla TFM) and the recursive TFMs. For Standard TabDPT, NLML decreases only through improved data fit: the data-fit term decreases on all 35/35 datasets, while the complexity term never decreases (0/35 datasets; see Table 2, row 3). Thus, wherever NLML decreases, the improvement comes from only better data fit. This pattern is consistent with improved data fitting rather than an considering both terms data-fit and complexity together (and possibly their trade-off). In contrast, recursion reduces complexity on 28/35 datasets for TRM-TabDPT and 18/35 for Looped-TabDPT. More importantly, on 13/35 and 6/35 datasets, respectively, NLML decreases despite worse data fit, as the reduction in complexity outweighs the loss in fit. Thus, our claim specific to recursion is not merely that NLML decreases, but that it can decrease through complexity reduction, even at the expense of data fit. This is the signature of Bayesian (Occam’s razor) model selection, and the vanilla TFM never exhibits it.

Bayesian model selection applies only to the surrogate model

We agree that matching the TFM's predictions with a kernel predictor does not prove that the recursive TFM explicitly performs Bayesian model selection or optimizes NLML. As stated in the paper (e.g., lines 21, 91, 358), our claim is about empirical evidence of Bayesian model-selection behavior, not a proof of the underlying mechanism.

Importantly, however, the kernel is not constructed independently of the TFM. It is derived directly from the TFM's internal attention patterns at each recursive step, and its predictions closely approximate the recursive TFM's actual predictions (median R2=0.98R^2 = 0.98 for both recursive models; Table 1). Prior work has also motivated viewing TFMs as similarity-based predictors over the context (see lines 206–208). Therefore, the observed NLML trajectories characterize the evolution of the TFM's own learned similarity structure, providing empirical evidence consistent with Bayesian model-selection behavior during recursive refinement.

Pretraining budget and inference cost

Our goal was to study the benefits of recursive reasoning under a controlled, equal pretraining compute budget. For this purpose, we believe that six A100 GPU-days per model represents a reasonable computational budget for a controlled architectural study, although it is smaller than the resources used to train large-scale TFMs (e.g., TabDPT v1.1 was trained using eight GPUs, and TabICL-v1 required 60 A100 GPU-days, and, some TFMs, such as the TabPFN family and TabICL-v2, do not provide open-source pretraining pipelines, making controlled reproduction difficult). We acknowledge that our results do not establish whether the benefits of recursion would persist under substantially larger pretraining budgets. However, they demonstrate that recursion improves predictive performance under matched pretraining budget we could provide. Importantly, our objective was not to develop a TFM that outperforms existing large-scale models, but rather to investigate how recursive reasoning can benefit TFMs.

Regarding inference time, we agree that recursive models incur additional computational costs, and we will acknowledge this more explicitly in the revised paper. However, the central motivation of test-time compute scaling is precisely to allocate additional inference computation to improve predictive performance, as in other approaches such as chain-of-thought reasoning for LLMs and the original TRM paper. Our experiments demonstrate that recursive reasoning can effectively leverage this additional computation to improve predictive performance, particularly on regression tasks.

We address the reviewer's questions below.

Why is the enc_x_rbf kernel competitive.

The strong performance of enc_x_rbf is largely due to the effectiveness of RBF similarity in capturing local, nonlinear relationships, rather than solely the quality of the encoded representations. As shown in Table 1, replacing RBF with linear or cosine similarity substantially reduces R2R^2 for both enc_x and raw_x kernels. Moreover, RBF achieves high R2R^2 even when applied directly to raw input features raw_x, suggesting that the similarity function itself plays an important role.

To further support this point, we evaluate our attention-induced kernel, attn, using all three similarity functions and compare its performance with that of the two null kernels. The results below report median R2R^2 across recursive steps for TRM-TabDPT using seed 0 and fold 0. Therefore, they differ slightly from those in Table 1, which aggregates results over five seeds and two folds.

Kernellincosrbf
attn0.980.980.97
raw_x0.790.720.93
enc_x0.760.760.94

The attention-induced kernel achieves consistently high prediction agreement across all three similarity functions (linear, cosine, and RBF), with R2R^2 ranging from 0.97 to 0.98, whereas the input-based kernels are much more sensitive to the choice of similarity function. Even with RBF similarity, our attention kernel outperforms both null kernels, further supporting the effectiveness of our approach. We use cosine similarity for the attention kernel primarily because it provides a direct interpretation in terms of attention-profile alignment and yields a fixed kernel trace, simplifying the GP analysis (line 317).

Generalization beyond TFMs

TFMs are closely related to PFNs, as they share the same in-context learning formulation and Transformer-based architectures, and many TFMs, such as TabPFN, are themselves PFNs. Our analysis could therefore naturally extend to other PFN-style models with numerical inputs and scalar regression outputs. The key requirements are that queries attend to individual labeled context samples and that the outputs are scalar, allowing us to construct model-induced kernels and analyze their GP marginal likelihoods. In contrast, extending our analysis to other forms of in-context learning or recursive reasoning, such as text-based ICL with LLMs or TRM-style reasoning on structured puzzles like Sudoku and ARC-AGI, is less straightforward because context elements are text tokens or grid cells rather than numerical labeled samples.

We focus on TFMs for two main reasons. First, as described in Introduction, while scaling test-time computation through techniques such as chain-of-thought reasoning in LLMs or recursive reasoning of TRM has proven effective, it remains largely unexplored for TFMs, despite their growing importance (lines 36-43). Second, TFMs provide a natural setting for our Bayesian analysis: the context consists of labeled samples, and regression admits a direct interpretation through kernel predictors and GP marginal likelihood.


Clarification Requests

Thank you for pointing out these clarification issues. Regarding Table 2, an endpoint change refers to the difference between the final and initial values over the evaluated interval: recursive steps t=2t=2 to t=6t=6 for TRM-TabDPT, recursive steps t=3t=3 to t=6t=6 for Looped-TabDPT, and layers ℓ=8\ell=8 to ℓ=16\ell=16 for Standard TabDPT. We will carefully clarify this definition in Table 2 and improve the readability of Figure 2's legends in the revised version.

We thank the reviewer for the valuable suggestions and address them below:

Equalized Inference-Time Budget.

To address this suggestion, we are considering training a 24-layer Standard TabDPT for comparison with the 6-step Looped-TabDPT and will report the results in our rebuttal if the experiments are completed in time (24 rather than 36 because Looped-TabDPT propagates gradients through only the last four of its six recursive steps; so, only a 24-layer standard model could be trained within our computational budget without changing the pre-training setting or incorporating additional techniques such as gradient checkpointing). However, importantly, the point is that recursive reasoning frameworks can achieve greater effective depth while maintaining a smaller parameter count and lower training memory requirements by reusing the same backbone and limiting gradient propagation through recursion (the last four steps for Looped-TabDPT and deep supervision for TRM-TabDPT; see Appendix A). For example, each of TRM-TabDPT's six outer steps applies the 6-layer backbone four times (Figure 4b), resulting in 6×(3+1)×6=1446 \times (3+1) \times 6 = 144 effective layer passes (see lines 599-600). Training a standard Transformer of comparable depth would be impractical under our computational budget, whereas TRM framework enables substantially greater effective depth with manageable training resources.

Performance Across Pre-training Budgets.

Based on our observations during pre-training, Standard TabDPT improves relatively quickly and reaches a performance plateau earlier, whereas recursive models require more pre-training to reach their best performance. Our six-day pre-training budget was intended to allow both standard and recursive models to reach stable performance, and our observations suggest that their performance had largely saturated by this point. Therefore, the improvements achieved by recursive models are not simply by insufficient pre-training of Standard TabDPT.

Reviewer_LAYXOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether looped transformer can improve tabular foundation models and how their prediction rules change across recursive steps. Using TabDPT as the backbone, the authors develop variants based on looped transformers and Tiny Recursive Models. To interpret their behavior, they construct attention-based kernel predictors that closely reproduce the networks’ predictions and track the corresponding GP marginal likelihood across steps. The results show that recursion often improves marginal likelihood by adjusting the balance between data fit and model complexity penalty term, providing empirical evidence for Bayesian model-selection behavior.

Critical strengths
  • The paper connects recursive computation in tabular foundation models with a familiar statistical principle: balancing data fit and model complexity. This provides a useful way to understand how additional computation changes the prediction rule.
  • The experiments are sound and reasonably extensive. The comparison uses a comparable pretraining budget, covers both classification and regression, and includes different architectures and recursion settings.
Critical weaknesses
  • The kernel analysis describes how predictions change but does not directly identify the computation responsible for that change. A probe of intermediate states could make the interpretation more concrete, as suggested in Q1.
  • The evidence for test-time scaling is currently limited by the fixed recursion count used during pretraining. Performance largely saturates around that count. It remains unclear whether this reflects a limit of the approach or the training procedure, as discussed in Q2.
Questions for authors

Q1. The kernel-regression interpretation can also be motivated by Kang et al. [1], who show by construction that transformers can implement gradient-descent updates for GP posterior predictive inference. Beyond the Bayesian model-selection perspective, could the authors probe whether the recursive updates behave like an iterative loss-minimization process? For example, on a controlled GP regression task with a known, fixed kernel, one could probe intermediate states and check whether the decoded updates align with descent directions or reduce the corresponding kernel-regression objective. This would provide a complementary explanation of how recursion refines predictions.

[1] Kang, G., Lee, C. J., and Cheng, X. (2026). Transformers Can Learn Posterior Predictive Distributions In-Context. ICML

Q2. I am interested in whether the gains from recursion could extend further. The current models are pretrained with six recursion steps, and additional test-time steps provide little benefit. Could the authors pretrain with a randomly sampled iteration count, following Huginn [2], and then test whether increasing the number of loops at inference improves prediction? Additionally, recent work on fixed points in looped transformers [3] studies how the training-depth distribution affects behavior across inference depths. A comparison under a similar training budget would help clarify whether random-depth pretraining enables stronger test-time scaling in PFNs and other tabular foundation models, as it does for LLM reasoning.

[2] Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B. R., Kailkhura, B., Bhatele, A., and Goldstein, T. (2025). Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. NeurIPS

[3] Huang, B., Shi, C., Chen, J., Wen, S., Liu, Z., Xing, E., and Ma, X. (2026). Towards Looped Models Done Right, Part II: Rethinking at Fixed Points. arXiv:2610.06833.

3 comments

We sincerely thank the reviewer for their valuable feedback and constructive suggestions.

Response to Question 1

Thank you for raising this insightful question and suggesting a complementary perspective.

Kang et al. [1] consider GP posterior inference with a prespecified, fixed kernel and theoretically show that Transformers can implement gradient descent optimization to approximate the corresponding kernel ridge regression (KRR) solution. In their setting, the kernel remains fixed while iterative updates refine the KRR coefficients. In contrast, our analysis focuses on how the model-induced kernel itself evolves across recursive steps. We construct a kernel from the model's attention patterns over context samples, and as these patterns change across recursive steps, the induced kernel also changes. To characterize this evolution, we analyze the GP negative log marginal likelihood (NLML), which captures changes in the kernel through its data-fit and complexity terms. Importantly, unlike the KRR objective, NLML includes a log-determinant complexity term defined over the kernel, allowing us to examine how recursion changes the balance between data fit and model complexity. Our analysis also provides an iterative refinement perspective: we empirically observe that the model-induced kernels evolve across recursive steps in a way that often decreases NLML, resembling iterative optimization of this objective. However, this is an empirical observation, and we do not claim that the model explicitly performs gradient descent on NLML or follows any particular optimization algorithm.

The experiment suggested by the reviewer provides an opportunity to further investigate two perspectives: (1) iterative GP posterior inference through KRR, as studied by Kang et al., and (2) our view of evolving model-induced kernels through their GP NLML. We focus on a simple, high-noise regression setting, where balancing data fit and model complexity is particularly relevant (similar to our noisy sine regression example in Figure 1).

Specifically, we conducted a controlled experiment on synthetic GP regression tasks with a known RBF kernel. We sampled 300 context and 500 query points uniformly from [−1,1][-1,1] and generated the underlying function from a zero-mean GP with an RBF kernel (lengthscale ℓ=0.2\ell=0.2, signal variance σf2=1\sigma_f^2=1). We added Gaussian observation noise with standard deviation σn=1.0\sigma_n=1.0 to the context targets. To investigate whether recursive refinement progressively approaches the solution of the underlying KRR problem, we computed the exact GP posterior mean using the known kernel and measured the MSE between this posterior mean and the model's intermediate predictions across recursive steps. This provides a prediction-level measure of convergence toward the KRR solution, rather than a direct test of gradient-descent-like updates. We also applied our attention-induced kernel analysis to track the evolution of the data-fit, complexity, and NLML terms. The results for TRM-TabDPT across 100 independently generated tasks (with different seeds), comparing recursion steps 2 and 6, are summarized below:

  • MSE relative to the exact GP posterior mean (KRR solution): Decreased in 72/100 tasks.
  • NLML data-fit term: Decreased in 3/100 tasks.
  • NLML complexity term: Decreased in 98/100 tasks.
  • NLML: Decreased in 97/100 tasks.

These results suggest that both perspectives capture part of recursive refinement. In most tasks (72/100), the model’s predictions move closer to the exact posterior mean under the fixed kernel, consistent with the iterative-inference view of Kang et al. At the same time, the NLML of the attention-induced GP decreases almost universally (97/100). This decrease is consistently driven by a lower complexity term (98/100), even though data fit worsens (97/100). This suggests that our Bayesian model-selection perspective (based on NLML), provides a more consistent characterization of recursive refinement than convergence toward the fixed-kernel KRR solution alone. We will include this experiment (with more detail) to the revised paper and discuss its connection to Kang et al.

Response to Question 2

Thank you for this interesting suggestion. We agree that it is currently unclear whether the saturation around six test-time recursion steps reflects a limit of recursive refinement or a consequence of fixed-depth pretraining procedure. Randomly sampling the recursion count during pretraining is an interesting direction for enabling further improvements beyond the trained recursion horizon. However, thoroughly investigating this approach requires many additional experiments. We have started exploring this idea, but given the limited rebuttal period, we may not be able to provide comprehensive results at this stage. We consider this as a promising direction for future work.

Thank you for the additional experiment. What I had in mind was a direct analysis of the internal states across recursive steps, to examine whether their updates resemble an optimization process. The new experiment provides useful evidence at the prediction level, although it does not directly answer that question. However, this was a suggestion for further investigation rather than a major concern. Please feel free to prioritize the other reviewers’ remaining concerns. I will maintain my score.

Thank you so much for your thoughtful feedback, constructive suggestions, and encouragement throughout the review process. We sincerely appreciate your interest in our work and your recognition of the value of our GP surrogate analysis.

Reviewer_K24ROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper the authors propose the use of looped models in the tabular foundation model setting. They consider two looped architectures - TRM and Looped Transformer. They then argue that looped TFNs implicitly perform Bayesian model selection. The authors compare their models to state of the art TabDPT.

Critical strengths

The authors apply the looped concepts to the PFN setting, which is interesting. They further try to make a connection between looping in TFMs and the Bayesian model selection using the ability of the looped models to exit at arbitrary step to gauge the level of hypothesis refinement.

Critical weaknesses
  1. The empirical performance advantage is unclear: despite the relative small scale of the experiments, no spread metrics are provided. We see the differences go into the thrid significant figure. A course of action to fix this would be to, possibly, select more difciult problems, since these appear to be quite saturated. The methods are also not compute matched, which is important for comparing the looped models to their non-looped counterparts.
  2. Sections 2,3,4 are the same sections spead over the paper. The subsection 5.2 cites no work that one could lean on to understand what is the context of this and why is it important. The authors, instead, fail to describe in more detail how do they apply the looped models to the TFN setting, especially considering that the PFNs usually employ an encoder-decoder architecture.
  3. Attrocious figure 2. Nothing can be seen due to nanoscopic scale, yet it occupies a good portion of the page. The bold highlights in Tab.4. make no sense.
Questions for authors
  1. Am I reading it right that in Tab.2 you are measuring the outcomes between timesteps for the TRM and Looped Transformer but at a layer 8 cutoff for the TabDPT? Was it ever trained for that and how would that make sense?
  2. Was a stopping criteria trained for either of the two looped architectures?
2 comments

We sincerely thank the reviewer for their valuable feedback and constructive suggestions.

Response to Question 1

Yes, Table 2 compares changes across recursion steps for TRM-TabDPT and Looped-TabDPT, and across Transformer layers for Standard TabDPT. For Standard TabDPT, to get intermediate predictions, we froze the pretrained backbone and trained separate decoder heads on the representations from layers 8, 10, 12, 14, and 16 (lines 305–307). Thus, for example, the prediction at layer 8 is obtained by applying a separately trained decoder to that layer's output, without updating the backbone. Our objective was to compare how predictions evolve across recursive steps versus increasing depth in a standard model. Training readout heads on intermediate layers is a common practice for analyzing model behavior (e.g., see Balef et al., 2026, Is One Layer Enough).

Response to Question 2

No, both models were trained with a fixed recursion horizon of T=6T=6, which we also used as the default at inference time. As shown in Table 3, performance generally improves and stabilizes around step 6, while additional steps provide little benefit and can slightly degrade performance. We therefore use the trained horizon as the stopping point. We also evaluated several training recursion number configurations in our ablation studies (Appendix B, Tables 5 and 6).


Response to Weakness 1

Regarding task difficulty, our evaluation covers 72 classification datasets (CC18) and 35 regression datasets (CTR23), following the TabDPT benchmark. While classification performance is relatively saturated, the improvements on regression are substantial. Compared with the 16-layer Standard TabDPT, our 6-layer TRM-TabDPT improves Pearson correlation from 0.896 to 0.917 and R2R^2 from 0.807 to 0.843 (Table 4). These gains are considerably larger than the differences in the third decimal place observed for classification.

Regarding compute matching, all models were trained under the same pre-training compute budget. Inference compute was intentionally not matched, as our objective is to scale test-time computation through recursion to improve predictions beyond those of a fixed-depth model, following the motivation of chain-of-thought techniques, looped Transformers, and TRM (lines 36–49). Thus, our comparison investigates whether scaling inference-time computation through recursion can improve predictive performance under the same pre-training budget, even with fewer model parameters, rather than comparing models under a fixed inference compute budget. In our revision, we will report the exact increase in inference time to make this computational trade-off more transparent.

Response to Weakness 2

Regarding Sections 2, 3, and 4, they serve distinct purposes. Section 2 reviews related work, Section 3 introduces the standard TFM architecture, and Section 4 describes how we adapt TFMs to recursive architectures.

Regarding Section 5.2, the decomposition of GP marginal likelihood into data-fit and complexity terms, and its interpretation through Bayesian Occam's razor, are established concepts discussed in standard textbooks. We will add a citation to "Gaussian Processes for Machine Learning, by Rasmussen and Williams". The importance of this decomposition in our work is that it provides a principled framework for analyzing how the model-induced kernels evolve across recursive steps, particularly how they balance data fit and model complexity.

Regarding the architectural details, Section 3 explicitly describes the encoder–Transformer–decoder structure of standard TFMs (starting at line 155). Section 4 then explains how we extend this architecture by retaining the encoder and decoder while replacing the Transformer backbone with recursive computation. We also provide pseudocode for both recursive architectures in Appendix Figure 4.

Response to Weakness 3

Thank you for pointing out these presentation issues. We agree that Figure 2 contains too many small plots, making it difficult to read. Our intention was to show the GP marginal-likelihood trajectories across all 35 regression datasets to provide a complete view, but this comes at the cost of readability. In the revised paper, we will move the complete figure to appendix with larger, more readable plots.

Regarding Table 4, the bold rows highlight the main models we intended to compare: the 6-layer recursive models and the 16-layer Standard-TabDPT. The other configurations were included to illustrate points like increasing the number of layers improves the performance of Standard TabDPT, whereas the recursive models achieve better performance with fewer layers. We will clarify the purpose of the bold highlighting in the table caption.

I thank the authors for providing a rebuttal.

Q1: While linear probing intermediate layers is a good technique for analysis of the model handling, it might be subpar for comparisons of model performance. The authors do not provide the error bars to compare close-call evals. The 'verbatim' results from other papers achieve the best performances of the looped variants, suggesting that something might be missing from the reproduction.

Q2: Exit condition is one of the important things that distinguishes looped models from parameter sharing.

W1: The claim on lack of inference compute matching does not withstand the fact that the final performance is not improved against the baselines in 2/4 conditions and at all compared to the literature reported numbers. Moreover, it points to the fact, that perhaps some other aspect is responsible for these extra 0-1.5% of performance (on 2/4 conditions) than looping.

W2: To quote the paper:

L105: 2. RELATED WORK

L106: Tabular foundation models ...

L151: 3. PRELIMINARIES: TABULAR FOUNDATION MODELS

L175: 4. RECURSIVE TABULAR FOUNDATION MODELS

(this section 4 is actually about just normal recursive models)

(I do not agree with argumentation in favor of this structure is what I mean)

W3. The Fig.2 situation improved substantially, also there is still room for improvement.

Comments on the paper

3 comments
Reviewer_LAYXOpenReview ↗

A Solid Interpretability Contribution, but No Need to Oversell

In my view, the main contribution is the construction and validation of a tractable GP surrogate that closely reproduces the network’s predictions, and its use to study how those predictions change across recursive steps. This is more interesting to me than the accuracy gain from looping itself, which is less surprising. The marginal-likelihood decomposition provides an understandable account of this refinement: additional loops can improve data fit, but they can also reduce complexity while accepting worse data fit. This helps characterize how additional computation changes the effective prediction rule.

Admittedly, looped architectures and GP surrogates extracted from networks are established ideas, and the current experiments mainly demonstrate refinement within the trained recursion horizon. Nevertheless, I find their combination here useful for understanding tabular prediction. I value the surrogate analysis as an interpretability contribution in its own right. The concerns about compute comparisons and uncertainty reporting deserve attention, but they do not fundamentally undermine the observed regression gains. Overall, I think the paper clears the bar for acceptance.

I agree, however, that the Bayesian model-selection interpretation should be qualified, especially as readers and reviewers are becoming increasingly sensitive to overselling. The supported conclusion is that recursive refinement produces a sequence of GP surrogates whose marginal likelihood often improves through changes in data fit and complexity. This is consistent with Bayesian model-selection behavior, but does not establish that the network itself internally performs model selection. I encourage the authors to make this distinction explicit and consider softening the title and abstract as well, to prevent disappointed readers.

Dear Reviewers,

We are nearing the end of the discussion phase. If you haven't already done so, please read the authors' responses, reply to acknowledge their feedback, and indicate whether your initial concerns have been addressed (along with any score adjustments, if warranted).

Thank you for your dedicated efforts and valuable insights!

Best, Your AC

AuthorsOpenReview ↗

Summary of Revisions to the Manuscript

We sincerely thank all reviewers for their thoughtful feedback, constructive suggestions, and engagement throughout the review process. We are very glad that Reviewer LAYX finds our contributions valuable and strongly supports our work, and that Reviewer taee also finds our Bayesian model-selection contribution very interesting and valuable.

We have carefully considered the feedback and substantially revised our manuscript to address the concerns raised. Following the recommendations of Reviewers LAYX and taee, we have revised the title to "Recursive Tabular Foundation Models Exhibit Behavior Consistent with Bayesian Model Selection." We have also refined the presentation of our main contribution, drawing directly on Reviewer LAYX's insightful characterization of our work.

In addition, we have added a new experiment with a 24-layer Standard TabDPT and substantially improved the presentation of Section 6, particularly the experimental methodology, metrics, and conclusions.

Please see our revised submission, where all major changes are highlighted in blue.

Comparison with a Deeper Standard Model

We have completed an additional experiment with a 24-layer Standard TabDPT, trained under the same pre-training settings as other models. This is the largest standard Transformer we could train and evaluate within our computational resources (a single A100 40GB GPU). Training a 36-layer standard model was not feasible under our GPU memory constraints. In contrast, recursive architectures allow us to scale computation with lower memory requirements by reusing a smaller backbone across steps and limiting gradient propagation during pre-training.

The new results, compared with our main model configurations in Table 4, are:

ModelCC18 AUC ↑CC18 Acc. ↑CTR23 Corr. ↑CTR23 R² ↑
TRM-TabDPT (6-layer)0.9750.9260.9170.843
Looped-TabDPT (6-layer)0.9760.9270.9170.842
Standard TabDPT (16-layer)0.9750.9240.8960.807
Standard TabDPT (24-layer) [New]0.9740.9240.8980.810

Increasing the depth of Standard TabDPT from 16 to 24 layers yields only marginal improvements on regression (Pearson correlation from 0.896 to 0.898 and R² from 0.807 to 0.810), while classification performance remains essentially unchanged. Importantly, both 6-layer recursive models still substantially outperform the deeper 24-layer standard model on regression, achieving a correlation of 0.917 and R² values of 0.842–0.843.

Major Changes in Our Revision

In addition to the new experiment, we have made the following changes to the manuscript:

  1. Revised title and claims: Following the recommendations of Reviewers LAYX and taee, we have revised the title and qualified the claims in the abstract, contributions, and conclusion. We now emphasize that our GP surrogate analysis provides empirical evidence consistent with Bayesian model-selection behavior, rather than establishing that the recursive models themselves perform Bayesian model selection.

  2. Clarified limitations and future work: We have expanded the discussion of limitations and future research directions, including investigating the internal mechanisms underlying the observed behavior and exploring whether recursive refinement can continue beyond the fixed pre-training recursion horizon, as suggested by Reviewer LAYX.

  3. Improved GP background context: Following Reviewer K24R's suggestion, we have added references to established GP literature to support the marginal-likelihood decomposition and its interpretation in terms of data fit and model complexity.

  4. Substantially improved the presentation of Section 6: In response to the concerns raised by Reviewers taee and K24R, we have reorganized the experimental section to state the objectives of each experiment more explicitly, clarify the evaluation procedures and metrics, and present the main findings more clearly. We have also improved the table captions and figure readability. In particular, Figure 2 now shows five representative datasets per model, with larger, more readable plots, while the complete GP trajectories for all 35 regression datasets are provided in the appendix.

We again sincerely appreciate the reviewers' constructive feedback, which has helped us substantially improve the manuscript. We believe the current revision addresses the major concerns raised during the review process, and we remain committed to further improving the clarity and presentation of the paper in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.