Recursive Tabular Foundation Models Exhibit Bayesian Model-Selection Behavior

Tabular foundation models (TFMs) have recently achieved strong performance on new prediction tasks through in-context learning without dataset-specific training. However, most existing TFMs perform inference in one computational pass, which may limit their ability to capture complex relationships in the context. We study latent recursive reasoning to scale test-time computation, developing recursive TFMs that repeatedly revisit the context and progressively refine predictions. On regression tasks, we characterize this refinement with attention-induced kernel predictors that closely reproduce the networks' predictions and analyze how the corresponding Gaussian-process (GP) marginal likelihood evolves across recursive steps. We find that recursive refinement typically improves marginal likelihood by changing the balance between data fit and model complexity. These findings provide empirical evidence for Bayesian model-selection behavior in recursive TFMs, although neither predictive pre-training nor the kernel predictors' prediction-matching calibration explicitly optimizes marginal likelihood.