Mechanisms and Symmetries of Tabular Foundation Models

Tabular foundation models with different architectures converge in accuracy across many classification and regression tasks. This raises questions a leaderboard cannot answer: (i) whether the models execute the same in-context algorithm, and (ii) where row, column, and class-permutation invariances originate. We study released checkpoints from three families through readout rules, interventions, and controlled pretraining comparisons. We can characterize the model behavior with simple similarity-based readouts: TabPFNv2 is better matched by a label vote than a prototype; TabICLv2 shows the reverse pattern. We show a kernel-based model is able to preserve row, column and class symmetries without incurring representation collapse. Motivated by this, we perform full pretraining for different architectural choices and demonstrate that a fully invariant model can match or outperform the default reference model. Together, these results give a mechanistic account of contemporary tabular foundation models and examine how architectural choices relate to their predictions and symmetries.