acceptodds
Under review as a conference paper at ICLR 2027

Mechanisms and Symmetries of Tabular Foundation Models

Abstract

Tabular foundation models with different architectures converge in accuracy across many classification and regression tasks. This raises questions a leaderboard cannot answer: (i) whether the models execute the same in-context algorithm, and (ii) where row, column, and class-permutation invariances originate. We study released checkpoints from three families through readout rules, interventions, and controlled pretraining comparisons. We can characterize the model behavior with simple similarity-based readouts: TabPFNv2 is better matched by a label vote than a prototype; TabICLv2 shows the reverse pattern. We show a kernel-based model is able to preserve row, column and class symmetries without incurring representation collapse. Motivated by this, we perform full pretraining for different architectural choices and demonstrate that a fully invariant model can match or outperform the default reference model. Together, these results give a mechanistic account of contemporary tabular foundation models and examine how architectural choices relate to their predictions and symmetries.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tabular foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Gw56OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how tabular foundation models use labeled context examples to make predictions and how architectural choices affect permutation invariance. The authors analyze pretrained TabPFNv2, TabICLv2, and Mitra checkpoints by testing how well simple prediction rules, such as weighted label voting and nearest-class prototypes, reproduce the models' predictions. They also modify attention computations in selected Transformer layers to assess their importance for predictive performance. They find that TabPFNv2's predictions are better approximated by attention-based label voting, whereas TabICLv2 learns representations that work well with simple classifiers, such as nearest-class prototypes and nearest neighbors.

The paper also investigates whether predictions remain consistent when rows or columns are reordered or class labels are renamed. The authors investigate ways to enforce these properties through inference-time modifications and architectural changes. They also provide a mathematical example showing that permutation invariance can be achieved without losing important predictive information. Through controlled pretraining of 13 TabICLv2 variants and evaluation on 38 TabArena tasks, they demonstrate that a model enforcing these symmetries can achieve predictive performance comparable to the reference architecture.

Critical strengths
  1. Demonstrating that permutation symmetries can be enforced without sacrificing predictive performance. The paper shows that a modified TabICLv2 architecture can guarantee invariance to row and column permutations and equivariance to class relabeling while achieving performance comparable to the reference model. This is a meaningful contribution because these properties are desirable for tabular learning, yet existing models do not consistently satisfy them. The controlled pretraining experiments provide convincing evidence that enforcing these symmetries need not come at the expense of predictive accuracy.

  2. Insights into how tabular foundation models use labeled context. The comparison of TabPFNv2, TabICLv2, and Mitra reveals interesting differences in their learned representations and prediction behavior. In particular, TabICLv2's representations support accurate classification using simple prototype and nearest-neighbor methods, whereas TabPFNv2 is better approximated by attention-based label voting.

Critical weaknesses

In Section 2.1, the authors extract hidden representations from a pretrained TabICLv2 model and use them to classify new observations. They compute the average representation of each class (the class prototype) and assign each query to the closest prototype. This simple classifier achieves 85.4% accuracy, close to the native TabICLv2 model's 86.4%. However, nearest-neighbor and linear classifiers also perform well, achieving 85.6% and 85.9%, respectively (Figure 2). These results show that TabICLv2 learns useful representations, but they do not establish that the model itself uses prototype classification.

Furthermore, Table 11 reports a large difference in expected calibration error between the prototype classifier (46.3%) and the native model (4.2%). The prototype probabilities are not calibrated, so this difference alone does not rule out a prototype-based mechanism.

Two additional experiments could help clarify this issue:

A. Compare predicted probabilities. The authors could first calibrate the prototype classifier on validation data, then compare its predicted probabilities with those of the native model on the same test observations. Close agreement would provide stronger evidence that the two methods behave similarly beyond simply predicting the same class.

B. Test whether individual examples matter beyond their class averages. The authors could perform controlled interventions that change individual context representations while preserving the average representation of each class. For example, two representations could change from 0 and 2 to 0.9 and 1.1. Both pairs have the same average of 1. A prototype classifier would treat them identically if the query representation and other class prototypes were unchanged. If the native model's predictions change substantially under a properly controlled intervention, this would suggest that it uses information beyond class prototypes.

These experiments would help distinguish between a model whose representations support prototype classification and one whose predictions actually depend on class prototypes. This distinction is important given the paper's goal of understanding how tabular foundation models make predictions.

Questions for authors

Do the learned representations reveal the model's actual prediction mechanism?

In Section 2.1, the prototype classifier achieves 85.4% accuracy, close to TabICLv2's native accuracy of 86.4%. However, kNN and linear classifiers also achieve similar accuracy (85.6% and 85.9%, respectively). Furthermore, Table 11 reports a much higher calibration error for the prototype classifier (46.3%) than for the native model (4.2%).

These results show that TabICLv2 learns representations that work well with several simple classifiers. However, it remains unclear whether the native model actually uses a prototype-based prediction mechanism.

Could the authors clarify what evidence supports this stronger interpretation? In particular, could they compare the native and prototype predicted probabilities after calibrating the prototype scores?

A further experiment could test whether the native predictions depend on individual context representations beyond their class averages. For example, changing two same-class embeddings from 0 and 2 to 0.9 and 1.1 preserves their average of 1. A pure prototype classifier would give the same prediction if the query and other class prototypes remained unchanged. A carefully controlled intervention could test whether TabICLv2 behaves similarly.

Such evidence would help determine whether the results reveal the model's actual prediction mechanism or mainly demonstrate that its representations support accurate prototype classification.

1 comment

Thank you for the careful reading and for the constructive feedback. Your central point is a fair and important one. Good prototype, kNN and linear readouts show that the representation is useful, and they do not by themselves show that the model computes a prototype. We agree, and what we claim in Section 2.1 is a statement about the representation, which supports accurate class choices.

Representation vs mechanism. The pretraining experiment is our strongest evidence on this question. We trained TabICLv2 from scratch with a prototype head and with a vote head, and both match the reference model, with the vote head slightly ahead. The same architecture can therefore be trained to support several different readouts.

Calibration. We ran the calibrated comparison you suggest. The 46.3% against 4.2% gap in Table 11 comes from the scale of the prototype scores. Cosine similarities lie in [-1, 1], so a unit-temperature softmax gives nearly uniform probabilities. We fitted a single temperature on held-out calibration rows, which the model treated as queries and never saw labels for, and evaluated on separate test rows. We used 38 TabArena classification tasks with 5 seeds each, 190 task-seed pairs in total. The prototype ECE falls from 0.380 to 0.034, close to the native model's 0.026, and the log loss falls from 0.753 to 0.329, against 0.283 for native. The fitted temperature transfers well from calibration rows to test rows, and a temperature fitted on the test rows themselves gives almost the same numbers.

Probabilities beyond class agreement. After calibration, the prototype and native probabilities are close on the same test rows, with a mean total variation distance of 0.075 and a mean KL divergence of 0.052. They are not identical. The two agree on the predicted class for 94% of rows, and the prototype is 1.9 accuracy points below native. A prototype readout on this representation is therefore a very good approximation of the native output, which is the claim we make about the representation. It does not show that the model computes a prototype, and we do not claim that.

Intervention on class averages. This is a good test of whether one released checkpoint depends on class averages. The pretraining experiment speaks to the broader question. Prototype and vote heads trained from scratch both match the reference, so reference-level performance does not depend on one particular readout. Our claim is about what the representation supports, and the calibrated comparison above addresses that directly.

Revisions. We will add the calibrated comparison to the appendix, and we will state the distinction between what the representation supports and what the model computes explicitly in Section 2.1.

Reviewer_vzmbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies prediction mechanisms in tabular foundation models (TabPFNv2, Mitra and TabICLv2) through readout comparisons, attention and depth interventions. It claims that TabPFN and

It also investigates permutation symmetries and trains modified TabICLv2 models to enforce them.

Critical strengths

I appreciate the focus on analysis. Understanding why TFM architectures have their particular structure and which choices matter is an important direction.

Critical weaknesses

The analysis is not sufficiently strict and full

The paper misses relevant work, particularly Do Tabular Foundation Models Learn Rules or Memorize Exemplars? and Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models.

The findings regarding vote vs prototype that make conclusion that TabICLv2 "shaped" models are different in mechanisms are in conflict with Mitra results in the same section (where its not clear which explanation -- vote vs prototype is clearly better). There also may be the issue in the choice of layers for the readouts (+ vote, knn and attention are actually very related and do not clearly explain and differentiate the mechanisms behind predictions)

I would recommend looking into the existing analysis around PFNs outlined above, clarifying what additional understanding the present analysis provides beyond the one in the papers (in terms of actual new empirical observations). Making comparisons deeper and more mechanistic is also a priority.

Equivariance: missing comparison and limited empirical benefit

The empirical benefits from permutation equivariance is limited. The fully invariant model has 20 wins versus 18 losses, a mean binary AUC gain of 0.22 percentage points, and essentially unchanged multiclass log loss.

EquiTabPFN also already studies class equivariance and uses an attention-weighted label readout, yet is neither cited nor compared against. Joint column/class symmetry may be a distinct contribution here, but the paper needs to explain and evaluate this difference.

The results support enforcing symmetry at similar accuracy; it remains unclear how much this adds beyond existing equivariant PFNs.

Questions for authors
  • What is the main additional mechanistic insight relative to the related work above?
  • How does the invariant model compare with EquiTabPFN conceptually?
1 comment

Thank you for the review. We will cite both papers and EquiTabPFN.

Related work and what we add (Q1). Both papers are useful early looks at how tabular models work, and they are focused on narrow questions. Rules or Exemplars is a workshop study built on one synthetic 2D binary task, and the authors note that real data and regression are future work. Is One Layer Enough? follows how predictions develop across layers. Its main text uses binary tasks on two small benchmarks. As the authors note, the looped-layer experiment is small scale. Neither paper asks how the labeled context is turned into a prediction. Neither retrains a model with a different readout, and neither studies row, column or class symmetry.

Our contribution is a controlled pretraining study of multiple TabICLv2 variants. All use the same data generator and the same training budget. It gives three findings.

  1. The readout is not fixed by the architecture. On the released TabICLv2, prototype, kNN and linear readouts come close to the native model, and a frozen label vote does not. But when we pretrain the same backbone from scratch with a vote head or a prototype head, both reach the level of the reference (Fig. 1A, Fig. 4). So the preference of a frozen checkpoint comes from what it happened to learn. It is not a limit of the architecture, and we do not claim a single true mechanism.
  2. Some design choices matter much more than others. Removing depth or the label-injection pathway loses on most tasks. Swapping the readout, or removing RoPE and grouping, stays within two tasks of parity. This matches the history of the field. Going from generic models, such as an LLM reading serialized tables, to attention over rows and columns gave the big gain. Later refinements, including exact invariance, give much less.
  3. Exact symmetry costs no accuracy. The fully invariant model is invariant to row and column order and equivariant to class relabeling. It matches the reference on 38 tasks. We will write "match" instead of "match or outperform".

We agree that Mitra does not clearly favor vote or prototype, and we describe it only as loosely vote-like. This fits our main result, since pretraining the same backbone with either a vote head or a prototype head reaches the reference level, so the readout depends on what training produced and is not set by the architecture.

Benefit of equivariance. We agree that it is not an accuracy gain, and we do not claim one. A model trained long enough already learns most of the symmetry. Our reference's worst-case change in probabilities is 0.003 under column permutations and 0.004 under class relabeling (Table 19). What exact symmetry adds is a guarantee, and released models lack it. Relabeling classes changes 3.3% of TabPFNv2 predictions and 2.3% of TabICLv2 predictions (Table 8).

EquiTabPFN (Q2). It makes the model equivariant to class relabeling. Each class gets its own token built from one shared embedding, and the output is a weighted average of the training label vectors. The features enter through a single linear projection, so column order is not handled. We share two ideas with it: nothing depends on the class index, and the readout averages labels. There are three differences:

  • We use a TabICLv2 backbone, which pools features within each row.
  • Label values never enter our network, only which rows share a label.
  • We add exact column and row invariance while keeping the feature alignment (see our section on representation collapse).

Depth of analysis. We combine frozen readouts, attention interventions and 13 controlled pretraining runs. The pretraining runs go beyond probing alone. We describe the work as a characterization of behavior plus controlled retraining.

Reviewer_PZHbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how tabular foundation models use labeled context and whether their predictions respect permutation symmetries. Frozen readouts reveal different preferences in TabPFNv2, TabICLv2, and Mitra. The authors then pretrain 13 TabICLv2 variants to examine readout, depth, and symmetry-related design choices. They distinguish joint column invariance from row-wise pooling that can cause representation collapse, and provide a symmetric kernel construction with a consistency guarantee. A model with row and column invariance and class equivariance achieves performance comparable to the reference on 38 TabArena tasks.

Critical strengths
  1. The controlled pretraining experiments are useful. The frozen vote performs poorly on TabICLv2, yet jointly training a vote head recovers reference-level performance. This separates the behavior of a released checkpoint from what its architecture can learn.

  2. The distinction between joint column permutations and independent within-row permutations is clear and useful. The kernel construction and trained invariant model show that preserving feature alignment is compatible with permutation symmetry, providing a concrete direction for architecture design.

Critical weaknesses
  1. The probes establish which readouts fit each checkpoint, but do not identify its native computation. Making attention constant or uniform disrupts several possible mechanisms, so the accuracy drop does not specifically support label voting. The retraining experiments also leave unclear whether alternative heads reshape representations throughout the backbone or mainly affect the final ICL blocks. The mechanistic explanation therefore remains incomplete.

  2. The performance comparison uses one pretraining seed per configuration. Winning 20 of 38 tasks supports comparable observed performance, but does not establish that this result is stable across training runs. The symmetry validation also needs clarification: Section 3.3 describes empirical verification of the invariant model, while Table 19 states that its reported bound was measured before pretraining. The architecture may guarantee symmetry, but the reported results do not clearly verify the final trained prediction pipeline.

Questions for authors
  1. Does changing the prediction head alter representations throughout the backbone or mainly in the final ICL blocks? A layer-wise CKA comparison of the reference, prototype, and vote checkpoints on matched inputs would help clarify this.

  2. Is exact symmetry primarily intended to ensure prediction consistency or to improve generalization? What makes architectural guarantees preferable to approximate symmetry or inference-time wrappers?

  3. With one pretraining seed per variant, how stable is the performance comparison? Was symmetry also verified for the final trained checkpoint under the full inference pipeline, given that Table 19 reports a check performed before training?

1 comment

Thank you for the review and insightful comments.

Probes and the native mechanism. We do not claim that a released model computes a vote or a prototype. We claim that its representation supports certain simple rules, and the pretraining runs show why we are careful. The same backbone trained from scratch with a vote head, a prototype head or the original head reaches the reference level (Fig. 1A, Fig. 4). The readout is therefore not fixed by the architecture, and there is no single native mechanism to isolate. Probing a fixed model and retraining all weights are the two tools available, and we use both.

Uniform attention. We use it as a coarse test of whether a block matters, and we do not read it as evidence for voting. The TabICLv2 results show that no single block carries the readout (median loss 0.9 pp for the first block, 0.0 for the last, 30.0 for all twelve, Fig. 3).

Where the heads change the representation. All weights are trained from the initialization, so we do not claim the change is limited to the last blocks. Which blocks change is an interesting follow-up, and our conclusions do not depend on it.

One seed. Each run is on four GPUs and takes over a week, so several seeds per variant are very expensive. This is also standard for pretraining studies at this scale, all the released models we compare against are single runs too. Our 13 variants help as an informal check. The near-equivalent variants (no RoPE, no RoPE or grouping, vote head, prototype head, fully invariant) all land within AUC .8466 to .8490 and log loss .2481 to .2499 (Table 16). The depth cuts fall well outside that range. We describe the invariant model as matching the reference, not beating it.

Symmetry check. The symmetry is exact by construction for any weights, since no operation depends on row order, column order or class index. We numerically checked the final models to ensure this.

Consistency or generalization, and why build it in. The aim is consistency since released models change 3.3% of TabPFNv2 and 2.3% of TabICLv2 predictions when classes are relabeled (Table 8). A trained model learns most of the symmetry, but only approximately. The built-in version is exact in one pass, with no accuracy cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.