- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates how tabular foundation models use labeled context examples to make predictions and how architectural choices affect permutation invariance. The authors analyze pretrained TabPFNv2, TabICLv2, and Mitra checkpoints by testing how well simple prediction rules, such as weighted label voting and nearest-class prototypes, reproduce the models' predictions. They also modify attention computations in selected Transformer layers to assess their importance for predictive performance. They find that TabPFNv2's predictions are better approximated by attention-based label voting, whereas TabICLv2 learns representations that work well with simple classifiers, such as nearest-class prototypes and nearest neighbors.
The paper also investigates whether predictions remain consistent when rows or columns are reordered or class labels are renamed. The authors investigate ways to enforce these properties through inference-time modifications and architectural changes. They also provide a mathematical example showing that permutation invariance can be achieved without losing important predictive information. Through controlled pretraining of 13 TabICLv2 variants and evaluation on 38 TabArena tasks, they demonstrate that a model enforcing these symmetries can achieve predictive performance comparable to the reference architecture.
Critical strengths
-
Demonstrating that permutation symmetries can be enforced without sacrificing predictive performance. The paper shows that a modified TabICLv2 architecture can guarantee invariance to row and column permutations and equivariance to class relabeling while achieving performance comparable to the reference model. This is a meaningful contribution because these properties are desirable for tabular learning, yet existing models do not consistently satisfy them. The controlled pretraining experiments provide convincing evidence that enforcing these symmetries need not come at the expense of predictive accuracy.
-
Insights into how tabular foundation models use labeled context. The comparison of TabPFNv2, TabICLv2, and Mitra reveals interesting differences in their learned representations and prediction behavior. In particular, TabICLv2's representations support accurate classification using simple prototype and nearest-neighbor methods, whereas TabPFNv2 is better approximated by attention-based label voting.
Critical weaknesses
In Section 2.1, the authors extract hidden representations from a pretrained TabICLv2 model and use them to classify new observations. They compute the average representation of each class (the class prototype) and assign each query to the closest prototype. This simple classifier achieves 85.4% accuracy, close to the native TabICLv2 model's 86.4%. However, nearest-neighbor and linear classifiers also perform well, achieving 85.6% and 85.9%, respectively (Figure 2). These results show that TabICLv2 learns useful representations, but they do not establish that the model itself uses prototype classification.
Furthermore, Table 11 reports a large difference in expected calibration error between the prototype classifier (46.3%) and the native model (4.2%). The prototype probabilities are not calibrated, so this difference alone does not rule out a prototype-based mechanism.
Two additional experiments could help clarify this issue:
A. Compare predicted probabilities. The authors could first calibrate the prototype classifier on validation data, then compare its predicted probabilities with those of the native model on the same test observations. Close agreement would provide stronger evidence that the two methods behave similarly beyond simply predicting the same class.
B. Test whether individual examples matter beyond their class averages. The authors could perform controlled interventions that change individual context representations while preserving the average representation of each class. For example, two representations could change from 0 and 2 to 0.9 and 1.1. Both pairs have the same average of 1. A prototype classifier would treat them identically if the query representation and other class prototypes were unchanged. If the native model's predictions change substantially under a properly controlled intervention, this would suggest that it uses information beyond class prototypes.
These experiments would help distinguish between a model whose representations support prototype classification and one whose predictions actually depend on class prototypes. This distinction is important given the paper's goal of understanding how tabular foundation models make predictions.
Questions for authors
Do the learned representations reveal the model's actual prediction mechanism?
In Section 2.1, the prototype classifier achieves 85.4% accuracy, close to TabICLv2's native accuracy of 86.4%. However, kNN and linear classifiers also achieve similar accuracy (85.6% and 85.9%, respectively). Furthermore, Table 11 reports a much higher calibration error for the prototype classifier (46.3%) than for the native model (4.2%).
These results show that TabICLv2 learns representations that work well with several simple classifiers. However, it remains unclear whether the native model actually uses a prototype-based prediction mechanism.
Could the authors clarify what evidence supports this stronger interpretation? In particular, could they compare the native and prototype predicted probabilities after calibrating the prototype scores?
A further experiment could test whether the native predictions depend on individual context representations beyond their class averages. For example, changing two same-class embeddings from 0 and 2 to 0.9 and 1.1 preserves their average of 1. A pure prototype classifier would give the same prediction if the query and other class prototypes remained unchanged. A carefully controlled intervention could test whether TabICLv2 behaves similarly.
Such evidence would help determine whether the results reveal the model's actual prediction mechanism or mainly demonstrate that its representations support accurate prototype classification.
Thank you for the careful reading and for the constructive feedback. Your central point is a fair and important one. Good prototype, kNN and linear readouts show that the representation is useful, and they do not by themselves show that the model computes a prototype. We agree, and what we claim in Section 2.1 is a statement about the representation, which supports accurate class choices.
Representation vs mechanism. The pretraining experiment is our strongest evidence on this question. We trained TabICLv2 from scratch with a prototype head and with a vote head, and both match the reference model, with the vote head slightly ahead. The same architecture can therefore be trained to support several different readouts.
Calibration. We ran the calibrated comparison you suggest. The 46.3% against 4.2% gap in Table 11 comes from the scale of the prototype scores. Cosine similarities lie in [-1, 1], so a unit-temperature softmax gives nearly uniform probabilities. We fitted a single temperature on held-out calibration rows, which the model treated as queries and never saw labels for, and evaluated on separate test rows. We used 38 TabArena classification tasks with 5 seeds each, 190 task-seed pairs in total. The prototype ECE falls from 0.380 to 0.034, close to the native model's 0.026, and the log loss falls from 0.753 to 0.329, against 0.283 for native. The fitted temperature transfers well from calibration rows to test rows, and a temperature fitted on the test rows themselves gives almost the same numbers.
Probabilities beyond class agreement. After calibration, the prototype and native probabilities are close on the same test rows, with a mean total variation distance of 0.075 and a mean KL divergence of 0.052. They are not identical. The two agree on the predicted class for 94% of rows, and the prototype is 1.9 accuracy points below native. A prototype readout on this representation is therefore a very good approximation of the native output, which is the claim we make about the representation. It does not show that the model computes a prototype, and we do not claim that.
Intervention on class averages. This is a good test of whether one released checkpoint depends on class averages. The pretraining experiment speaks to the broader question. Prototype and vote heads trained from scratch both match the reference, so reference-level performance does not depend on one particular readout. Our claim is about what the representation supports, and the calibrated comparison above addresses that directly.
Revisions. We will add the calibrated comparison to the appendix, and we will state the distinction between what the representation supports and what the model computes explicitly in Section 2.1.