- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies different variations of tabular foundation models that can either use the features directly or use labels propagated through a function of the Gram matrix, which makes the model invariant to rotations of the data. Based on different ablations and experiments, the Gram matrix approach works well and even without nonlinear transformations of the Gram matrix, permuting the rows but not the labels hurts, and the Gram matrix describes the similarities of embeddings of other tabular foundation models reasonably well.
Critical strengths
The approach of the paper is quite unique and lends itself to potentially interesting follow-up questions. It creates "kernelized" tabular foundation models and shows that they generalize better to feature counts not seen during pre-training (although the comparison is not to cell-based architectures which are used by the best models nowadays).
Critical weaknesses
- It is quite difficult to understand what is going on, especially in the second half of the paper, where a lot of details are deferred to the appendix. E.g., I think Section 4 could benefit from some mathematical notation to clarify what is going on: For example, does the label-only variant receive features and if not, what is the point? Does the features-only variant receive labels and if not, what is the point? l. 282: what is "the geometry stream without cross-row propagation"? what is the "identity variant"? is it just receiving the labels? Additionally, the comparisons seem to average over many backbones and different kinds of test cases, which makes them harder to interpret.
- I find some of the choices a bit unusual and I'm not sure what to take away from the paper. First, IMHO Proposition 2 is trivial and should not be a proposition. And IMHO it is obvious that permuting rows without permuting labels is going to destroy predictive performance, and I don't see the point in doing an experiment on it. Second, why is the main evaluation only on OOD tasks? (It is not so clear from the main paper that they are meta-OOD rather than OOD.) Third, as we know from the Grinsztajn benchmark paper (https://papers.neurips.cc/paper_files/paper/2022/file/0378c7692da36807bdec87ab043cdadc-Paper-Datasets_and_Benchmarks.pdf), real tabular datasets are often axis-aligned and not rotation-invariant, so I'm not sure if the premise of constructing a rotation-invariant method is that great, and whether the row-geometry model seems good mainly because of the synthetic nature of the main benchmark and because the real-data comparisons (e.g. Table 1) are only to other weak methods like KRR and kNN. If the row-geometry model was as good as the plots suggest, why did the authors not pretrain it on e.g. the TabICLv2 prior and evaluate it on an established benchmark? Fourth, taking functions of the full (train+test) Gram matrix makes the predictions on test samples depend on other present test samples, which regular TFMs avoid. It is unclear what effect this has. If the spectral functions were only applied to the train part of the matrix, then algorithms like linear regression could be realized through spectral functions like . Spectral functions could also be learned with an MLP instead of just using a fixed bank of functions. (And, if one removes the elementwise square operation, then spectral functions can be applied in O(N^2d) instead of O(N^3) by using the SVD of .)
Questions for authors
See above. Some minor comments:
- l. 247-249: The formulation is a bit hard to understand
- l. 293: is any of those backbones related to other TFMs? (except MotherNet)
- l. 225: say that is a pre-trained in-context predictor or so (otherwise it's a bit unclear whether this is a TFM or something else)
- l. 296: it says the geometry model "receives the Gram matrix through the geometry stream [...]" but in Eq. (5) the Gram matrix processing is not part of the model
- ll. 446 is not understandable without more context
- The current standardization approach erases norm information and can therefore not work for . It could help to normalize instead.
15 comments
Author Responses (Part 2)
Regarding the main takeaway (W2). Thank you for the opportunity to state the takeaway directly. The paper asks what tabular in-context models use from the relations between rows, and finds that (i) the Gram matrix and the support labels suffice for prediction in our models, without direct access to the raw features (Section 4.2); (ii) the models use through first-order propagation of the support labels along correctly attached support-query relations (Section 4.3); and (iii) pretrained TabPFN [4] and TabICL [3] rely on correctly attached support-query relations in their own attention, along which first-order propagation recovers their predictions (Section 4.5 and Table 2; Appendix I, Table 31). Following your comment, Section 4 now opens with these three findings.
Regarding Proposition 2 and the row-permutation test (W2). Thank you for this comment. Following it, we now state this result as a lemma (Lemma 2, formerly Proposition 2), and the text after the lemma describes its role. The lemma identifies the block of the row operator that carries the support labels to the queries, and shows that attaching the same relations to other rows gives the query signal , which equals when every attached label is kept and can change when the attached labels change, while every permutation-invariant summary of the operator, such as its spectrum and its norm, stays unchanged. This separates two ways in which a trained model could use : through the support row that each relation points to, or through permutation-invariant summaries of together with the feature tokens that the model also receives. The lemma yields tests that change one of these at a time:
- The same-spectrum misaligned operator keeps the eigenvalues and the norm of the operator and changes which rows are connected. It lowers the accuracy of the trained models to 50.75% (Section 4.3), whereas a model that relied on the spectrum would keep its accuracy.
- Permuting the support rows within their classes keeps the accuracy, and changing the class attached to a fraction of the support rows lowers it (Table 33):
| Operator of the six trained models of Table 9 (Table 7 in the original submission) | OOD accuracy (%) |
|---|---|
| Learned operator, evaluated in this test | 63.14 |
| Support rows permuted within their classes () | 63.14 |
| Class changed for | 60.34 |
| Class changed for | 57.11 |
| Class changed for | 50.43 |
| Class changed for (every attached label inverted) | 37.94 |
| Support-query block shuffled | 50.21 |
| Support-support block shuffled | 58.87 |
- Inside pretrained TabPFN and TabICL, attaching the attention of each query to randomly permuted support rows lowers the AUC on all 25 tables, whereas permuting the support rows within each class retains 72.1% and 79.3% (Table 2).
Therefore, the trained models use the label attached to each support-query relation: their accuracy decreases steadily with the fraction of changed labels, from 63.14% to 37.94%, whereas a model that relied on permutation-invariant summaries of or on its feature tokens would keep its accuracy. Shuffling the support-support block alone lowers the accuracy less (58.87%), because it changes the signal of the support rows that the backbone reads, and not the labels attached to the queries.
Regarding the evaluation on feature-space OOD tasks (W2). The main evaluation is out of distribution because the paper asks what survives a change of feature space. Following your question, Section 4.1 now states that every evaluation task is a new task with its own support and query rows, and that the OOD suites also change the raw features: they use more features than any training task, rotate them, or add irrelevant features. These shifts of the task distribution make the evaluation on the OOD suites meta-OOD. Section 4.1 also defines the IID tasks, drawn independently from the training distribution, and points to their results, where the geometry models reach 64.9%, compared with 53.5% for the token model (Table 13, Table 11 in the original submission, and the IID column of Table 25), and the new evaluation on OpenML-CC18 uses its official splits (Table 35).
Author Responses (Part 3)
Regarding axis-aligned real data, baselines, and an established benchmark (W2). Thank you for raising the connection to the benchmark of Grinsztajn et al. [5], which we now discuss in Section 3.1. Canonicalization standardizes each column separately, and therefore an orthogonal transformation of the raw features can change . The rotation suite of Section 4.1 evaluates the models under such transformations. Section 3.1 now states that, like natural tabular data [5], is not invariant to rotations that mix the raw features, and Appendix I illustrates this on an example (Table 38): shifting a feature, changing its unit, flipping its sign, or permuting the features leaves unchanged, whereas a rotation that mixes the features before canonicalization changes entries of by up to 1.43. Following your suggestions, we added gradient-boosted trees, which are not invariant to rotations, an RBF SVM, and kernel ridge regression on the 25 natural tables (Table 22), and pretrained the models on the TabICLv2 prior [1], as you proposed, and on class-imbalanced synthetic tasks (Tables 30 and 32):
| 25 natural tables, accuracy (%) | |
|---|---|
| Gradient-boosted trees | 78.8 |
| Logistic regression | 79.8 |
| RBF SVM | 80.1 |
| RBF kernel ridge regression | 80.3 |
| KRR on | 80.0 |
| Token model, TabICLv2 prior | 69.8 |
| Geometry model, TabICLv2 prior | 78.9 |
| Token model, class-imbalanced synthetic tasks | 74.3 |
| Geometry model, class-imbalanced synthetic tasks | 78.9 |
KRR on exceeds gradient-boosted trees by 1.2% ([0.3, 2.1], on 17 of the 25 tables), and the geometry models trained on these priors reach the accuracy of gradient-boosted trees on these tables, improving on the token models by 9.0% and 4.7% ([3.9, 15.0] and [2.1, 7.7]). We then evaluated them on an established benchmark, OpenML-CC18 [2], with its official 10-fold splits, on the 60 of its 72 tasks with at most 10 classes and 500 encoded features, with the same 256 rows of each training fold as context for every method (Table 35):
| OpenML-CC18 | Accuracy (%), 60 datasets | Average rank, 60 datasets | Accuracy (%), 38 new datasets | Average rank, 38 new datasets |
|---|---|---|---|---|
| Gradient-boosted trees | 80.81 | 5.42 | 79.70 | 5.26 |
| Logistic regression | 80.77 | 5.22 | 79.27 | 5.89 |
| Polynomial-kernel ridge regression | 81.21 | 5.03 | 80.20 | 4.72 |
| RBF SVM | 81.75 | 4.09 | 80.74 | 4.09 |
| RBF kernel ridge regression | 81.67 | 3.83 | 80.74 | 3.45 |
| KRR on | 81.36 | 4.52 | 80.30 | 4.37 |
| KRR with the exponential function of | 81.61 | 4.04 | 80.49 | 4.21 |
| Geometry model with support-only operators (Ours) | 81.65 | 3.85 | 80.52 | 4.00 |
Our geometry model with the support-only operators that you suggested (Table 29), trained on the TabICLv2 prior and on real OpenML tables that share no dataset with OpenML-CC18, is within 0.1% of the RBF SVM and of RBF kernel ridge regression, and above gradient-boosted trees on 42 of the 60 datasets. Inspired by your comment, we also report the 38 OpenML-CC18 datasets that are not among the 25 natural tables of the paper, which the paper did not use before. On these datasets, our geometry model reaches 80.52%, within 0.25% of the RBF SVM and of RBF kernel ridge regression (80.74%), above logistic regression by 1.25%, and above gradient-boosted trees on 27 of the 38 datasets. Following your comment, Section 4.4 now also states that we use the models trained on synthetic data to compare geometry and token models under the same training protocol, and Section 4.5 analyzes established TFMs directly, now with interventions on their own attention (Table 2).
Author Responses (Part 1)
Thank you for the careful reading, for finding the approach of the paper "quite unique", and for noting that it "lends itself to potentially interesting follow-up questions". We are also glad that you noted that our models "generalize better to feature counts not seen during pre-training".
In short: your review asks what the reader should take away, and whether it holds beyond synthetic tasks. The takeaway is that our models predict through first-order propagation of the support labels along correctly attached row relations, and that the pretrained TFMs that we tested rely on correctly attached relations in their own attention, along which first-order propagation recovers their predictions. The new results show this inside frozen TabPFN and TabICL, and evaluate the geometry model on an established benchmark. Following your suggestions, we (i) defined every input condition in a table and in the notation of Equation (5) (W1); (ii) pretrained the models on the TabICLv2 prior [1], as you suggested, and evaluated them on an established benchmark, OpenML-CC18 [2], with its official splits, next to gradient-boosted trees, logistic regression, an RBF SVM, and kernel ridge regression (W2); (iii) implemented your support-only spectral function 1/(λ+ρ), as a replacement and inside trained models, and an MLP spectral function (W2); and (iv) evaluated the Gram matrix of (x,1) at small d (m6).
Regarding cell-based architectures. Thank you for pointing to cell-based architectures. One of our six backbones is axial cell attention, which attends along the rows and the feature cells, as in TabICL [3], and Appendix B.2 now relates each backbone to existing models. On the feature-count suite, which varies the number of features beyond the training range, this cell-based backbone improves from 52.8% with feature tokens to 55.7% with K (Table 25, below).
Regarding the notation of Section 4 (W1). Thank you for the concrete suggestions, which helped us make Section 4 easier to follow. Every condition receives the support labels as label embeddings on the support rows, and the query rows carry an unknown-label embedding. Table 4 in Appendix B.2 now defines every condition, and Section 4.2 now states it in the notation of Equation (5):
The label-only condition reaches 50.0%, the level of random guessing, and is the reference for the conditions that add row information. The identity variant of l. 282 is the geometry stream without cross-row propagation: it receives feature tokens and the geometry stream with g(λ)=1, the identity operator, so that each row receives its own label signal. It separates the effect of the stream from the effect of propagation.
Regarding the averages over backbones and suites (W1). Following your comment, Table 25 in Appendix I now reports each backbone on each suite, for the token (T) and geometry (G) models:
The geometry model improves on the token model in all 24 combinations of backbone and suite.