acceptodds
Under review as a conference paper at ICLR 2027

Row Geometry for Tabular In-Context Learning: What Do the Models Use?

32 views

Abstract

Tabular foundation models perform in-context prediction by relating queries to a labeled support set, across tables whose columns differ in number, units, and meaning, so that relations between rows provide a shared structure. We study whether support-to-query prediction can be carried by these relations alone, and which part the models use. Our approach constructs the row Gram matrix of each task from the raw input features standardized with support statistics, and uses it as a row operator that propagates support labels to queries. This formulation connects tabular in-context learning with label propagation, admits several choices of geometry and propagation, and allows changing the row operator while the trained model stays fixed. Across six architectures, the Gram matrix can replace direct access to the raw input features, and changing only the row operator shows that these models rely on first-order propagation along correctly attached support–query relations. We further characterize the Gram matrix as the maximal orthogonal invariant of the normalized rows, show that it remains predictive for classical predictors on 25 OpenML tables, and find closely related internal row structure in frozen TabPFN and TabICL; adding it to TabPFN is neutral on natural tables and helps where the frozen model is weak. These results identify row geometry as a useful object for understanding and designing in-context learning over tabular data.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
tabular foundation models, in-context learning, tabular learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yWs1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies different variations of tabular foundation models that can either use the features directly or use labels propagated through a function of the Gram matrix, which makes the model invariant to rotations of the data. Based on different ablations and experiments, the Gram matrix approach works well and even without nonlinear transformations of the Gram matrix, permuting the rows but not the labels hurts, and the Gram matrix describes the similarities of embeddings of other tabular foundation models reasonably well.

Critical strengths

The approach of the paper is quite unique and lends itself to potentially interesting follow-up questions. It creates "kernelized" tabular foundation models and shows that they generalize better to feature counts not seen during pre-training (although the comparison is not to cell-based architectures which are used by the best models nowadays).

Critical weaknesses
  • It is quite difficult to understand what is going on, especially in the second half of the paper, where a lot of details are deferred to the appendix. E.g., I think Section 4 could benefit from some mathematical notation to clarify what is going on: For example, does the label-only variant receive features and if not, what is the point? Does the features-only variant receive labels and if not, what is the point? l. 282: what is "the geometry stream without cross-row propagation"? what is the "identity variant"? is it just receiving the labels? Additionally, the comparisons seem to average over many backbones and different kinds of test cases, which makes them harder to interpret.
  • I find some of the choices a bit unusual and I'm not sure what to take away from the paper. First, IMHO Proposition 2 is trivial and should not be a proposition. And IMHO it is obvious that permuting rows without permuting labels is going to destroy predictive performance, and I don't see the point in doing an experiment on it. Second, why is the main evaluation only on OOD tasks? (It is not so clear from the main paper that they are meta-OOD rather than OOD.) Third, as we know from the Grinsztajn benchmark paper (https://papers.neurips.cc/paper_files/paper/2022/file/0378c7692da36807bdec87ab043cdadc-Paper-Datasets_and_Benchmarks.pdf), real tabular datasets are often axis-aligned and not rotation-invariant, so I'm not sure if the premise of constructing a rotation-invariant method is that great, and whether the row-geometry model seems good mainly because of the synthetic nature of the main benchmark and because the real-data comparisons (e.g. Table 1) are only to other weak methods like KRR and kNN. If the row-geometry model was as good as the plots suggest, why did the authors not pretrain it on e.g. the TabICLv2 prior and evaluate it on an established benchmark? Fourth, taking functions of the full (train+test) Gram matrix makes the predictions on test samples depend on other present test samples, which regular TFMs avoid. It is unclear what effect this has. If the spectral functions were only applied to the train part of the matrix, then algorithms like linear regression could be realized through spectral functions like λ↦1/(λ+ρ)\lambda \mapsto 1/(\lambda + \rho). Spectral functions could also be learned with an MLP instead of just using a fixed bank of functions. (And, if one removes the elementwise square operation, then spectral functions can be applied in O(N^2d) instead of O(N^3) by using the SVD of Uˉ\bar U.)
Questions for authors

See above. Some minor comments:

  • l. 247-249: The formulation is a bit hard to understand
  • l. 293: is any of those backbones related to other TFMs? (except MotherNet)
  • l. 225: say that RθR_\theta is a pre-trained in-context predictor or so (otherwise it's a bit unclear whether this is a TFM or something else)
  • l. 296: it says the geometry model "receives the Gram matrix through the geometry stream [...]" but in Eq. (5) the Gram matrix processing is not part of the model RθR_\theta
  • ll. 446 is not understandable without more context
  • The current standardization approach erases norm information and can therefore not work for d=1d=1. It could help to normalize (x,1)(x, 1) instead.
15 comments
AuthorsOpenReview ↗

Author Responses (Part 1)

Thank you for the careful reading, for finding the approach of the paper "quite unique", and for noting that it "lends itself to potentially interesting follow-up questions". We are also glad that you noted that our models "generalize better to feature counts not seen during pre-training".

In short: your review asks what the reader should take away, and whether it holds beyond synthetic tasks. The takeaway is that our models predict through first-order propagation of the support labels along correctly attached row relations, and that the pretrained TFMs that we tested rely on correctly attached relations in their own attention, along which first-order propagation recovers their predictions. The new results show this inside frozen TabPFN and TabICL, and evaluate the geometry model on an established benchmark. Following your suggestions, we (i) defined every input condition in a table and in the notation of Equation (5) (W1); (ii) pretrained the models on the TabICLv2 prior [1], as you suggested, and evaluated them on an established benchmark, OpenML-CC18 [2], with its official splits, next to gradient-boosted trees, logistic regression, an RBF SVM, and kernel ridge regression (W2); (iii) implemented your support-only spectral function 1/(λ+ρ)1/(\lambda+\rho), as a replacement and inside trained models, and an MLP spectral function (W2); and (iv) evaluated the Gram matrix of (x,1)(x,1) at small dd (m6).

Regarding cell-based architectures. Thank you for pointing to cell-based architectures. One of our six backbones is axial cell attention, which attends along the rows and the feature cells, as in TabICL [3], and Appendix B.2 now relates each backbone to existing models. On the feature-count suite, which varies the number of features beyond the training range, this cell-based backbone improves from 52.8% with feature tokens to 55.7% with KK (Table 25, below).

Regarding the notation of Section 4 (W1). Thank you for the concrete suggestions, which helped us make Section 4 easier to follow. Every condition receives the support labels as label embeddings on the support rows, and the query rows carry an unknown-label embedding. Table 4 in Appendix B.2 now defines every condition, and Section 4.2 now states it in the notation of Equation (5):

Input conditionPredictorFeature tokens of XXGeometry stream ZKZ_KPropagation along KK
Label-onlyRθ(Y)R_\theta(Y)nonono
Features-only (token model)Rθ(X,Y)R_\theta(X,Y)yesnono
Geometry-onlyRθ(ZK,Y)R_\theta(Z_K,Y)noyesyes
Features+geometry (geometry model)Rθ(X,ZK,Y)R_\theta(X,Z_K,Y)yesyesyes
Identity variantRθ(X,ZI,Y)R_\theta(X,Z_I,Y)yesyes, with g(λ)=1g(\lambda)=1no

The label-only condition reaches 50.0%, the level of random guessing, and is the reference for the conditions that add row information. The identity variant of l. 282 is the geometry stream without cross-row propagation: it receives feature tokens and the geometry stream with g(λ)=1g(\lambda)=1, the identity operator, so that each row receives its own label signal. It separates the effect of the stream from the effect of propagation.

Regarding the averages over backbones and suites (W1). Following your comment, Table 25 in Appendix I now reports each backbone on each suite, for the token (T) and geometry (G) models:

Accuracy (%), T / GIIDFeature countRotationIrrelevant
Cell Attention55.3 / 65.252.8 / 55.756.8 / 69.854.4 / 63.4
Kernel Regression52.6 / 62.248.1 / 54.153.9 / 66.051.0 / 60.9
MotherNet52.7 / 65.352.9 / 56.452.7 / 69.852.6 / 64.0
Perceiver52.7 / 65.652.8 / 56.252.7 / 69.852.8 / 64.0
Prototype50.2 / 65.649.9 / 56.450.4 / 69.750.2 / 64.1
Retrieval57.6 / 65.652.9 / 56.359.5 / 69.755.7 / 64.0

The geometry model improves on the token model in all 24 combinations of backbone and suite.

AuthorsOpenReview ↗

Author Responses (Part 2)

Regarding the main takeaway (W2). Thank you for the opportunity to state the takeaway directly. The paper asks what tabular in-context models use from the relations between rows, and finds that (i) the Gram matrix KK and the support labels suffice for prediction in our models, without direct access to the raw features (Section 4.2); (ii) the models use KK through first-order propagation of the support labels along correctly attached support-query relations (Section 4.3); and (iii) pretrained TabPFN [4] and TabICL [3] rely on correctly attached support-query relations in their own attention, along which first-order propagation recovers their predictions (Section 4.5 and Table 2; Appendix I, Table 31). Following your comment, Section 4 now opens with these three findings.

Regarding Proposition 2 and the row-permutation test (W2). Thank you for this comment. Following it, we now state this result as a lemma (Lemma 2, formerly Proposition 2), and the text after the lemma describes its role. The lemma identifies the block GQSG_{QS} of the row operator that carries the support labels to the queries, and shows that attaching the same relations to other rows gives the query signal GQSPS⊤YSG_{QS}P_S^\top Y_S, which equals GQSYSG_{QS}Y_S when every attached label is kept and can change when the attached labels change, while every permutation-invariant summary of the operator, such as its spectrum and its norm, stays unchanged. This separates two ways in which a trained model could use KK: through the support row that each relation points to, or through permutation-invariant summaries of KK together with the feature tokens that the model also receives. The lemma yields tests that change one of these at a time:

  • The same-spectrum misaligned operator keeps the eigenvalues and the norm of the operator and changes which rows are connected. It lowers the accuracy of the trained models to 50.75% (Section 4.3), whereas a model that relied on the spectrum would keep its accuracy.
  • Permuting the support rows within their classes keeps the accuracy, and changing the class attached to a fraction ff of the support rows lowers it (Table 33):
Operator of the six trained models of Table 9 (Table 7 in the original submission)OOD accuracy (%)
Learned operator, evaluated in this test63.14
Support rows permuted within their classes (f=0f=0)63.14
Class changed for f=0.125f=0.12560.34
Class changed for f=0.25f=0.2557.11
Class changed for f=0.5f=0.550.43
Class changed for f=1f=1 (every attached label inverted)37.94
Support-query block shuffled50.21
Support-support block shuffled58.87
  • Inside pretrained TabPFN and TabICL, attaching the attention of each query to randomly permuted support rows lowers the AUC on all 25 tables, whereas permuting the support rows within each class retains 72.1% and 79.3% (Table 2).

Therefore, the trained models use the label attached to each support-query relation: their accuracy decreases steadily with the fraction of changed labels, from 63.14% to 37.94%, whereas a model that relied on permutation-invariant summaries of KK or on its feature tokens would keep its accuracy. Shuffling the support-support block alone lowers the accuracy less (58.87%), because it changes the signal of the support rows that the backbone reads, and not the labels attached to the queries.

Regarding the evaluation on feature-space OOD tasks (W2). The main evaluation is out of distribution because the paper asks what survives a change of feature space. Following your question, Section 4.1 now states that every evaluation task is a new task with its own support and query rows, and that the OOD suites also change the raw features: they use more features than any training task, rotate them, or add irrelevant features. These shifts of the task distribution make the evaluation on the OOD suites meta-OOD. Section 4.1 also defines the IID tasks, drawn independently from the training distribution, and points to their results, where the geometry models reach 64.9%, compared with 53.5% for the token model (Table 13, Table 11 in the original submission, and the IID column of Table 25), and the new evaluation on OpenML-CC18 uses its official splits (Table 35).

AuthorsOpenReview ↗

Author Responses (Part 3)

Regarding axis-aligned real data, baselines, and an established benchmark (W2). Thank you for raising the connection to the benchmark of Grinsztajn et al. [5], which we now discuss in Section 3.1. Canonicalization standardizes each column separately, and therefore an orthogonal transformation of the raw features can change KK. The rotation suite of Section 4.1 evaluates the models under such transformations. Section 3.1 now states that, like natural tabular data [5], KK is not invariant to rotations that mix the raw features, and Appendix I illustrates this on an example (Table 38): shifting a feature, changing its unit, flipping its sign, or permuting the features leaves KK unchanged, whereas a rotation that mixes the features before canonicalization changes entries of KK by up to 1.43. Following your suggestions, we added gradient-boosted trees, which are not invariant to rotations, an RBF SVM, and kernel ridge regression on the 25 natural tables (Table 22), and pretrained the models on the TabICLv2 prior [1], as you proposed, and on class-imbalanced synthetic tasks (Tables 30 and 32):

25 natural tables, accuracy (%)
Gradient-boosted trees78.8
Logistic regression79.8
RBF SVM80.1
RBF kernel ridge regression80.3
KRR on KK80.0
Token model, TabICLv2 prior69.8
Geometry model, TabICLv2 prior78.9
Token model, class-imbalanced synthetic tasks74.3
Geometry model, class-imbalanced synthetic tasks78.9

KRR on KK exceeds gradient-boosted trees by 1.2% ([0.3, 2.1], on 17 of the 25 tables), and the geometry models trained on these priors reach the accuracy of gradient-boosted trees on these tables, improving on the token models by 9.0% and 4.7% ([3.9, 15.0] and [2.1, 7.7]). We then evaluated them on an established benchmark, OpenML-CC18 [2], with its official 10-fold splits, on the 60 of its 72 tasks with at most 10 classes and 500 encoded features, with the same 256 rows of each training fold as context for every method (Table 35):

OpenML-CC18Accuracy (%), 60 datasetsAverage rank, 60 datasetsAccuracy (%), 38 new datasetsAverage rank, 38 new datasets
Gradient-boosted trees80.815.4279.705.26
Logistic regression80.775.2279.275.89
Polynomial-kernel ridge regression81.215.0380.204.72
RBF SVM81.754.0980.744.09
RBF kernel ridge regression81.673.8380.743.45
KRR on KK81.364.5280.304.37
KRR with the exponential function of KK81.614.0480.494.21
Geometry model with support-only operators (Ours)81.653.8580.524.00

Our geometry model with the support-only operators that you suggested (Table 29), trained on the TabICLv2 prior and on real OpenML tables that share no dataset with OpenML-CC18, is within 0.1% of the RBF SVM and of RBF kernel ridge regression, and above gradient-boosted trees on 42 of the 60 datasets. Inspired by your comment, we also report the 38 OpenML-CC18 datasets that are not among the 25 natural tables of the paper, which the paper did not use before. On these datasets, our geometry model reaches 80.52%, within 0.25% of the RBF SVM and of RBF kernel ridge regression (80.74%), above logistic regression by 1.25%, and above gradient-boosted trees on 27 of the 38 datasets. Following your comment, Section 4.4 now also states that we use the models trained on synthetic data to compare geometry and token models under the same training protocol, and Section 4.5 analyzes established TFMs directly, now with interventions on their own attention (Table 2).

Reviewer_MvQDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether tabular in-context learning can be explained through sample-wise geometry, using pairwise relationships between samples rather than direct access to raw input features. To test this hypothesis, the authors construct a Gram matrix KK and introduce a geometry stream that propagates support labels through spectral transformations of KK. The resulting embeddings are then fed into six custom-trained backbones to investigate whether explicit row geometry can replace direct access to raw features. The experiments show that geometry-only models perform comparably to models receiving both geometry and feature streams. The authors also find that a simple first-order propagation operator of the form aI+bKaI+bK achieves nearly the same performance as the more complicated learned spectral operators. Additional experiments show that certain layers within pretrained TabPFN and TabICL exhibit high CKA similarity with the input Gram matrix. Injecting explicit geometry into pretrained models also improves performance on certain synthetically constructed tasks, although it does not improve performance on the evaluated real-world benchmark datasets. Overall, the paper focuses on mechanistic interpretability. Its approach is to construct controlled tabular ICL models with explicit geometric representations, investigate which components are important for prediction, and examine whether the resulting insights can be extended to pretrained TabPFN and TabICL.

Critical strengths
  1. The experimental analysis is systematic and includes extensive controlled experiments across multiple architectures, synthetic tasks, and feature-space distribution shifts.

  2. The perspective of understanding tabular ICL through row geometry and label propagation is interesting and potentially useful for the future analysis and design of tabular foundation models.

  3. The finding that first-order label propagation is nearly sufficient in the controlled architectures is interesting, suggesting that relatively simple similarity-based mechanisms may explain much of the predictive behavior of these models.

Critical weaknesses
  1. [Major] The central mechanistic conclusions are not established for pretrained TFMs. The strongest causal experiments are conducted on custom-trained architectures explicitly equipped with a geometry-based label-propagation pathway. Consequently, showing that these models rely on correctly aligned propagation, or that first-order propagation is nearly sufficient, does not establish that pretrained TFMs use the same mechanism. The experiments on TabPFN and TabICL mainly demonstrate representational similarity through CKA and sensitivity to externally introduced geometry biases. We still do not know whether pretrained TFMs actually implement or rely on first-order label propagation. This leaves a substantial gap between the paper's motivation and its experimental conclusions.

  2. Several design choices remain unclear. For example, the motivation for choosing the particular spectral functions is not sufficiently explained. The task-dependent gating network also relies on six hand-designed statistics, but only four are explicitly identified in the appendix. It is unclear whether these choices are important for the reported findings or whether simpler alternatives would perform comparably.

  3. [Minor] The paper presents many experiments, but the primary findings are difficult to identify. Several experiments mainly serve as sanity checks or establish expected properties of the explicitly constructed label-propagation mechanism. Although these controls are reasonable, the large number of auxiliary experiments makes the paper unnecessarily dense and obscures the main scientific contributions.

  4. The contribution of the neural backbones is not sufficiently isolated. The proposed pipeline introduces several additional components after label propagation, including learned projections and neural backbones. However, since the propagated label representations already contain class-level prediction scores, it is unclear how much additional predictive value these components provide. A simpler baseline using direct label propagation would help clarify which components are actually necessary.

Questions for authors
  1. Why were 11, e−λe^{-\lambda}, and e−λe^{-\sqrt{\lambda}} chosen as the spectral functions? Is there a theoretical or empirical motivation for these choices? In particular, why is g(λ)=λg(\lambda)=\lambda not included in the training-time function bank, given that the simpler operator aI+bKaI+bK achieves comparable performance?

  2. How were the six inputs to the task-dependent parameter prediction network chosen? Appendix B.3 explicitly identifies only four statistics. What are the remaining two? Have the authors evaluated whether these task-dependent coefficients outperform simpler alternatives, such as globally learned coefficients?

  3. Can the authors provide more direct causal evidence that pretrained TabPFN and TabICL use first-order label propagation? The current experiments mainly establish representational similarity rather than the underlying prediction mechanism. For example, could the authors directly intervene on internal row-attention relationships in frozen pretrained models to test whether simple first-order propagation can recover their predictions?

  4. Why are the neural backbones necessary for prediction? Since HH already contains propagated class-level scores, could the authors compare their geometry-augmented models against a simple predictor based directly on KQSYSK_{QS}Y_S? This would help clarify the additional contribution of the backbones beyond explicit label propagation.

5 comments
AuthorsOpenReview ↗

Author Responses (Part 1)

Thank you for the detailed and constructive review. We are glad that you found the experimental analysis "systematic", with "extensive controlled experiments across multiple architectures, synthetic tasks, and feature-space distribution shifts", the perspective of row geometry and label propagation "interesting and potentially useful for the future analysis and design of tabular foundation models", and the first-order finding "interesting".

In short: following your questions, we (i) intervened directly on the row attention of frozen TabPFN and TabICL and recovered their predictions by first-order propagation along their own attention (W1, Q3); (ii) retrained the six backbones with λ/λmax⁡\lambda/\lambda_{\max} in the bank, with globally learned coefficients, and with further designs (W2, Q1, Q2); (iii) compared the trained models with direct propagation KQSYSK_{QS}Y_S on the tasks of Table 1, the IID suite, and the feature-space OOD suites (W4, Q4); and (iv) stated the main findings at the start of Section 4 (W3).

Regarding the mechanism inside pretrained TFMs (W1, Q3). We appreciate the suggestion to intervene directly on the row attention of frozen TFMs. The interventions show that both TabPFN v2 [1] and TabICL [2] rely on correctly attached support-query relations, and that first-order propagation of the support labels along their own attention recovers their predictions. Following your suggestion, we intervened on the attention from the query rows to the support rows, in every layer, on the 25 natural OpenML tables with the protocol of Table 16 (Table 14 in the original submission) (Section 4.5, Table 2):

Query-to-support attention in every layer, AUC (%)TabPFN v2TabICL
Unmodified (as in Table 16)78.882.7
Same weights, attached to randomly permuted support rows58.471.0
Same weights, support rows permuted within each class72.179.3
Weights computed using only KK, wqs∝exp⁡(βKqs)w_{qs}\propto\exp(\beta K_{qs})70.677.1
The same weights with misaligned (shuffled) rows of KK55.765.9
Uniform weights55.872.8

Attaching the weights of each query to the wrong support rows lowers AUC on all 25 tables for each model. Permuting the support rows within each class, which keeps the class of each attended row, retains 72.1% and 79.3%, and weights computed using only KK exceed uniform weights on 20 of the 25 tables for TabPFN v2 and on 24 for TabICL. We then ran both models unmodified and propagated the support labels along their own query-to-support attention, r=WYSr=WY_S (Table 31):

First-order propagation along the attention of the modelTabPFN v2TabICL
Unmodified model, AUC (%)78.882.7
Propagation along the last-layer attention, AUC (%)76.681.6
Propagation along the attention averaged over layers, AUC (%)78.382.1
Agreement of the last-layer propagation with the model (% of query rows)96.293.6
Agreement of propagation along weights computed using only KK with the model (% of query rows)85.185.1

In the revision, Section 4.5 presents these interventions and this readout next to the CKA comparison with KK, and the discussion in Section 4.6 refers to the interventions.

AuthorsOpenReview ↗

Author Responses (Part 2)

Regarding the choice of spectral functions (W2, Q1). We appreciate the question on the bank, and on g(λ)=λg(\lambda)=\lambda in particular. Section 3.3 now states the two reasons for the bank. First, the constant response g(λ)=1g(\lambda)=1 gives the identity operator, which carries no label across rows, and the backbone also receives feature tokens, so that a model may propagate the labels along KK but is not forced to. Second, the heat functions e−λe^{-\lambda} and e−λe^{-\sqrt\lambda} are bounded on the spectrum of KK and nonlinear in its eigenvalues, so that the trained operator is not restricted to first-order propagation. The bank does not contain KK itself, so that first-order propagation is not built into the trained models, and KK enters in Section 4.3 as a replacement. Therefore, the finding that aI+bKaI+bK reaches 62.43%, compared with 63.13% for the learned operator (Table 9, Table 7 in the original submission), is informative. Following your question, we trained the six backbones with g(λ)=λ/λmax⁡g(\lambda)=\lambda/\lambda_{\max} added to the bank, which gives KK and K∘2K^{\circ 2} up to a scalar and is bounded in [0,1][0,1] like the heat functions, and with a first-order bank {1,λ/λmax⁡}\{1,\lambda/\lambda_{\max}\}: they reach 63.07% and 62.75%, compared with 63.13% for the paper design (Table 24, below).

Regarding the inputs and the form of the coefficients (W2, Q2). Thank you for the careful reading of Appendix B.3, which now lists all six inputs: log⁡∣S∣\log|S|, log⁡d\log d, the entropy of the support labels, the log stable rank of the standardized support rows, the spectral entropy of the eigenvalues of KK, and the mean squared value of the standardized support features. We chose them as permutation-invariant statistics of the size, the label balance, and the row geometry of a task. In addition, Appendix B.3 now relates your question to the affine operator aI+bKaI+bK of Section 4.3, whose two scalars are fixed across all tasks and which reaches 62.43%, compared with 63.13% for the learned operator. To address your question directly, we retrained the six backbones with a coefficient vector shared by all tasks (Table 24):

OOD accuracy (%)Paper design (Tables 9 and 10, Tables 7 and 8 in the original submission)Global coefficientsBank with λ/λmax⁡\lambda/\lambda_{\max}MLP spectral functionFirst-order bank
Cell Attention63.063.263.463.562.9
Kernel Regression62.662.762.263.062.2
MotherNet63.363.263.263.362.9
Perceiver63.263.362.963.462.9
Prototype63.463.563.363.362.7
Retrieval63.363.363.463.362.8
Mean63.1363.2263.0763.3162.75

Globally learned coefficients reach 63.22%, compared with 63.13% for the task-dependent coefficients. The global coefficients, the bank with λ/λmax⁡\lambda/\lambda_{\max}, and the MLP spectral function are within 0.5% of the paper design on each backbone, and the first-order bank is within 0.7%. Therefore, the findings are stable across the form of the coefficients and the choice of the bank, in every design that we tested.

AuthorsOpenReview ↗

Author Responses (Part 3)

Regarding the importance of the design choices (W2). To answer this question, we repeated the operator replacements of Section 4.3 on each retrained design (Table 26):

Row operator, OOD accuracy (%)Paper design (Table 9, Table 7 in the original submission)Global coefficientsBank with λ/λmax⁡\lambda/\lambda_{\max}MLP spectral functionFirst-order bank
Learned operator63.1363.2263.0763.3162.75
aI+bKaI+bK62.4362.8062.7662.7662.65
Diagonal operator51.2351.4151.1751.1450.54
Row-shuffled operator50.4550.2750.5950.3850.58

In each retrained design, aI+bKaI+bK reaches at least 62.65%, whereas the diagonal and row-shuffled operators decrease to 50.3% to 51.4%. Therefore, the findings of Section 4.3 hold across these design choices.

Regarding the presentation of the main findings (W3). Thank you for this comment. The revision makes the main findings easier to identify: (i) Section 4 now opens with the three main findings, each with the subsection that establishes it; (ii) Section 4.2 states every input condition in the notation of Equation (5), and Table 4 in Appendix B.2 defines what the predictor receives in each; (iii) Section 4.5 presents the interventions on the attention of frozen TFMs next to the descriptive results; and (iv) the other new results are collected in Appendix I, so that the main text keeps the three findings in front.

Regarding the role of the backbones (W4, Q4). We appreciate the suggestion to compare with direct label propagation, which connects to the first-order finding of Section 4.3. The backbones are the in-context models whose use of KK we study, and their role depends on the setting: on the feature-space OOD suites, the first-order propagated signal alone reaches their accuracy, and on the tasks of Table 1 and the IID suite, the backbones add a learned readout of this signal. Following your suggestion, we evaluated arg⁡max⁡c(KQSYS)c\arg\max_c (K_{QS}Y_S)_c, which has no trained weights, on the same tasks as the retrieval and MotherNet geometry models of Table 1 and as the models of Table 13 (Table 11 in the original submission) on the suites (Appendix I, Table 21):

Mean accuracy (%)Table 1 tasks, retrievalTable 1 tasks, MotherNetIID suiteOOD suites
Direct propagation arg⁡max⁡c(KQSYS)c\arg\max_c (K_{QS}Y_S)_c66.366.362.562.8
Geometry models (Ours)70.670.464.962.8

On the feature-space OOD suites, direct propagation reaches the accuracy of the trained geometry models, which extends the first-order finding of Section 4.3, where aI+bKaI+bK reaches 62.43%, compared with 63.13% for the learned operator. On the tasks of Table 1 and the IID suite, the trained models improve on direct propagation by 4.1% to 4.3% and by 2.4%. The geometry stream with a linear head and no backbone reaches 63.2% on the OOD suites and 63.4% on the IID suite, and the backbones raise the IID accuracy to 64.9% (Table 21). Following your comment, Section 4.3 now states this role of the backbones and refers to these results.

Thank you again for the detailed review, which helped us to improve the quality of our work. We hope that the interventions on frozen TFMs, the retrained designs, and the comparison with direct propagation address your concerns, and that you will consider revising your score. We are happy to discuss any remaining questions.

References:

[1] Accurate Predictions on Small Data with a Tabular Foundation Model
[2] TabICL: A Tabular Foundation Model for In-Context Learning on Large Data

Reviewer_618xOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether tabular in-context learning can be explained through relationships between rows rather than direct access to raw features. The authors construct a support-standardized Gram matrix and introduce a geometry-based propagation framework to analyze how support labels influence query predictions. Experiments across six controlled architectures suggest that row geometry can replace direct feature access and that first-order propagation along correctly aligned support-query relations accounts for most of the predictive gains. The paper further evaluates the representation using classical predictors on OpenML datasets and examines its relationship with internal row structures in pretrained TabPFN and TabICL.

Critical strengths

The paper presents a well-motivated framework for separating row relationships from the predictor, enabling controlled interventions on the propagation operator while keeping model parameters fixed. The operator replacement experiments across six architectures provide useful evidence that correctly aligned support-query relationships are important for prediction in the controlled settings. This experimental design offers a valuable perspective for studying the mechanisms of tabular in-context learning.

Critical weaknesses

Limited evidence supporting the central mechanistic claim for pretrained TFMs: The main causal evidence comes from controlled architectures explicitly equipped with the proposed geometry propagation mechanism. Although these experiments are informative, they do not establish that pretrained TFMs inherently rely on the same first-order propagation mechanism. The additional analyses of TabPFN and TabICL are largely correlational or based on externally injected geometry biases, leaving a substantial gap between the experimental evidence and the paper's broader mechanistic claims.

Questions for authors
  1. The central causal evidence is obtained from controlled models explicitly trained with a geometry propagation stream. While these experiments demonstrate that the proposed architectures rely on this mechanism, it is unclear whether the same conclusion extends to pretrained TFMs such as TabPFN and TabICL. How do the authors establish that first-order propagation is an intrinsic mechanism of pretrained TFMs rather than a consequence of the proposed architecture?

  2. On the 25 natural OpenML datasets, the synthetic-trained geometry models achieve substantially lower accuracy than classical KRR using the same Gram matrix. What explains this generalization gap? Furthermore, comparisons with standard kernel methods operating directly on normalized raw features would help clarify whether the proposed row geometry provides benefits beyond conventional kernel representations.

  3. In the natural-table experiments with frozen TabPFN and TabICL, the aligned geometry bias does not outperform the row-shuffled bias. This appears inconsistent with the paper's central claim that correct row correspondence is essential for prediction. How do the authors reconcile these results with the conclusions drawn from the controlled-model experiments?

6 comments
AuthorsOpenReview ↗

Author Responses (Part 1)

Thank you for the constructive feedback. We are glad that you found our framework "well-motivated", the operator replacements across six architectures "useful evidence that correctly aligned support-query relationships are important for prediction in the controlled settings", and the experimental design "a valuable perspective for studying the mechanisms of tabular in-context learning".

In short: your review asks whether the conclusions of our models extend to pretrained TFMs (Weakness, Q1, Q3) and to natural tables (Q2). Following your questions, we (i) intervened on the row attention of frozen TabPFN and TabICL and recovered their predictions by first-order propagation along their own attention; (ii) clarified the role of the natural tables in the paper and showed that trained models also benefit from KK on these tables; (iii) compared KRR on KK with standard kernel methods on the raw features, on the 25 tables and on OpenML-CC18; and (iv) compared aligned and shuffled geometry in frozen TFMs on the natural tables of Table 16 (Table 14 in the original submission) and on the control tasks of Table 17 (Table 15 in the original submission).

Regarding first-order propagation in pretrained TFMs (Weakness, Q1). We appreciate this question. Pretrained TFMs rely on correctly attached support-query relations, and first-order propagation of the support labels along their own attention recovers their predictions. These are measurements on the TFMs themselves: TabPFN v2 [1] and TabICL [2] were trained without KK and without our architecture, and we intervened on their own row attention on the 25 natural OpenML tables with the protocol of Table 16 (Section 4.5, Tables 2 and 31):

Frozen TFM, 25 natural tablesTabPFN v2TabICL
Unmodified model, AUC (%)78.882.7
Query-to-support weights attached to randomly permuted support rows, AUC (%)58.471.0
Same weights, support rows permuted within each class, AUC (%)72.179.3
Labels propagated along the attention of the model, averaged over layers, AUC (%)78.382.1
Agreement of propagation along the last-layer attention with the model (% of query rows)96.293.6

Attaching the attention weights of each query to the wrong support rows lowers AUC on all 25 tables for each model, whereas a permutation within each class, which keeps the class of each attended row, retains 72.1% and 79.3%. Following your comment, Section 4.5 now presents these interventions on the attention of the pretrained models.

AuthorsOpenReview ↗

Author Responses (Part 2)

Regarding the natural tables and KRR on K (Q2). Thank you for asking what explains the difference on the natural tables, which helped us clarify their role in the paper. The natural tables test whether the Gram matrix KK carries the predictive information of real tables when it is paired with a predictor that adapts to each table, and KRR on KK reaches 80.0%. The models trained on synthetic data are the instruments of the experiments in Sections 4.2 and 4.3: they are trained on class-balanced binary synthetic tasks, whereas the class frequencies of the natural tables vary. Following your comment, we revised Section 4.4 and Appendix G to state this distinction, and we added results showing that trained models also benefit from KK on the natural tables once their training prior covers these class frequencies (Tables 28, 30, and 32):

25 natural tables, accuracy (%)Features+geometryToken
Trained on class-balanced synthetic tasks (Appendix G)51.939.0
Same models, re-evaluated with the standard correction of the class prior [3, 4] (Table 28)77.572.8
Trained on class-imbalanced synthetic tasks, without correction78.974.3
Trained on the TabICLv2 prior [5], without correction78.969.8

Trained on tasks whose class frequencies vary, the features+geometry model reaches 78.9% without any correction ([74.0, 83.6] and [73.7, 83.8]), the accuracy of gradient-boosted trees on the raw features (78.8%), and improves on the token model by 4.7% and 9.0% ([2.1, 7.7] and [3.9, 15.0]). Its accuracy is within 1.1% of KRR on KK (80.0%), inside the range of the standard methods on these tables, from 77.9% for Euclidean kkNN to 80.3% for RBF kernel ridge regression (Table 22).

Regarding standard kernel methods on the normalized raw features (Q2). Thank you for this suggestion. Following it, we compared KRR on KK with standard methods on the same 25 tables, shift families, and episodes (Table 22), and on OpenML-CC18 [6], with its official 10-fold splits, on the 60 of its 72 tasks with at most 10 classes and 500 encoded features, with the same 256 rows of each training fold as context for every method (Table 35):

25 natural tables, accuracy (%)
Euclidean kkNN, standardized features77.9
Gradient-boosted trees, raw features78.8
Logistic regression, standardized features79.8
Polynomial-kernel ridge regression, standardized features80.0
RBF SVM, standardized features80.1
RBF kernel ridge regression, standardized features80.3
KRR on KK (Section 4.4)80.0
KRR with the exponential function of KK80.8
OpenML-CC18, 60 datasetsAccuracy (%)Average rank (of 8)
Gradient-boosted trees80.815.42
Logistic regression80.775.22
Polynomial-kernel ridge regression81.215.03
RBF SVM81.754.09
RBF kernel ridge regression81.673.83
KRR on KK81.364.52
KRR with the exponential function of KK81.614.04
Geometry model with support-only operators (Ours)81.653.85

On the 25 tables, KRR on KK is within 0.3% of each standard kernel method and exceeds gradient-boosted trees by 1.2% ([0.3, 2.1], on 17 of the 25 tables). On OpenML-CC18, KRR with the exponential function of KK is within 0.15% of the RBF SVM and of RBF kernel ridge regression, KRR on KK is within 0.4% of them, and both exceed logistic regression and gradient-boosted trees. Our geometry model with support-only operators, following the spectral function λ↦1/(λ+ρ)\lambda\mapsto 1/(\lambda+\rho) suggested by Reviewer yWs1 and trained on the TabICLv2 prior [5] and on real OpenML tables that share no dataset with OpenML-CC18, is within 0.1% of the RBF SVM and of RBF kernel ridge regression, and above gradient-boosted trees on 42 of the 60 datasets.

AuthorsOpenReview ↗

Author Responses (Part 3)

Regarding benefits beyond conventional kernel representations (Q2). We appreciate this question on what KK adds beyond conventional kernel representations. In accuracy, KK is level with conventional kernels, and kernel methods different from ours support the same conclusion: a predictor that reads only the relations between the rows of a task, through KK, an RBF kernel, or a polynomial kernel, reaches the accuracy of standard methods (Tables 22 and 35). Inside the trained models, an RBF Gram matrix of the standardized rows, before row normalization, gives a similar accuracy on five of the six backbones (Table 37):

Feature-space OOD accuracy (%)KKRBF Gram matrix
Cell Attention63.3763.56
MotherNet63.3963.80
Perceiver63.4763.72
Prototype63.4163.67
Retrieval63.2863.71
Mean, five backbones63.3863.69

The benefits of KK are three properties. First, it has no bandwidth or degree, and its only setting, the clipping threshold of Equation (2), is fixed for all tables, so that KK is defined in the same way for every table, without a per-table selection step. Second, it is the maximal orthogonal invariant of the canonicalized rows Uˉ\bar U (Proposition 1), so that every predictor that is invariant to orthogonal transformations of these rows can be written as a function of KK and the support labels. As a consequence, every kernel of these rows that depends only on their inner products or distances, such as a polynomial or an RBF kernel, is a function of KK, because ∥uˉi−uˉj∥2=Kii+Kjj−2Kij\|\bar u_i-\bar u_j\|^2=K_{ii}+K_{jj}-2K_{ij}. For example, the diagonal of KK equals 1 up to the stabilizer of Equation (3), and the RBF kernel of these rows is, up to a constant factor, the exponential function of KK in Tables 22 and 35. Third, its factorized form K=UˉUˉ⊤K=\bar U\bar U^\top gives first-order propagation KQSYSK_{QS}Y_S in O(NdC)O(NdC) time, without forming an N×NN\times N matrix (Appendix E).

We embrace your suggestion to compare with standard kernel methods, which shows that other kernels of the rows are useful for this purpose as well. Following your guidance, Section 4.4 now states that other kernels of the rows, such as the RBF kernel, reach the same accuracy as KK in kernel methods and in five of the six trained backbones (Tables 22, 35, and 37), and that our findings therefore regard the relations between rows, of which KK is an instance without a per-table selection step. The conclusion now states that the findings regard the relations between rows.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers (Part 1)

We would like to thank all reviewers for their valuable comments and for their efforts in providing actionable feedback. We uploaded a revised paper in which every addition is marked in blue, with the new results in Section 4.5 and Appendix I.

The reviewers appreciated the approach of the paper, described as "quite unique" (yWs1), our framework for separating the row relations of a task from the predictor, found "well-motivated" (618x), and our perspective of tabular ICL through row geometry and label propagation, found "interesting and potentially useful for the future analysis and design of tabular foundation models" (MvQD). They also valued the analysis as "systematic" (MvQD), the operator replacements across six architectures as "useful evidence that correctly aligned support-query relationships are important for prediction in the controlled settings" (618x), and noted that our models "generalize better to feature counts not seen during pre-training" (yWs1).

The reviews ask whether the mechanism that we identify in our models, first-order propagation of the support labels along correctly attached row relations, also holds in pretrained TFMs and on natural tables. Following the suggestions of the reviewers, we obtained the following results:

  • Pretrained TFMs rely on correctly attached relations (MvQD, 618x). We intervened on the row attention of frozen TabPFN v2 [1] and TabICL [2] on the 25 natural OpenML tables, with the protocol of Table 16 (Table 14 in the original submission). Attaching the attention of each query to randomly permuted support rows lowers AUC from 78.8% to 58.4% and from 82.7% to 71.0%, on all 25 tables for each model, whereas permuting the support rows within each class keeps 72.1% and 79.3% (Section 4.5, Table 2).
  • First-order propagation along their own attention recovers their predictions (MvQD, 618x). Propagating the support labels along the last-layer query-to-support attention of each unmodified model, r=WYSr=WY_S, agrees with TabPFN on 96.2% and with TabICL on 93.6% of the query rows, and along the attention averaged over layers it reaches 78.3% and 82.1% AUC, compared with 78.8% and 82.7% for the models (Table 31).
  • The findings hold across design choices (MvQD, yWs1). Retrained with coefficients shared by all tasks, with g(λ)=λ/λmax⁡g(\lambda)=\lambda/\lambda_{\max} in the bank, with a first-order bank, or with a spectral function learned by an MLP, the six backbones reach 62.75% to 63.31%, compared with 63.13% (Table 24), and aI+bKaI+bK reaches at least 62.65% in each retrained design (Table 26).
  • The backbones read the first-order signal (MvQD). Direct propagation arg⁡max⁡c(KQSYS)c\arg\max_c(K_{QS}Y_S)_c, with no trained weights, reaches 62.8% on the feature-space OOD suites, the mean accuracy of the trained geometry models in Table 13 (Table 11 in the original submission), and the trained models improve on it on the tasks of Table 1 (+4.1% to +4.3%) and the IID suite (+2.4%) (Appendix I, Table 21).
AuthorsOpenReview ↗

General response to all reviewers (Part 2)

On the natural tables and on an established benchmark, the new results are the following:

  • KRR on the Gram matrix reaches the accuracy of standard kernel methods (618x, yWs1). On the 25 natural tables, KRR on KK (80.0%) is within 0.3% of RBF kernel ridge regression, an RBF SVM, and polynomial-kernel ridge regression on the standardized raw features, and exceeds gradient-boosted trees by 1.2% ([0.3, 2.1]) (Table 22). On OpenML-CC18 [3], with its official 10-fold splits, KRR with the exponential function of KK is within 0.15% of RBF kernel ridge regression and of the RBF SVM, and exceeds logistic regression by 0.85% (Table 35). With a conventional RBF Gram matrix in place of KK, five of the six trained backbones reach the same accuracy (Table 37).
  • An established benchmark (yWs1, 618x). We evaluated on the 60 OpenML-CC18 tasks with at most 10 classes and 500 encoded features, with the official 10-fold splits and the same 256 rows of each training fold as context for every method. Our geometry model with support-only operators, following the spectral function λ↦1/(λ+ρ)\lambda\mapsto 1/(\lambda+\rho) suggested by Reviewer yWs1 and trained on the TabICLv2 prior [4] and on real OpenML tables that share no dataset with OpenML-CC18, reaches 81.65%, an average rank of 3.85 of 8 methods. It is within 0.1% of the RBF SVM and of RBF kernel ridge regression, above logistic regression and polynomial-kernel ridge regression, and above gradient-boosted trees on 42 of the 60 datasets (Table 35).
  • Trained models also benefit from the Gram matrix on natural tables (618x, yWs1). Trained on class-imbalanced synthetic tasks or on the TabICLv2 prior, the features+geometry model reaches 78.9% on the 25 natural tables without any correction, the accuracy of gradient-boosted trees, and improves on the token model by 4.7% and 9.0% (Tables 30 and 32). With the standard correction of the class prior [5, 6], the features+geometry model of the paper reaches 77.5% (Table 28).
  • Pretrained TFMs depend on aligned rather than shuffled geometry (618x). When the query-to-support attention of frozen TabPFN v2 [1] and TabICL [2] is computed from KK on the 25 natural tables of Table 16 (Table 14 in the original submission), aligned KK exceeds KK with misaligned (shuffled) rows by 14.9% and 11.2% AUC, on 21 and 25 of the 25 tables (Table 2). As an added bias on the 10 control tasks of Table 17 (Table 15 in the original submission), the aligned bias exceeds the row-shuffled bias on all 10 tasks for TabPFN v2 [1], v2.5 [7], and v3 [8] (Table 27).

Added material (in blue). Section 3: the motivation of the spectral bank, the dependence on other queries, and the relation to the benchmark of Grinsztajn et al. on axis-aligned data (MvQD, yWs1). Section 4: the notation of the input conditions, the main findings, the evaluation protocol and the IID tasks, the role of the backbones, the role of the natural tables and of the Gram matrix, the interventions on frozen TFMs (Table 2), and the definition of the added bias (MvQD, 618x, yWs1). Proposition 2 is now stated as a lemma (yWs1), and the conclusion states that the findings regard the relations between rows (618x). Appendices B, E, and G: the definition of every input condition (Table 4), the relation of the backbones to existing models, the six coefficient inputs, spectral functions through the SVD of Uˉ\bar U, and the role of the natural tables (MvQD, 618x, yWs1). Appendix I: Tables 21 to 38 (MvQD, 618x, yWs1). Each reply to a reviewer shows its tables directly.

We thank all reviewers again for their constructive feedback, which helped us to improve the paper and to sharpen the presentation of its contributions. We hope that these additions address your concerns, and that you will consider revising your scores. We are happy to discuss any further questions.

References:

[1] Accurate Predictions on Small Data with a Tabular Foundation Model
[2] TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
[3] OpenML Benchmarking Suites
[4] TabICLv2: A Better, Faster, Scalable, and Open Tabular Foundation Model
[5] Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure
[6] Long-tail learning via logit adjustment
[7] TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models
[8] TabPFN-3: Technical Report

Area_Chair_gY9COpenReview ↗

Please Join The Discussion

Dear Reviewers,

We're halfway through the discussion stage. The authors have provided their responses to your reviews. If their responses have not addressed your concerns, please raise and leave some time for the authors to respond again. If yes, please still leave a comment to the authors. Thanks in advance for keeping the conversation going.

Many thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.