acceptodds
Under review as a conference paper at ICLR 2027

Architecture Alignment with Sparse Priors in Tabular Foundation Models

Abstract

Tabular foundation models (TFMs) are increasingly popular because they deliver strong predictions on new datasets through in-context learning, without task-specific training or extensive tuning. Yet released TFMs differ simultaneously in their pretraining priors, architectures, and objectives, obscuring their respective inductive biases. We therefore examine one concrete capability: irrelevant-feature suppression. Across synthetic tasks and real-world datasets, adding null features causes substantially greater predictive degradation in the row-token model TabDPT, whereas the cell-token alternating-axis model TabPFN v2 and other TFMs remain comparatively stable. This gap motivates us to ask whether architecture contributes to irrelevant-feature suppression. Because released TFMs remain confounded by other design choices, we train streamlined row-token and alternating-axis transformers under identical sparse-to-dense linear priors. Exact Bayes analysis shows that sparse prediction requires context-dependent feature gating, whereas the dense endpoint requires only uniform feature weighting. Consistent with this distinction, the alternating-axis model is substantially closer to the Bayesian optimal predictor on sparse tasks, while the architecture gap becomes negligible on dense tasks; almost all of the sparse gap arises from linear coefficient-estimation error. Finally, in both the controlled model and frozen TabPFN v2, we examine the effect of interventions on the feature-attention outputs on the linear coefficients, finding evidence of task-dependent selective routing of computation through feature-indexed pathways. Together, these results support architecture–prior alignment: preserving an addressable feature axis provides an inductive bias for task-adaptive relevance inference. Code available at anonymous.4open.science/r/ARCHITECTURE-PRIOR-TFM.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Architecture, Tabular foundation model, TabPFN, prior, sparsity, feature selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VBQYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether architecture, independent of the pretraining prior, contributes to irrelevant-feature suppression in tabular foundation models (TFMs). It proceeds in three stages.

Observational stress test (Sec. 2, App. C). Null features are added to Friedman tasks and to 42 OpenML regression datasets, using nested doses ρ ∈ [0, 7] and five construction repeats. The row-token TabDPT degrades far more than TabPFN v2/v2.5/v3 and TabICL v2: the median R² drop at ρ = 7 is 0.037/0.071 for TabDPT versus 0.0008/0.0014 for TabPFN v2. TabDPT also shows larger null-feature permutation shifts. The gap shrinks across TabDPT v1.0 → v1.3 but persists. Controlled comparison (Sec. 3, App. D–E). The authors construct a matched family of orthogonal-design sparse linear priors P_k. Changing k alters the support structure but not the coefficient covariance or signal energy, and every member admits a closed-form posterior mean with a context-dependent inclusion gate q_k(c). Theorem 1: response-affine predictors attain the dense Bayes rule but incur a strictly positive approximation gap for every k < d. Proposition 3: an idealized alternating-axis network can approximate the sparse Bayes predictor for any k. Experiments: streamlined row-token and nanoTabPFN-style alternating-axis models (≈0.36–0.39M parameters) are trained under identical priors and evaluated against the exact Bayes target. The alternating-axis model has substantially lower normalized excess error under sparsity; at k = 1 the gap is ≈0.34 for d = 10 and ≈0.63 for d = 20. The gap nearly vanishes at k = d. An exact decomposition (Lemma 2) attributes ~98% of the sparse gap to error in the linear coefficients. Interventions (Sec. 4, App. F). Feature-attention messages are attenuated in the controlled model and in frozen TabPFN v2, and the change in the fitted linear coefficients is measured. Active sources show higher own-coefficient specificity and sensitivity than inactive ones, concentrated in final-layer query messages. The authors interpret this as task-dependent selective routing through feature-indexed pathways.

The overall claim is “architecture–prior alignment”: an addressable feature axis is an inductive bias that pays off when relevance varies across tasks, not a universally better design.

Critical strengths
  1. A controlled design scored against an exact Bayes target.

Released TFMs confound architecture with prior, preprocessing and scale. The matched family P_k varies only support structure at fixed second moments and has closed-form posteriors. Both architectures are therefore trained identically and scored by distance to the Bayes optimum, with k = d as a negative control.

Why it is consequential: It makes the question falsifiable with a reusable protocol, and it is my main reason to support acceptance.

  1. Theory and experiments agree on mechanism, not only outcome.

Theorem 1 shows that response-affine predictors can do no better than the dense Bayes rule, which is strictly suboptimal for k < d. In the experiments, the alternating-axis model wins on ≥98% of sparse tasks, and the gap nearly vanishes at k = d. Lemma 2 attributes >97% of the sparse gap to coefficient error, which is where the inclusion gate acts.

Why it is consequential: It lifts “architecture–prior alignment” above a post-hoc story.

Critical weaknesses
  1. The causal claim is not separated from optimization and compute.

Theorem 1 covers only response-affine predictors. The trained row-token model, with five softmax-attention blocks, is not one and could in principle learn feature gating. The gap may therefore reflect under-training rather than a representational barrier:

At d = 10, k = 1, all row-token seeds stop at the 10,000-step cap (Table 7). Compute is not matched between the two architectures. Ablations isolating feature-axis attention are missing.

Why it is consequential: This is the main reason my score is not higher. If longer training, matched compute or feature gating closes the gap, the finding becomes “slower learning at this budget”, not an architectural advantage. Learning curves, a 3–5× training budget and these ablations would make this a clear accept.

  1. The link back to released models is suggestive, not demonstrated.

The controlled setting is far from released TFMs: it uses orthogonal linear priors, d ≤ 20, and one model per known k, whereas released models face tasks in which sparsity is unknown. Moreover, TabDPT’s Friedman null-feature sensitivity falls 16-fold across releases while keeping the same row-token design (App. C.4). Training choices alone therefore move this capability substantially.

Why it is consequential: The result holds for a stylized prior but does not yet explain the motivating phenomenon. Training on a mixture over k, and adding one correlated or nonlinear prior, would test whether the result carries over. This limits the paper’s significance but not Strengths 1–2, so I still lean towards acceptance.

Questions for authors
  1. Do the sparse gaps persist when both models are trained to convergence, e.g., with 3–5× the step budget? Can you show validation curves of the normalized Bayes error over training? Does a FLOP-matched or deeper/wider row-token model close the gap?
  2. What happens when both architectures are trained on a mixture over k (unknown sparsity) and evaluated per k? Does the alternating-axis advantage grow, shrink or stay the same?
  3. Can you run the ablations isolating (i) per-feature tokenization, (ii) learned versus uniform feature-axis attention, and (iii) a feature-gated row-token model?
  4. How do TabDPT’s results change with retrieval enabled, as in its default configuration? What changed between TabDPT v1.0 and v1.3 that so strongly reduced null-feature sensitivity, and how does that bear on the architectural interpretation?
  5. For Section 4, can you add norm-matched random-direction perturbations and a Bayes-map reference Jacobian, and replicate across training seeds? How should the frequent negative own-feature responses (56/128 tasks at d = 10) be interpreted under the routing hypothesis?
  6. Can you report the controlled gap as unnormalized excess risk, and relative to the irreducible noise?
  7. Do the conclusions extend to correlated designs or to a sparse nonlinear prior where the exact Bayes rule is still computable (e.g., by Monte Carlo)?
6 comments

We thank the reviewer for the careful reading and constructive suggestions. We appreciate your recognition that our matched-prior comparison against an exact Bayes target provides a falsifiable, reusable protocol for studying architecture–prior alignment. We also appreciate your assessment that the agreement between theory, sparse-to-dense experiments, and coefficient-error decomposition provides evidence about the underlying computation. We respond to your questions and concerns point by point below.

1. Training budget, matched compute, and model capacity (W1, Q1)

Following your suggestion, we added longer training runs, compute-matched comparisons, and width and depth sweeps for the Row-token model.

Matched cumulative training FLOPs. We compared the two architectures at matched cumulative training FLOPs. Because Alternating-axis costs more per update, we allowed Row-token more updates at each common budget. Row-token improves with compute, but Alternating-axis retains lower normalized Bayes error under sparse priors in both dimensions at the largest shared budgets. Dense-prior errors are close. See the figure in this link.

Extended training (10× steps). We increased the training limit from 10,000 to 100,000 steps for d=10d=10 and from 20,000 to 200,000 steps for d=20d=20. We disabled early stopping and recorded validation Bayes-error trajectories throughout these runs. The sparse-prior gap remains clear at the extended endpoints, while the dense curves nearly overlap. See Fig. 28 in Appendix E.3.5 (or in this link).

Wider Row-token models. We trained wider Row-token models and compared them with the original three-block Alternating-axis baseline, shown by the horizontal reference lines; the x-axis gives Row-token embedding width. Increasing width does not close the sparse gap in either dimension over the tested range, while dense results remain close. See the figure in this link.

Deeper Row-token models. We trained Row-token models with 3 to 10 blocks. More depth helps in some settings, but the original three-block Alternating-axis baseline retains substantially lower sparse-prior error at the evaluated training-step caps. Dense-prior errors stay close. See Fig. 30 in Appendix E.3.7 (or in this link).

The sparse-prior gap persists after extending training, matching compute, and varying Row-token capacity within the tested ranges. Together with the close dense-prior results, this supports architecture–prior alignment under the evaluated training protocols. Some trajectories continue improving, so the empirical comparison is limited to the tested budgets. We use the response-affine theorem as a motivating example of the computational distinction between dense and sparse priors. It does not cover unrestricted deep Row-token models; our evidence for an architectural advantage comes from the controlled empirical comparisons.

2. Adaptation to unknown sparsity (W2, Q2)

Following your suggestion, we trained both architectures on mixtures over support sizes without providing the generating KK to either model.

We used two mixtures: KK is uniform over 1,d/2,d\\{1,d/2,d\\} or over 1,…,d\\{1,\ldots,d\\}. Both preserve the coefficient covariance and expected signal energy of the matched prior family. We evaluated each mixture against its exact Bayes rule, which infers KK from the context by posterior weighting. The generating KK was used only to stratify evaluation. We scaled the maximum training-step budget by the number of candidate support sizes to match expected exposure per support size at the fixed-kk cap, using the same mixture-training protocol for both architectures.

We compared fixed-kk and mixed-KK test errors in the figure below. Each training prior has its own Bayes predictor, so we compare the architectures within each prior. See Fig. 25 in Appendix E.3.4 (or in this link).

Alternating-axis retains an advantage under both mixtures. At K=1K=1, the normalized gap is smaller than under fixed-kk training; at K=d/2K=d/2, the gaps are similar. At K=dK=d, the near tie under fixed-kk training becomes an Alternating-axis advantage under mixture training. For mixed-KK tasks generated with K=dK=d, the Bayes target still marginalizes over unknown support size and differs from the known-dense reference.

We also extended the three-point mixture training to 10× its original mixture-training step cap, with early stopping disabled. At d=10d=10, Row-token continues improving, especially for K=1K=1, but Alternating-axis remains better late in training. At d=20d=20, Row-token flattens earlier and Alternating-axis retains lower error across support sizes. See Fig. 26 in Appendix E.3.4 (or in this link).

The Alternating-axis advantage persists when support size varies across tasks and must be inferred from context, including under extended training.

3. Learned attention, tokenization, and feature gating (W1, Q3)

(i) Per-feature tokenization and cross-feature communication. Our Alternating-axis model already assigns a separate token to each feature cell and an additional response token to each row. In this implementation, sample-axis attention operates independently within each column, the MLP acts on each token separately, and the decoder reads the query response token. Removing feature-axis attention from every block would therefore disconnect the covariates from the decoder. We used uniform mixing to retain this information path while examining the role of learned feature-attention weights.

(ii) Learned versus uniform feature-axis attention. Following your suggestion, we added two controls that set feature-axis attention weights to uniform in every layer while retaining feature tokens and the remaining computational pathways. We first imposed uniform attention on frozen checkpoints, with no parameter updates, to test their reliance on the learned attention configuration. We then trained new models from random initialization with uniform feature attention throughout, allowing the remaining parameters to adapt. The learned Alternating-axis baseline is labelled “Learned attention” below.

Normalized Bayes approximation error (mean ± SD across training seeds; lower is better):

(d, k)Learned attentionFrozen to UniformUniform from scratch
(10, 1)0.0561 ± 0.00110.9677 ± 0.03790.0667 ± 0.0045
(10, 5)0.0214 ± 0.00060.8461 ± 0.14390.3181 ± 0.5102
(10, 10)0.0183 ± 0.00040.7775 ± 0.18440.3135 ± 0.5113
(20, 1)0.0779 ± 0.00601.0785 ± 0.09860.0858 ± 0.0137
(20, 10)0.0248 ± 0.00130.9404 ± 0.06760.0272 ± 0.0007
(20, 20)0.0221 ± 0.00020.9489 ± 0.08220.0240 ± 0.0020

Imposing uniform attention on frozen checkpoints sharply increases error. Training with uniform attention from scratch recovers much of the performance, particularly at k=1k=1 in both dimensions, although some d=10d=10 cases show large variation across seeds. Thus, the trained checkpoints rely on their learned attention configuration, while low error can also be obtained by training with a fixed mixing rule.

To examine how the model adapts to fixed mixing weights, we look at the learned value representations. For intuition, we compare value magnitudes before and after attention weighting, using the simplified expressions ∥vj∥2\lVert v_j\rVert_2 and ∥αpjvj∥2\lVert \alpha_{pj} v_j\rVert_2. Here, αpj\alpha_{pj} is the attention weight from query target token pp to feature token jj. The precise multi-head definitions used in our experiments are given in Appendix E.3.9. The figure reports the layerwise contrast as log⁡2R\log_2 R for both k=1k=1 settings, where RR is the active-to-inactive ratio of mean norms on query rows from 3,200 held-out tasks per setting. See Fig. 31 in Appendix E.3.9 (or in this link). We observe the following phenomenon:

  • Adaptation under uniform attention. Before attention weighting, the uniform-trained model has a larger active-to-inactive value-norm ratio than the model with learned attention. This suggests that the model compensates for equal attention weights by learning larger values for active features relative to inactive ones.
  • Joint effect of learned attention and values. Attention weights and value representations jointly determine the feature messages. Active and inactive features are more clearly separated in the norms of these weighted messages than in the value norms alone.

These controls refine the mechanism interpretation of the original Row-token versus Alternating-axis comparison. The observed architecture gap under matched priors remains the empirical basis for our architectural claim. Low error with uniform attention is consistent with a benefit from preserving feature-indexed computation, while showing that learning feature-attention weights is not the only route to low error in this setting. The controls retain both feature tokens and cross-feature mixing, so they do not separately identify the contribution of tokenization.

(iii) Feature-gated Row-token models. We agree that adding a context-dependent feature gate to Row-token would be a useful alternative architecture. An input gate would test whether one-time feature selection is sufficient to close the gap, whereas Alternating-axis can update feature representations across layers. A systematic comparison requires specifying how context evidence is aggregated, where gates are applied, and how model capacity is allocated. We leave the design and evaluation of this alternative architecture to future work.

Reviewer_MywuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks whether the way a tabular foundation model (TFM) tokenizes (row-based or cell-based tokenization) affects its ability to ignore irrelevant features in a tabular dataset. It adds null columns to Friedman and OpenML regression tasks. The row-token model TabDPT loses far more accuracy than the cell-token TabPFN v2, while TabICL v2 and TabPFN v3 hold up their accuracy. The authors opine that these models differ in too many ways for this to pin anything on architecture. To isolate the design, they build a family of sparse linear regression tasks where only k of d features matter. They prove that any predictor that weights the labels in a fixed way is optimal when every feature matters but never when only some do, and show a cell-token network has a natural route to the sparse rule. Training small versions of both designs on identical tasks, the cell-token model is much closer to optimal when few features matter, the gap nearly vanishes when all do, and almost all of it comes from the per-feature weights.

Critical strengths
  • The prior family is built so that changing how many features matter leaves the overall signal strength and feature covariance untouched, and the setup is simple to compute the exact Bayes-optimal prediction. The paper can say how far each model is from perfect, not just which one wins.

  • Because the ideal prediction is linear in the query, the error splits cleanly into wrong slopes, a wrong offset and leftover curvature, and 97.5–99.7% of the sparse gap turns out to be wrong slopes. The OLS baseline does far worse than either model, leading to the fact both networks learn to down-weight features and the cell-token one does it much better.

Critical weaknesses
  • The two trained models differ in tokenization, feature attention, depth (5 vs 3 blocks) and compute all at once, so the comparison can’t say which of these produces the gap. The paper’s thesis points to one of them, the feature axis, and the test that would isolate it. Removing only feature attention from the cell-token model is missing.

  • As per my understanding, the theorem separates fixed from adaptive use of the labels. It does not separate models with a feature axis from models without one. Going from sparse to dense removes both the per-feature choice and the need to adapt at all. So the shrinking gap could mean row-token models are just worse at any adaptive computation. A dense prior with a random signal scale per task needs adaptation but no per-feature choice, and would tell the two apart.

  • Minor point, there is a typo in section 4.2 that says “a collection same 128 datasets”.

Questions for authors
  1. What happens to the k=1 gap in Figure 4 if you remove feature-axis attention from the alternating-axis model but keep one token per cell? The paper credits the feature axis, but the two models also differ in tokenization, depth and compute, and this is the test that would isolate the part that matters.

  2. Could you train both controlled models on a dense prior where the signal scale v² is drawn fresh for each task, so the Bayes rule must adapt to the context but treats every feature the same, and report the architecture gap as in Figure 4? Going from sparse to dense removes both the per-feature choice and the need to adapt at all, so the current comparison can't show which of the two the row-token model is missing.

  3. How does the k=1 gap in Figure 4 change if the row-token model is given more than five blocks, or the alternating-axis model is matched to it in depth? Depth is one of the things that differs between the two models, and if the gap shrinks with depth, it points to depth rather than the feature axis as the cause.

4 comments

We thank the reviewer for recognizing the value of our exact Bayes benchmark and coefficient-error analysis. We have conducted additional experiments to address your suggestions and respond to each point below.

1. Isolating feature-axis attention (W1, Q1)

Following your suggestion, we added two uniform-attention controls that retain one token per cell and preserve communication between the covariates and the target token.

Simply deleting feature-axis attention would disconnect the covariates from the decoder: sample-axis attention operates independently within each column, the MLP acts independently on each token, and the decoder reads only the query target token. We therefore replaced feature-axis attention weights with uniform weights in every layer. This preserves cross-feature mixing while removing input-dependent attention weights along that axis.

Frozen-checkpoint intervention. We imposed uniform attention on the trained Alternating-axis checkpoints to test their reliance on the learned attention configuration. Training from scratch. We also retrained models with uniform feature-axis attention to test whether the network can learn under a fixed mixing rule. The learned Alternating-axis baseline is labelled “Learned Attention” below.

Normalized Bayes approximation error (mean ± SD across training seeds; lower is better):

(d, k)Learned AttentionFrozen to UniformUniform from scratch
(10, 1)0.0561 ± 0.00110.9677 ± 0.03790.0667 ± 0.0045
(10, 5)0.0214 ± 0.00060.8461 ± 0.14390.3181 ± 0.5102
(10, 10)0.0183 ± 0.00040.7775 ± 0.18440.3135 ± 0.5113
(20, 1)0.0779 ± 0.00601.0785 ± 0.09860.0858 ± 0.0137
(20, 10)0.0248 ± 0.00130.9404 ± 0.06760.0272 ± 0.0007
(20, 20)0.0221 ± 0.00020.9489 ± 0.08220.0240 ± 0.0020

Imposing uniform attention on frozen checkpoints sharply increases error. Training with uniform attention from scratch recovers much of the performance, particularly at k=1k=1 in both dimensions, although some d=10d=10 cases show large variation across seeds. Thus, the trained checkpoints rely on their learned attention configuration, while low error can also be obtained by training with a fixed mixing rule.

To examine how the model adapts to uniform mixing weights, we look at the learned value representations. For intuition, we compare value magnitudes before and after attention weighting, using the simplified expressions ∥vj∥2\lVert v_j\rVert_2 and ∥αpjvj∥2\lVert \alpha_{pj} v_j\rVert_2. Here, αpj\alpha_{pj} is the attention weight from query target token pp to feature token jj. The precise definitions used in our experiments are given in Appendix E.3.9. The figure reports the layerwise contrast as log⁡2R\log_2 R for both k=1k=1 settings, where RR is the active-to-inactive ratio of mean norms on query rows from 3,200 held-out tasks per setting. See Fig. 31 in Appendix E.3.9 (or in this link). We observe the following phenomenon:

  • Adaptation under uniform attention. Before attention weighting, the uniform-trained model has a larger active-to-inactive value-norm ratio than the model with learned attention. This suggests that the model compensates for equal attention weights by learning larger values for active features relative to inactive ones.
  • Joint effect of learned attention and values. Attention weights and value representations jointly determine the feature messages. Active and inactive features are more clearly separated in the norms of these weighted messages than in the value norms alone.

These controls refine the mechanism interpretation of the original Row-token versus Alternating-axis comparison. The observed architecture gap under matched priors remains the empirical basis for our architectural claim. Low error with uniform attention is consistent with a benefit from preserving feature-indexed computation, while showing that learning feature-attention weights is not the only route to low error in this setting. The controls retain both feature tokens and cross-feature mixing, so they do not separately identify the contribution of tokenization.

2. Dense prior with a task-dependent signal scale (W2, Q2)

Following your suggestion, we trained both architectures on a random-scale dense prior. All features remain active, while a shared signal variance is drawn log-uniformly across tasks over a range with max-to-min ratio rr. The Bayes rule must adapt to the observed responses through a shrinkage factor shared by all features, so this control requires response-dependent adaptation without feature-specific selection.

The table reports d=10d=10, where r=1r=1 is the fixed-scale dense prior. We set r=75r=75 to approximately match the mean unnormalized Bayes gain over the best response-affine predictor of the sparse k=d/2k=d/2 prior, so both priors reward adaptation by a similar amount; r=10r=10 and r=1000r=1000 bracket this value. Appendix E.3.3 shows the same pattern at d=20d=20. Normalized Bayes approximation error (mean ± SD across three training seeds; 32,000 test tasks; lower is better):

rRow-tokenAlternating-axis
10.0191 ± 0.00030.0183 ± 0.0004
100.0247 ± 0.00090.0235 ± 0.0003
750.0411 ± 0.00050.0405 ± 0.0003
10000.0786 ± 0.00270.0787 ± 0.0058

The two architectures have comparable errors across the tested scale ratios, including the strongest scale variation at r=1000r=1000. Thus, Row-token can learn task-level response adaptation in this setting. Its much larger gap under sparse priors is consistent with an advantage for Alternating-axis when adaptation requires feature-specific weighting.

We agree with your interpretation of the theorem: it concerns response-affine predictors, whose label weights are fixed given the covariates. The architectural evidence comes from the controlled experiments. The random-scale dense control helps distinguish a difficulty with feature-specific adaptation from a general inability to adapt to responses.

3. Matched depth and training compute (W1, Q3)

Following your suggestion, we trained Row-token models with 3, 6, and 10 blocks and compared them with the same three-block Alternating-axis baseline. We kept the original training protocol and embedding width of 96. The original five-block Row-token and three-block Alternating-axis models were chosen to keep total parameter counts comparable, at approximately 0.39M and 0.36M, respectively.

Depth comparison. The figure reports normalized Bayes approximation error (mean ± SD) for all six (d,k)(d,k) conditions. At matched depth of three blocks, Alternating-axis already has substantially lower k=1k=1 error in both dimensions. Increasing Row-token to ten blocks reduces its k=1k=1 error only modestly, leaving a substantial gap. Dense-prior errors remain close. See Fig. 30 in Appendix E.3.7 (or in this link).

Matched cumulative training FLOPs. To address the compute difference separately, we compared the architectures at common training FLOP budgets, giving Row-token more updates because it costs less per update. We used 32,000 test tasks per condition and three independent training seeds. See the figure in this link.

Row-token improves as compute increases, but Alternating-axis retains lower sparse-prior error at the largest shared budgets in both dimensions. Dense-prior errors are close, with Row-token slightly better in some settings. These controls show that the sparse gap persists under matched depth and matched compute within the evaluated protocols and budgets.

4. Typographical correction (W3)

Thank you for catching this. We replace “a collection same 128 datasets” with “the same collection of 128 datasets” in Section 4.2.

Thank you for showing the additional ablation results. The additional results solidifies the analysis presented in the paper. They can be appropriately included in the revised version of the paper. I maintain my positive score for this paper.

Dear Reviewer Mywu,

Thank you for taking the time to review our paper and follow up on our rebuttal. Your feedback helped us strengthen the experimental analysis.

We appreciate your suggestion and have added results on uniform attention (Appendix E.3.9), the random-scale dense control (Appendix E.3.3), Row-token depth (Appendix E.3.7), and matched FLOPs (Section 3.3 and Appendix E.3.5) to the revised manuscript.

Best,

The Authors

Reviewer_TcanOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the architecture of a tabular foundation model affects its ability to ignore irrelevant features. It starts with the observation that TabDBT, using a TabPFN v1 style row-attention architecture, is confused much more by irrelevant features than TabPFN v2, TabICL v2 or TabPFN v3. Then, a controlled experiment trains very small (nanoTabPFN-style) architectures, with the same priors, ranging from dense to sparse linear regression. In the sparse case, alternating attention again does much better, whereas in the dense case there is no difference.

A confounder for the first experiment: the training data is from real OpenML tables and thus likely doesn't have a lot of uninformative features. In contrast, TabPFN v2 actively generated noise features in the prior. TabPFN v1 didn't do that and was also confused by uninformative features. The TabDBT vs TabPFN v2 experiment only repeats that known finding.

A confounding factor for the two training experiments is that they are not compute matched, and that the embedding size is not matched per row. Thus, the alternating attention gets much more descriptive power and compute than the row-token architecture. Thus, a crucial control experiment would be to run the two architectures in (1) a compute-matched fashion and (2) an embedding-size-matched fashion. I would expect that a TabPFN v1 style architecture can learn the sparse prior given a large enough embedding and enough compute. If the paper showed that this only happens at a much larger scale that would give me more confidence in its result.

Critical strengths
  • The paper studies a finding about TFM architectures, that, if it checks out, would be quite relevant for architecture design.
Critical weaknesses
  • There are alternative explanations for the finding that should be ruled out before acceptance.
Questions for authors

Can you please run the compute-matched and embedding-sized experiments, as well as ablations with hybrid TabICL-style architectures on a spectrum of compute and embedding size. Depending on the outcomes, I am happy to increase my score.

3 comments

We thank the reviewer for recognizing the relevance of our work and for the constructive suggestions. We have conducted additional experiments to address your concerns and respond to each point below.

1. Pretraining priors and the interpretation of Section 2

We agree that the released-model comparison is confounded by differences in pretraining distributions. TabDPT is trained on real OpenML tables, whereas TabPFN v2 explicitly includes uninformative features in its synthetic prior. Other aspects of training and inference may also differ. Therefore, the results in Section 2 should be interpreted as a motivating observation about null-feature sensitivity, not as evidence that architecture alone causes the performance gap. We will make this distinction explicit in the revision.

Section 3 compares Row-token and Alternating-axis models trained on the same prior and evaluated against the same exact Bayes predictor. To address the remaining concerns about training compute and per-row embedding size, we added the controls described below.

2. Matched training compute and extended training

Following your suggestion, we compared Row-token, Alternating-axis, and a TabICL-style hybrid at matched cumulative training FLOPs. Since the architectures have different costs per update, Row-token receives more training updates at a common FLOP budget. We evaluate all three architectures on the same six (d,k)(d,k) conditions, using 32,000 test tasks per condition and three independent training seeds. The bold curves show seed means and the thin curves show individual seeds. See Fig. 27 in Appendix E.3.5 (or in this link). Row-token improves as compute increases, but Alternating-axis retains substantially lower normalized Bayes approximation error on the sparse priors at the largest shared FLOP budgets for both d=10d=10 and d=20d=20. On the dense priors, the two architectures have comparable errors, with Row-token slightly better in some settings. The separation is therefore most pronounced when task-specific feature selection is required, rather than being a uniform advantage across all priors.

We also tested whether the gap disappears with 10× longer training. We increased the training-step limits from 10,000 to 100,000 for d=10d=10 and from 20,000 to 200,000 for d=20d=20, disabled early stopping, and examined the validation trajectories. The sparse-prior gap remains clear at the extended endpoints, while the dense-prior curves nearly overlap. See Fig. 28 in Appendix E.3.5 (or in this link).

These results show that the sparse-prior advantage is not explained by the original step caps or by the unequal cost per update within the budgets tested.

3. Per-row embedding-size-matched comparison

We agree that equal total parameter counts do not give the two tokenizations equal representational width per row. With cell-token width EE, Alternating-axis represents each row using dd feature tokens and one response token. We therefore set the Row-token embedding width to (d+1)E(d+1)E for a per-row embedding-size match.

To make this comparison computationally feasible, we evaluated E=8,16,32E=8,16,32. Both architectures use three blocks and four attention heads, with the original training-step budgets and early stopping disabled. We evaluated three training seeds for all six (d,k)(d,k) conditions. See Fig. 29 in Appendix E.3.6 (or in this link). The table shows the resulting widths and parameter counts.

Cell width EEAlternating-axis parametersRow width (d=10d=10)Row-token parameters (d=10d=10)Row width (d=20d=20)Row-token parameters (d=20d=20)
82,90588205,745168743,905
1610,673176814,1773362,955,457
3240,8013523,239,10567211,781,505

The sparse-prior performance gap remains across the tested embedding widths, even though the matched Row-token models have approximately 70–290× more parameters than Alternating-axis. Increasing width does not consistently improve performance. The dense-prior results remain much closer.

The per-row embedding match is intentionally different from the FLOP match: increasing the Row-token width changes its parameter count and compute, so we report this control separately from the FLOP match.

4. TabICL-style hybrid architecture

Following your request, we also included a TabICL-style hybrid in the matched-FLOP study and across the cell-width settings above. The hybrid combines feature-indexed processing with Row-token in-context learning. At the largest shared FLOP budgets and at every evaluated cell width, the hybrid has lower normalized Bayes error than Row-token for k=1k=1 and k=d/2k=d/2 in both dimensions, suggesting that the sparse-prior advantage is not restricted to an exclusively Alternating-axis design.

For the embedding-size study, the hybrid uses the same cell width EE as Alternating-axis and serves as a per-feature-width reference; unlike the Row-token control, it is not matched in total per-row embedding width or parameter count. Because the hybrid changes several architectural components together, these results do not isolate which component is responsible for its performance. We view a more detailed hybrid-architecture study as useful future work.

Taken together, the new experiments support the conclusion that alternating-axis provides a more favorable inductive bias for learning the tested sparse priors under the evaluated compute and capacity regimes. If Row-token models can close this gap, doing so requires more compute or capacity than tested here, including 10× longer training or up to 290× more parameters. We will clarify this scope, as well as the non-causal role of the released-model comparison, in the revised manuscript.

Dear Reviewer Tcan,

Thank you again for taking the time to review our paper. We have uploaded a revised manuscript and posted our response, which includes the compute-matched and embedding-matched comparisons you suggested.

Before the discussion closes, we would appreciate hearing whether our response addresses your concerns. We would be happy to clarify any remaining points.

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.