- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper asks whether architecture, independent of the pretraining prior, contributes to irrelevant-feature suppression in tabular foundation models (TFMs). It proceeds in three stages.
Observational stress test (Sec. 2, App. C). Null features are added to Friedman tasks and to 42 OpenML regression datasets, using nested doses ρ ∈ [0, 7] and five construction repeats. The row-token TabDPT degrades far more than TabPFN v2/v2.5/v3 and TabICL v2: the median R² drop at ρ = 7 is 0.037/0.071 for TabDPT versus 0.0008/0.0014 for TabPFN v2. TabDPT also shows larger null-feature permutation shifts. The gap shrinks across TabDPT v1.0 → v1.3 but persists. Controlled comparison (Sec. 3, App. D–E). The authors construct a matched family of orthogonal-design sparse linear priors P_k. Changing k alters the support structure but not the coefficient covariance or signal energy, and every member admits a closed-form posterior mean with a context-dependent inclusion gate q_k(c). Theorem 1: response-affine predictors attain the dense Bayes rule but incur a strictly positive approximation gap for every k < d. Proposition 3: an idealized alternating-axis network can approximate the sparse Bayes predictor for any k. Experiments: streamlined row-token and nanoTabPFN-style alternating-axis models (≈0.36–0.39M parameters) are trained under identical priors and evaluated against the exact Bayes target. The alternating-axis model has substantially lower normalized excess error under sparsity; at k = 1 the gap is ≈0.34 for d = 10 and ≈0.63 for d = 20. The gap nearly vanishes at k = d. An exact decomposition (Lemma 2) attributes ~98% of the sparse gap to error in the linear coefficients. Interventions (Sec. 4, App. F). Feature-attention messages are attenuated in the controlled model and in frozen TabPFN v2, and the change in the fitted linear coefficients is measured. Active sources show higher own-coefficient specificity and sensitivity than inactive ones, concentrated in final-layer query messages. The authors interpret this as task-dependent selective routing through feature-indexed pathways.
The overall claim is “architecture–prior alignment”: an addressable feature axis is an inductive bias that pays off when relevance varies across tasks, not a universally better design.
Critical strengths
- A controlled design scored against an exact Bayes target.
Released TFMs confound architecture with prior, preprocessing and scale. The matched family P_k varies only support structure at fixed second moments and has closed-form posteriors. Both architectures are therefore trained identically and scored by distance to the Bayes optimum, with k = d as a negative control.
Why it is consequential: It makes the question falsifiable with a reusable protocol, and it is my main reason to support acceptance.
- Theory and experiments agree on mechanism, not only outcome.
Theorem 1 shows that response-affine predictors can do no better than the dense Bayes rule, which is strictly suboptimal for k < d. In the experiments, the alternating-axis model wins on ≥98% of sparse tasks, and the gap nearly vanishes at k = d. Lemma 2 attributes >97% of the sparse gap to coefficient error, which is where the inclusion gate acts.
Why it is consequential: It lifts “architecture–prior alignment” above a post-hoc story.
Critical weaknesses
- The causal claim is not separated from optimization and compute.
Theorem 1 covers only response-affine predictors. The trained row-token model, with five softmax-attention blocks, is not one and could in principle learn feature gating. The gap may therefore reflect under-training rather than a representational barrier:
At d = 10, k = 1, all row-token seeds stop at the 10,000-step cap (Table 7). Compute is not matched between the two architectures. Ablations isolating feature-axis attention are missing.
Why it is consequential: This is the main reason my score is not higher. If longer training, matched compute or feature gating closes the gap, the finding becomes “slower learning at this budget”, not an architectural advantage. Learning curves, a 3–5× training budget and these ablations would make this a clear accept.
- The link back to released models is suggestive, not demonstrated.
The controlled setting is far from released TFMs: it uses orthogonal linear priors, d ≤ 20, and one model per known k, whereas released models face tasks in which sparsity is unknown. Moreover, TabDPT’s Friedman null-feature sensitivity falls 16-fold across releases while keeping the same row-token design (App. C.4). Training choices alone therefore move this capability substantially.
Why it is consequential: The result holds for a stylized prior but does not yet explain the motivating phenomenon. Training on a mixture over k, and adding one correlated or nonlinear prior, would test whether the result carries over. This limits the paper’s significance but not Strengths 1–2, so I still lean towards acceptance.
Questions for authors
- Do the sparse gaps persist when both models are trained to convergence, e.g., with 3–5× the step budget? Can you show validation curves of the normalized Bayes error over training? Does a FLOP-matched or deeper/wider row-token model close the gap?
- What happens when both architectures are trained on a mixture over k (unknown sparsity) and evaluated per k? Does the alternating-axis advantage grow, shrink or stay the same?
- Can you run the ablations isolating (i) per-feature tokenization, (ii) learned versus uniform feature-axis attention, and (iii) a feature-gated row-token model?
- How do TabDPT’s results change with retrieval enabled, as in its default configuration? What changed between TabDPT v1.0 and v1.3 that so strongly reduced null-feature sensitivity, and how does that bear on the architectural interpretation?
- For Section 4, can you add norm-matched random-direction perturbations and a Bayes-map reference Jacobian, and replicate across training seeds? How should the frequent negative own-feature responses (56/128 tasks at d = 10) be interpreted under the routing hypothesis?
- Can you report the controlled gap as unnormalized excess risk, and relative to the irreducible noise?
- Do the conclusions extend to correlated designs or to a sparse nonlinear prior where the exact Bayes rule is still computable (e.g., by Monte Carlo)?
6 comments
2. Adaptation to unknown sparsity (W2, Q2)
Following your suggestion, we trained both architectures on mixtures over support sizes without providing the generating to either model.
We used two mixtures: is uniform over or over . Both preserve the coefficient covariance and expected signal energy of the matched prior family. We evaluated each mixture against its exact Bayes rule, which infers from the context by posterior weighting. The generating was used only to stratify evaluation. We scaled the maximum training-step budget by the number of candidate support sizes to match expected exposure per support size at the fixed- cap, using the same mixture-training protocol for both architectures.
We compared fixed- and mixed- test errors in the figure below. Each training prior has its own Bayes predictor, so we compare the architectures within each prior. See Fig. 25 in Appendix E.3.4 (or in this link).
Alternating-axis retains an advantage under both mixtures. At , the normalized gap is smaller than under fixed- training; at , the gaps are similar. At , the near tie under fixed- training becomes an Alternating-axis advantage under mixture training. For mixed- tasks generated with , the Bayes target still marginalizes over unknown support size and differs from the known-dense reference.
We also extended the three-point mixture training to 10× its original mixture-training step cap, with early stopping disabled. At , Row-token continues improving, especially for , but Alternating-axis remains better late in training. At , Row-token flattens earlier and Alternating-axis retains lower error across support sizes. See Fig. 26 in Appendix E.3.4 (or in this link).
The Alternating-axis advantage persists when support size varies across tasks and must be inferred from context, including under extended training.
3. Learned attention, tokenization, and feature gating (W1, Q3)
(i) Per-feature tokenization and cross-feature communication. Our Alternating-axis model already assigns a separate token to each feature cell and an additional response token to each row. In this implementation, sample-axis attention operates independently within each column, the MLP acts on each token separately, and the decoder reads the query response token. Removing feature-axis attention from every block would therefore disconnect the covariates from the decoder. We used uniform mixing to retain this information path while examining the role of learned feature-attention weights.
(ii) Learned versus uniform feature-axis attention. Following your suggestion, we added two controls that set feature-axis attention weights to uniform in every layer while retaining feature tokens and the remaining computational pathways. We first imposed uniform attention on frozen checkpoints, with no parameter updates, to test their reliance on the learned attention configuration. We then trained new models from random initialization with uniform feature attention throughout, allowing the remaining parameters to adapt. The learned Alternating-axis baseline is labelled “Learned attention” below.
Normalized Bayes approximation error (mean ± SD across training seeds; lower is better):
| (d, k) | Learned attention | Frozen to Uniform | Uniform from scratch |
|---|---|---|---|
| (10, 1) | 0.0561 ± 0.0011 | 0.9677 ± 0.0379 | 0.0667 ± 0.0045 |
| (10, 5) | 0.0214 ± 0.0006 | 0.8461 ± 0.1439 | 0.3181 ± 0.5102 |
| (10, 10) | 0.0183 ± 0.0004 | 0.7775 ± 0.1844 | 0.3135 ± 0.5113 |
| (20, 1) | 0.0779 ± 0.0060 | 1.0785 ± 0.0986 | 0.0858 ± 0.0137 |
| (20, 10) | 0.0248 ± 0.0013 | 0.9404 ± 0.0676 | 0.0272 ± 0.0007 |
| (20, 20) | 0.0221 ± 0.0002 | 0.9489 ± 0.0822 | 0.0240 ± 0.0020 |
Imposing uniform attention on frozen checkpoints sharply increases error. Training with uniform attention from scratch recovers much of the performance, particularly at in both dimensions, although some cases show large variation across seeds. Thus, the trained checkpoints rely on their learned attention configuration, while low error can also be obtained by training with a fixed mixing rule.
To examine how the model adapts to fixed mixing weights, we look at the learned value representations. For intuition, we compare value magnitudes before and after attention weighting, using the simplified expressions and . Here, is the attention weight from query target token to feature token . The precise multi-head definitions used in our experiments are given in Appendix E.3.9. The figure reports the layerwise contrast as for both settings, where is the active-to-inactive ratio of mean norms on query rows from 3,200 held-out tasks per setting. See Fig. 31 in Appendix E.3.9 (or in this link). We observe the following phenomenon:
- Adaptation under uniform attention. Before attention weighting, the uniform-trained model has a larger active-to-inactive value-norm ratio than the model with learned attention. This suggests that the model compensates for equal attention weights by learning larger values for active features relative to inactive ones.
- Joint effect of learned attention and values. Attention weights and value representations jointly determine the feature messages. Active and inactive features are more clearly separated in the norms of these weighted messages than in the value norms alone.
These controls refine the mechanism interpretation of the original Row-token versus Alternating-axis comparison. The observed architecture gap under matched priors remains the empirical basis for our architectural claim. Low error with uniform attention is consistent with a benefit from preserving feature-indexed computation, while showing that learning feature-attention weights is not the only route to low error in this setting. The controls retain both feature tokens and cross-feature mixing, so they do not separately identify the contribution of tokenization.
(iii) Feature-gated Row-token models. We agree that adding a context-dependent feature gate to Row-token would be a useful alternative architecture. An input gate would test whether one-time feature selection is sufficient to close the gap, whereas Alternating-axis can update feature representations across layers. A systematic comparison requires specifying how context evidence is aggregated, where gates are applied, and how model capacity is allocated. We leave the design and evaluation of this alternative architecture to future work.
We thank the reviewer for the careful reading and constructive suggestions. We appreciate your recognition that our matched-prior comparison against an exact Bayes target provides a falsifiable, reusable protocol for studying architecture–prior alignment. We also appreciate your assessment that the agreement between theory, sparse-to-dense experiments, and coefficient-error decomposition provides evidence about the underlying computation. We respond to your questions and concerns point by point below.
1. Training budget, matched compute, and model capacity (W1, Q1)
Following your suggestion, we added longer training runs, compute-matched comparisons, and width and depth sweeps for the Row-token model.
Matched cumulative training FLOPs. We compared the two architectures at matched cumulative training FLOPs. Because Alternating-axis costs more per update, we allowed Row-token more updates at each common budget. Row-token improves with compute, but Alternating-axis retains lower normalized Bayes error under sparse priors in both dimensions at the largest shared budgets. Dense-prior errors are close. See the figure in this link.
Extended training (10× steps). We increased the training limit from 10,000 to 100,000 steps for d=10 and from 20,000 to 200,000 steps for d=20. We disabled early stopping and recorded validation Bayes-error trajectories throughout these runs. The sparse-prior gap remains clear at the extended endpoints, while the dense curves nearly overlap. See Fig. 28 in Appendix E.3.5 (or in this link).
Wider Row-token models. We trained wider Row-token models and compared them with the original three-block Alternating-axis baseline, shown by the horizontal reference lines; the x-axis gives Row-token embedding width. Increasing width does not close the sparse gap in either dimension over the tested range, while dense results remain close. See the figure in this link.
Deeper Row-token models. We trained Row-token models with 3 to 10 blocks. More depth helps in some settings, but the original three-block Alternating-axis baseline retains substantially lower sparse-prior error at the evaluated training-step caps. Dense-prior errors stay close. See Fig. 30 in Appendix E.3.7 (or in this link).
The sparse-prior gap persists after extending training, matching compute, and varying Row-token capacity within the tested ranges. Together with the close dense-prior results, this supports architecture–prior alignment under the evaluated training protocols. Some trajectories continue improving, so the empirical comparison is limited to the tested budgets. We use the response-affine theorem as a motivating example of the computational distinction between dense and sparse priors. It does not cover unrestricted deep Row-token models; our evidence for an architectural advantage comes from the controlled empirical comparisons.