- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a controlled experimental ablation study of various linear attention mechanisms. Its investigations cover sequence mixing, causal vs. non-causal setups, and context length extrapolation.
Critical strengths
The key strength of this paper is the sweep of insights it creates. The paper provides many experimental results and ablation insights that are likely to be of interest to the community. The general state of linear attention for tabular foundation models is moved forward by the contributions of this paper.
Critical weaknesses
The two critical weaknesses I see both stem from the controlled evaluation setup. Both make it hard to judge whether, and how, the generated insights generalize to real model development.
First, the fairness assumptions, which fix the prior, parameter counts, and hyperparameters, yield a generally weak model. All follow the TabPFNv1 setup, which is a very weak model at this point. Moreover, it remains unclear if the prior, hyperparameters, or parameter counts are even "good enough" for a linear attention-based model to work. They were barely good enough for a competitive softmax attention model. The work would benefit from justifying the choice of the assumption in depth and ensuring that experiments under these assumptions generalize to developing a real linear attention model using state-of-the-art technology. Moreover, with competitive, simple implementations like nanoTabPFN and nanoTabICL, it is unclear why the older PFN codebase was used, whose hyperparameter choices are outdated and weak.
Second, the evaluation also seems problematic. To stay within their assumptions, the benchmarks heavily subsample the feature dimension. Moreover, the evaluation metrics follow older protocols; instead, the latest recommended evaluation protocols (e.g., the authors use non-normalized aggregations). Both make the evaluations hard to trust. Subsampling destroys the meaningfulness of many datasets in these benchmarks, as real-world data does not consist of random feature subsamples. Likewise, the evaluation metrics may hide real differences in aggregation. Furthermore, neither seems necessary; the assumptions could have been met by a more careful dataset selection and benchmark design.
Questions for authors
N/A
7 comments
Response to Reviewer HXWn (2/2)
2. Evaluation. We agree that random feature subsampling is not ideal. The 20-feature limit affected 13 of 30 OpenML-CC18 and 21 of 38 TabArena datasets. Following your suggestion of a more careful dataset selection, we repeated the comparison on the 17 datasets per benchmark that fit the limit, so that no feature is removed:
| Model | OpenML-CC18* (17 datasets) rank | norm. | ROC-AUC | TabArena* (17 datasets) rank | norm. | ROC-AUC |
|---|---|---|---|---|---|---|
| Softmax Non-Causal | 4.35 | 0.72 | 0.8631 | 3.47 | 0.76 | 0.8289 |
| Delta-FS + Decay | 4.06 | 0.71 | 0.8658 | 3.18 | 0.72 | 0.8249 |
| Delta + Decay (Int-MT) | 6.47 | 0.50 | 0.8633 | 3.59 | 0.75 | 0.8252 |
| Delta + Decay | 6.12 | 0.52 | 0.8638 | 4.65 | 0.63 | 0.8225 |
| Delta-FS | 4.94 | 0.66 | 0.8642 | 6.53 | 0.47 | 0.8116 |
| Delta | 5.24 | 0.61 | 0.8635 | 6.12 | 0.58 | 0.8201 |
| Delta Gated | 6.00 | 0.54 | 0.8633 | 5.82 | 0.60 | 0.8200 |
| Linear Non-Causal | 5.71 | 0.56 | 0.8579 | 7.29 | 0.43 | 0.8103 |
| Linear Gated | 7.71 | 0.40 | 0.8609 | 7.82 | 0.44 | 0.8159 |
| Linear Causal | 7.35 | 0.41 | 0.8591 | 7.94 | 0.38 | 0.8152 |
| Mamba-2 | 8.06 | 0.31 | 0.8566 | 9.59 | 0.18 | 0.8033 |
The conclusions are the same as on the full benchmarks. Delta-FS + Decay and softmax attention are on par. On TabArena*, the decay clearly improves Delta-FS (rank 3.18 instead of 6.53), and Delta-FS + Decay stays well ahead of non-causal linear attention (7.29). The TabICLv2 models of point 1 take up to 100 features, so only five TabArena datasets still need subsampling there, with the same findings. The revision adds this analysis, including paired tests, as Appendix D.10 and refers to it in the evaluation protocol (Section 3.1).
On the aggregation, the submission compared all model pairs with paired tests over datasets and Holm correction (Appendices C.2 and D.8), but its main tables reported only raw means. Tables 2 and 3 now also report the mean ROC-AUC rank and the min-max-normalised ROC-AUC:
| Model | OpenML-CC18* rank (30 datasets) | norm. | ROC-AUC | TabArena* rank (38 datasets) | norm. | ROC-AUC |
|---|---|---|---|---|---|---|
| Softmax Non-Causal | 3.10 | 0.82 | 0.8900 | 3.05 | 0.82 | 0.8172 |
| Delta-FS + Decay | 4.00 | 0.69 | 0.8884 | 3.34 | 0.73 | 0.8139 |
| Delta + Decay (Int-MT) | 5.53 | 0.56 | 0.8873 | 3.42 | 0.76 | 0.8141 |
| Delta + Decay | 6.20 | 0.49 | 0.8864 | 4.76 | 0.62 | 0.8100 |
| Delta-FS | 4.53 | 0.65 | 0.8874 | 6.45 | 0.52 | 0.8022 |
| Delta | 5.70 | 0.55 | 0.8860 | 6.00 | 0.57 | 0.8074 |
| Delta Gated | 6.43 | 0.49 | 0.8857 | 5.92 | 0.57 | 0.8078 |
| Linear Non-Causal | 5.83 | 0.53 | 0.8833 | 7.89 | 0.38 | 0.8004 |
| Linear Gated | 8.03 | 0.36 | 0.8827 | 7.55 | 0.45 | 0.8038 |
| Linear Causal | 8.00 | 0.34 | 0.8817 | 7.71 | 0.41 | 0.8033 |
| Mamba-2 | 8.63 | 0.23 | 0.8784 | 9.89 | 0.14 | 0.7912 |
Rank is the mean ROC-AUC rank among the 11 models, and norm. the min-max-normalised ROC-AUC averaged over datasets (details in Appendix C.1). The best value per column is in bold. The conclusions of the paper hold under each aggregation. Softmax attention keeps a small lead over Delta-FS + Decay, which is not significant after Holm correction (Appendix D.8). The decay matters where contexts exceed the pretraining length, improving the rank of Delta-FS from 6.45 to 3.34 on TabArena* but only from 4.53 to 4.00 on OpenML-CC18*. We keep the raw means and paired tests as the primary comparison, because ranks and min-max scores depend on the set of compared models.
Thank you again for these suggestions. The new experiments and analyses are in the revised paper, apart from the nanoTabPFN check, and we believe they strengthen both the evaluation and the evidence that the findings generalise. We hope that this addresses your concerns and would be grateful if you would consider updating your score. In case there are any remaining questions or concerns, we are happy to follow up.
Thank you for the additional experiments and new experiments. The current responses to my raised weaknesses are not sufficient to change my mind.
Let me follow up on both points with questions:
(1) Now that you switched to a newer prior, how did you make sure, for these next experiments and in general, that the sampling configuration of the prior, the deep learning hyperparameters (learning rates, etc.), or parameter counts are optimal for your linear attention-based models?
(2) Adding rank and normalized metrics is helpful, but the current setup seems to always limit it to only the new models suggested in this work. A proper evaluation requires comparison to all baselines. What is the Elo of your "Softmax Non-Causal" version on TabArena, or its normalized performance or mean rank when computed together with baselines? How would the rank scale in Table 2 look with TabPFN v2.5 and the other models included? You claim "Decay matches or exceeds the released TabFlex". TabFlex has a mean rank of ~86 on TabArena, comparable to a default Random Forest.
Response to Reviewer HXWn (1/2)
Thank you for your review and for noting that our experiments and ablations provide insights of interest to the community and move linear attention for tabular foundation models forward. You raise two concerns, about the controlled setup and about the evaluation, which both ask whether the insights generalise. We answer them below and have marked changes in the revised PDF in orange.
1. Controlled setup. We fixed the prior, model size and hyperparameters so that differences between models can be attributed to the sequence mixer. The row-wise TabPFN v1 architecture has a single attention mechanism, which makes this attribution possible, and it matches TabFlex, the closest prior work. We agree that our models are below the state of the art in raw performance, as TabPFN v2.5 and TabICLv2 lead our softmax model by 0.007 to 0.009 ROC-AUC on OpenML-CC18* and by 0.02 on TabArena* (Table 2). This setup is nevertheless strong enough for linear attention to work. Delta-FS + Decay matches or exceeds the released TabFlex model on all four metrics (ROC-AUC 0.8884 vs 0.8859 on OpenML-CC18*, 0.8139 vs 0.8089 on TabArena*), although TabFlex had a much larger pretraining budget. The hidden-state oracle (Section 4.1) also shows that a fixed-size state of our model size can outperform softmax attention, so state capacity is not the limit. Section 3.1 now motivates the setup, and the limitations in Section 6 state the gap to the state of the art.
To test whether the findings generalise to a modern prior, we retrained softmax attention, non-causal linear attention, standard DeltaNet, Delta-FS and Delta-FS + Decay on the public TabICLv2 prior with up to 100 input features. Because of the rebuttal timeline, training uses half the steps of the submission. Furthermore, no OpenML-CC18 dataset then needs feature subsampling, and only the five TabArena datasets with more than 100 features are subsampled (Appendix D.11).
We report mean ROC-AUC, with the rank among the five models in brackets. The length columns use 200 held-out tables from the TabICLv2 prior, predicting 100 rows from a context of the given length, and pretraining contexts have at most 1k rows. Delta-FS + Decay again outperforms non-causal linear attention, clearly on TabArena* and slightly on OpenML-CC18*, and the decay again prevents the degradation beyond the pretraining length. Softmax attention remains the best model, but Delta-FS + Decay is tied with it on TabArena* and 0.004 behind on OpenML-CC18*.
Aside from this check for generalisation to a modern prior, the submission also reproduced the failure under a simpler prior (Section 4.3), with pretraining contexts of up to 64k rows (Appendix D.4), and with TabPFN-v2-style two-axis attention (Appendix D.9). We have now also tested the write-rate decay in the two-axis model. Applied only at evaluation to its frozen weights, it removes the degradation and raises the ROC-AUC at 128k rows from 0.706 to 0.782, against 0.781 at 1k rows.
We also checked our findings on nanoTabPFN, as nanoTabICL provides no pretraining code and refers to nanoTabPFN for it. nanoTabPFN is an independent pipeline with its own two-axis architecture, optimiser and prior data, although much smaller than ours (3 layers, 0.36M parameters, pretraining tables with 150 rows, 5 features and 2 classes). We kept all of this apart from a tuned learning rate and replaced only the attention between rows by our mixers, with five seeds each:
Pretraining contexts have at most 134 rows. Beyond them, DeltaNet degrades both with causal and final-state readout, while with the decay it stays level with the softmax model. The decay constant is c = 38, which is our paper's c = 256 scaled linearly to the 150-row pretraining tables.