acceptodds
Under review as a conference paper at ICLR 2027

Adapting Linear-Time Architectures for Tabular In-Context Learning

Abstract

Tabular foundation models achieve strong performance by conditioning on labelled examples in context, but softmax attention limits their use on large datasets. Existing linear-time alternatives, however, are mostly causal, and their potential for tabular in-context learning (ICL) remains underexplored. To address this, we (1) revisit causal training setups, (2) compare linear sequence mixers, and (3) investigate their ICL generalisation beyond the pretraining context length. First, we show that the best training setup for causal models resembles next-token prediction. Then, perhaps surprisingly, the most promising linear sequence mixer is causal: DeltaNet outperforms even non-causal linear attention. However, it degrades beyond - the pretraining context length, and existing mitigation strategies such as bidirectionality defer the problem at best. A hidden-state oracle shows that this is not a capacity problem. Instead, our analysis points to an instability in the recurrent state, which drifts in deeper layers of causal models. Since DeltaNet's learned write rates overfit to the pretraining regime, we modulate them with a time-dependent decay schedule intervention to stabilise length generalisation. Finally, re-introducing non-causality by reading out from the final state allows us to closely match a controlled softmax attention baseline on OpenML-CC18 and TabArena.

Primary area:
foundation or frontier models, including LLMs
Keywords:
In-context learning, Linear attention, Tabular foundation models, Length generalisation, Recurrent sequence models, DeltaNet

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_HXWnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a controlled experimental ablation study of various linear attention mechanisms. Its investigations cover sequence mixing, causal vs. non-causal setups, and context length extrapolation.

Critical strengths

The key strength of this paper is the sweep of insights it creates. The paper provides many experimental results and ablation insights that are likely to be of interest to the community. The general state of linear attention for tabular foundation models is moved forward by the contributions of this paper.

Critical weaknesses

The two critical weaknesses I see both stem from the controlled evaluation setup. Both make it hard to judge whether, and how, the generated insights generalize to real model development.

First, the fairness assumptions, which fix the prior, parameter counts, and hyperparameters, yield a generally weak model. All follow the TabPFNv1 setup, which is a very weak model at this point. Moreover, it remains unclear if the prior, hyperparameters, or parameter counts are even "good enough" for a linear attention-based model to work. They were barely good enough for a competitive softmax attention model. The work would benefit from justifying the choice of the assumption in depth and ensuring that experiments under these assumptions generalize to developing a real linear attention model using state-of-the-art technology. Moreover, with competitive, simple implementations like nanoTabPFN and nanoTabICL, it is unclear why the older PFN codebase was used, whose hyperparameter choices are outdated and weak.

Second, the evaluation also seems problematic. To stay within their assumptions, the benchmarks heavily subsample the feature dimension. Moreover, the evaluation metrics follow older protocols; instead, the latest recommended evaluation protocols (e.g., the authors use non-normalized aggregations). Both make the evaluations hard to trust. Subsampling destroys the meaningfulness of many datasets in these benchmarks, as real-world data does not consist of random feature subsamples. Likewise, the evaluation metrics may hide real differences in aggregation. Furthermore, neither seems necessary; the assumptions could have been met by a more careful dataset selection and benchmark design.

Questions for authors

N/A

7 comments
AuthorsOpenReview ↗

Response to Reviewer HXWn (1/2)

Thank you for your review and for noting that our experiments and ablations provide insights of interest to the community and move linear attention for tabular foundation models forward. You raise two concerns, about the controlled setup and about the evaluation, which both ask whether the insights generalise. We answer them below and have marked changes in the revised PDF in orange.

1. Controlled setup. We fixed the prior, model size and hyperparameters so that differences between models can be attributed to the sequence mixer. The row-wise TabPFN v1 architecture has a single attention mechanism, which makes this attribution possible, and it matches TabFlex, the closest prior work. We agree that our models are below the state of the art in raw performance, as TabPFN v2.5 and TabICLv2 lead our softmax model by 0.007 to 0.009 ROC-AUC on OpenML-CC18* and by 0.02 on TabArena* (Table 2). This setup is nevertheless strong enough for linear attention to work. Delta-FS + Decay matches or exceeds the released TabFlex model on all four metrics (ROC-AUC 0.8884 vs 0.8859 on OpenML-CC18*, 0.8139 vs 0.8089 on TabArena*), although TabFlex had a much larger pretraining budget. The hidden-state oracle (Section 4.1) also shows that a fixed-size state of our model size can outperform softmax attention, so state capacity is not the limit. Section 3.1 now motivates the setup, and the limitations in Section 6 state the gap to the state of the art.

To test whether the findings generalise to a modern prior, we retrained softmax attention, non-causal linear attention, standard DeltaNet, Delta-FS and Delta-FS + Decay on the public TabICLv2 prior with up to 100 input features. Because of the rebuttal timeline, training uses half the steps of the submission. Furthermore, no OpenML-CC18 dataset then needs feature subsampling, and only the five TabArena datasets with more than 100 features are subsampled (Appendix D.11).

ModelOpenML-CC18*TabArena*1k rows4k16k64k
Softmax0.8987 (2.08)0.8380 (2.32)0.71820.72870.72380.7267
Delta-FS + Decay0.8944 (2.53)0.8379 (2.39)0.71760.71970.72370.7216
Delta-FS0.8935 (3.27)0.8361 (2.47)0.71460.71990.71000.6871
Delta0.8926 (3.65)0.8323 (3.53)0.71070.71660.71900.7038
Linear Non-Causal0.8937 (3.47)0.8260 (4.29)0.71050.71750.71760.7208

We report mean ROC-AUC, with the rank among the five models in brackets. The length columns use 200 held-out tables from the TabICLv2 prior, predicting 100 rows from a context of the given length, and pretraining contexts have at most 1k rows. Delta-FS + Decay again outperforms non-causal linear attention, clearly on TabArena* and slightly on OpenML-CC18*, and the decay again prevents the degradation beyond the pretraining length. Softmax attention remains the best model, but Delta-FS + Decay is tied with it on TabArena* and 0.004 behind on OpenML-CC18*.

Aside from this check for generalisation to a modern prior, the submission also reproduced the failure under a simpler prior (Section 4.3), with pretraining contexts of up to 64k rows (Appendix D.4), and with TabPFN-v2-style two-axis attention (Appendix D.9). We have now also tested the write-rate decay in the two-axis model. Applied only at evaluation to its frozen weights, it removes the degradation and raises the ROC-AUC at 128k rows from 0.706 to 0.782, against 0.781 at 1k rows.

We also checked our findings on nanoTabPFN, as nanoTabICL provides no pretraining code and refers to nanoTabPFN for it. nanoTabPFN is an independent pipeline with its own two-axis architecture, optimiser and prior data, although much smaller than ours (3 layers, 0.36M parameters, pretraining tables with 150 rows, 5 features and 2 classes). We kept all of this apart from a tuned learning rate and replaced only the attention between rows by our mixers, with five seeds each:

Mean ROC-AUC on 200 held-out tables128 rows5122k8k16k
Softmax (nanoTabPFN default)0.6030.6050.6130.6140.615
Delta-FS + Decay0.6070.6120.6180.6160.615
Delta-FS0.6080.6040.5880.5560.545
Delta0.6090.6110.5950.5520.538
Linear Non-Causal0.6000.6020.6090.6100.610

Pretraining contexts have at most 134 rows. Beyond them, DeltaNet degrades both with causal and final-state readout, while with the decay it stays level with the softmax model. The decay constant is c = 38, which is our paper's c = 256 scaled linearly to the 150-row pretraining tables.

AuthorsOpenReview ↗

Response to Reviewer HXWn (2/2)

2. Evaluation. We agree that random feature subsampling is not ideal. The 20-feature limit affected 13 of 30 OpenML-CC18 and 21 of 38 TabArena datasets. Following your suggestion of a more careful dataset selection, we repeated the comparison on the 17 datasets per benchmark that fit the limit, so that no feature is removed:

ModelOpenML-CC18* (17 datasets) ranknorm.ROC-AUCTabArena* (17 datasets) ranknorm.ROC-AUC
Softmax Non-Causal4.350.720.86313.470.760.8289
Delta-FS + Decay4.060.710.86583.180.720.8249
Delta + Decay (Int-MT)6.470.500.86333.590.750.8252
Delta + Decay6.120.520.86384.650.630.8225
Delta-FS4.940.660.86426.530.470.8116
Delta5.240.610.86356.120.580.8201
Delta Gated6.000.540.86335.820.600.8200
Linear Non-Causal5.710.560.85797.290.430.8103
Linear Gated7.710.400.86097.820.440.8159
Linear Causal7.350.410.85917.940.380.8152
Mamba-28.060.310.85669.590.180.8033

The conclusions are the same as on the full benchmarks. Delta-FS + Decay and softmax attention are on par. On TabArena*, the decay clearly improves Delta-FS (rank 3.18 instead of 6.53), and Delta-FS + Decay stays well ahead of non-causal linear attention (7.29). The TabICLv2 models of point 1 take up to 100 features, so only five TabArena datasets still need subsampling there, with the same findings. The revision adds this analysis, including paired tests, as Appendix D.10 and refers to it in the evaluation protocol (Section 3.1).

On the aggregation, the submission compared all model pairs with paired tests over datasets and Holm correction (Appendices C.2 and D.8), but its main tables reported only raw means. Tables 2 and 3 now also report the mean ROC-AUC rank and the min-max-normalised ROC-AUC:

ModelOpenML-CC18* rank (30 datasets)norm.ROC-AUCTabArena* rank (38 datasets)norm.ROC-AUC
Softmax Non-Causal3.100.820.89003.050.820.8172
Delta-FS + Decay4.000.690.88843.340.730.8139
Delta + Decay (Int-MT)5.530.560.88733.420.760.8141
Delta + Decay6.200.490.88644.760.620.8100
Delta-FS4.530.650.88746.450.520.8022
Delta5.700.550.88606.000.570.8074
Delta Gated6.430.490.88575.920.570.8078
Linear Non-Causal5.830.530.88337.890.380.8004
Linear Gated8.030.360.88277.550.450.8038
Linear Causal8.000.340.88177.710.410.8033
Mamba-28.630.230.87849.890.140.7912

Rank is the mean ROC-AUC rank among the 11 models, and norm. the min-max-normalised ROC-AUC averaged over datasets (details in Appendix C.1). The best value per column is in bold. The conclusions of the paper hold under each aggregation. Softmax attention keeps a small lead over Delta-FS + Decay, which is not significant after Holm correction (Appendix D.8). The decay matters where contexts exceed the pretraining length, improving the rank of Delta-FS from 6.45 to 3.34 on TabArena* but only from 4.53 to 4.00 on OpenML-CC18*. We keep the raw means and paired tests as the primary comparison, because ranks and min-max scores depend on the set of compared models.

Thank you again for these suggestions. The new experiments and analyses are in the revised paper, apart from the nanoTabPFN check, and we believe they strengthen both the evaluation and the evidence that the findings generalise. We hope that this addresses your concerns and would be grateful if you would consider updating your score. In case there are any remaining questions or concerns, we are happy to follow up.

Thank you for the additional experiments and new experiments. The current responses to my raised weaknesses are not sufficient to change my mind.

Let me follow up on both points with questions:

(1) Now that you switched to a newer prior, how did you make sure, for these next experiments and in general, that the sampling configuration of the prior, the deep learning hyperparameters (learning rates, etc.), or parameter counts are optimal for your linear attention-based models?

(2) Adding rank and normalized metrics is helpful, but the current setup seems to always limit it to only the new models suggested in this work. A proper evaluation requires comparison to all baselines. What is the Elo of your "Softmax Non-Causal" version on TabArena, or its normalized performance or mean rank when computed together with baselines? How would the rank scale in Table 2 look with TabPFN v2.5 and the other models included? You claim "Decay matches or exceeds the released TabFlex". TabFlex has a mean rank of ~86 on TabArena, comparable to a default Random Forest.

Reviewer_gudmOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates various linear-time training setup / strategies for tabular models. It finds degradation (and their causes) amongst all strategies, even with mitigation strategies in the mix, and shows how to approximately match softmax attention.

Critical strengths
  • Very well-written, both technically and in terms of prose.
  • Very clear introduction, framing, and setup. Background section is very thorough.
  • A good, reflective, ethics section
  • Well-balanced discussion with explicit shortcomings for the given approach.
Critical weaknesses

There aren't 'critical' weaknesses per se, but the writing does degrade in the last page. The discussion and conclusion are essentially merged in, and the discussion itself does not read like a discussion. It is important to decouple the numbers from the results (sec. 4, 5) with their interpretation: this maintains objectivity. It should also try to explain why the results the authors observed are the way they are, not just summarise results. As it stands, it is quite hard to get a good takeaway of the work. Likewise, the conclusion is meant to tie together the interpretation to the hypothesis tested in the work, not necessarily repeat what the discussion said: you interpreted your results in some way--then what? Since this paper is meant to be an evaluation, these sections are very important.

This also extends to claims and their scope. It is very minor, but the sentence (L482) that stable-state construction is 'the key to learning a generalising ICL algorithm' should (re)state that this is within tabular foundation models, due to their differences in training/setup.

Aside, this is more of a style matter, but (L485) 'We hope that these insights pave the way...' is not necessarily something that the conclusion needs. In fact, the conclusion appears disjointed from the thesis itself: it shows a solution while the thesis is--for lack of a better word--diagnostic. Future work is interesting, but given how rushed the last section appears to be, I'd advice the authors to axe that in favour of a clearer tie-in between results and discussion.

There's also some work that could provide alternate explanations on the degradation / stability of the models over shots. I recommend the authors to do a slightly more thorough literature review in that aspect of their work, since it is the core comparison.

What is DeltaNet? (i.e., the transition in the abstract introducing DeltaNet is quite awkward; L18).

Finally, L37 ('so-called tabular models') -- this reads quite flippant, and should be removed.

Questions for authors

Table 2,3's bars are clever, but are a bit hard to interpret. What does the gradient mean? I might have missed it, but why do causal models degrade more? It seems that this work mostly finds what does not explain the degradation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer gudm

Thank you for the positive and constructive review. We are glad that you found the paper very well written, with a clear introduction, framing and setup. We are especially grateful for your concrete guidance on how to improve the discussion and conclusion, which we followed closely. All changes are marked in orange in the revised PDF.

We separated the discussion from the conclusion. The discussion (Section 6) now interprets the results and explains why they occur instead of summarising them. We also removed the future-work list and the "We hope..." sentence. Furthermore, we added a new paragraph that relates the degradation to alternative explanations by Buitrago & Gu, Chen et al. and Ben-Kish et al. Our write-rate finding is closest to Ben-Kish et al., who trace the failure of Mamba to decay rates fitted to the training length. The conclusion (Section 7) ties this interpretation back to the question of the paper, whether ordered updates can aggregate the exchangeable rows of a table.

Does the revised version match what you had in mind? We would be happy to further refine the discussion and conclusion.

We also stated the claim of L482 for tabular foundation models, introduced DeltaNet in the abstract as a recurrence that overwrites its fast-weight memory with the delta rule, and removed "so-called" (L37).

The bars in Tables 2 and 3 mark groups of models whose differences are not significant, so two models differ significantly exactly when they share no bar. The colours only identify the groups, and we coloured them from the group of the strongest to that of the weakest model, which created the impression of a gradient. Section 3.1 now explains this.

On why causal models degrade more, we agree that the submission mainly ruled out explanations. For DeltaNet, the primary cause is not causal access, since final-state readout and bidirectionality degrade as well. We found that DeltaNet keeps writing into its state at rates fitted to the pretraining length. We tested this by applying the write-rate decay only during evaluation to the frozen weights. This removed the degradation for both checkpoints in Table 8 and for a two-axis model (Appendices D.7 and D.9). For additive linear attention, the mechanism remains open, as Section 6 now states.

Thank you again for your guidance. We believe the changes substantially improved the discussion and conclusion, and we are happy to follow up on any feedback.

AuthorsOpenReview ↗

Revised Discussion (Section 6)

6. Discussion

In this paper, we studied linear-time architectures for tabular foundation models through the lens of parametric, as opposed to non-parametric, in-context learning. This perspective suggests that merely replacing softmax attention by a linear variant is a significant conceptual shift that requires careful adaptation.

Given this perspective, our results show that considerations like permutation invariance or non-causality are less important than the nature of the recurrent update rule. For example, at moderate context lengths, the causal DeltaNet outperforms a non-causal linear attention baseline and remains competitive with the externally trained TabFlex checkpoint. This advantage comes from the delta rule, which first removes what the state already predicts and only then writes, limiting the interference between correlated rows that additive updates suffer from. Our results further show that causal models benefit from an adapted training setup, in which input and label embeddings are interleaved and trained with multiple targets. We speculate that this implicitly enlarges the context seen during pretraining, similar to causal next-token prediction in language modelling. Beyond that, we explore non-causal extensions of linear-time architectures, such as bidirectionality (Afzal et al., 2025). Within the pretraining range, our results suggest reading out from the final state as a more powerful novel adaptation in the setting of test-time regression. This may be of particular interest for other non-causal modelling domains, such as bidirectional sequence modelling (Hwang et al., 2024). Finally, the parametric state enables previously underexplored compression mechanisms such as end-to-end optimisation at test time, and our experiments with the hidden-state oracle point towards a large potential to improve parametric update rules even in a non-transductive setting.

The parametric approach also introduces failure modes that require adaptation. For example, our length-generalisation results show that order-dependent, and specifically causal, models can fail to learn a generalising ICL algorithm. This has also been studied in language modelling, but the tabular setting with its stationary in-context distribution arguably allows us to isolate the phenomenon further. For tabular ICL, we found that neither existing mitigation strategies nor non-causal access through bidirectionality or final-state readout solves the failure, and the hidden-state oracle rules out limited state capacity. Instead, our analysis suggests that the recurrent dynamics overfit to the pretraining range and then drift when the context is extended beyond it. Specifically for DeltaNet, we design an intervention which modulates the write rates βt\beta_t with a time-dependent decay schedule and show that this adaptation stabilises the parametric estimator beyond the pretraining range, even when applied only at evaluation to frozen checkpoints (Appendices D.7 and D.9). The resulting model is the best-performing linear-time model in our evaluation, closing most of the gap to softmax attention, especially in the interleaved multi-target setting or when extended for non-causality via final-state readout. For additive linear attention, which has no write rates, the mechanism behind the drift remains open.

Compared to earlier length generalisation work in language modelling, our drift results are consistent with the unexplored-states hypothesis of Buitrago & Gu (2025), although the state passing they propose helps GLA but not DeltaNet (Section 4.2). Our write-rate finding is closest to Ben-Kish et al. (2025), who trace the long-context failure of Mamba to a limited effective receptive field, where early information does not propagate to the output. Our final-state contribution analysis shows the same for DeltaNet (Figure 11). Chen et al. (2025) instead find that Mamba learns to forget too little when trained on contexts that are short for its state size, which is the opposite of the overwriting behaviour we observe in DeltaNet.

Limitations. We prioritise a controlled comparison over absolute performance, so our models remain below the state of the art. Larger pretraining budgets and further tuning may disproportionately improve performance. We use row-wise tabular ICL with the TabPFN v1 prior and at most 20 input features. The conclusions hold on the datasets that never needed feature subsampling (Appendix D.10), and the main findings carry over to a TabPFN-v2-style two-axis model (Appendix D.9) and to models retrained on the TabICLv2 prior with up to 100 features and half the training budget (Appendix D.11). Much larger model scales remain open. Finally, the decay stops the degradation, but scores plateau from about 8k rows, so we do not claim that all additional rows are exploited effectively.

AuthorsOpenReview ↗

Revised Conclusion (Section 7)

7. Conclusion

This paper explores how linear-time architectures need to be adapted when employed as tabular foundation models. We showed that ordered updates can aggregate information across exchangeable rows in tables within the pretraining range, but beyond it they degrade because of how the state is constructed, while its capacity suffices. Across various considered settings, stable state construction is therefore the key challenge for tabular foundation models to learn a generalising ICL algorithm. The write-rate decay tests this diagnosis, and stabilising the state in this way, combined with a final-state readout, closes most of the gap to softmax attention in our controlled setting.

References

  • Afzal et al. (2025). Linear Attention for Efficient Bidirectional Sequence Modeling. NeurIPS 2025.
  • Ben-Kish et al. (2025). DeciMamba: Exploring the Length Extrapolation Potential of Mamba. ICLR 2025.
  • Buitrago & Gu (2025). Understanding and Improving Length Generalization in Recurrent Models. ICML 2025.
  • Chen et al. (2025). Stuffed Mamba: Oversized States Lead to the Inability to Forget. COLM 2025.
  • Hwang et al. (2024). Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers. NeurIPS 2024.

I appreciate the author's response and I believe the changes fully address my concerns. I keep my score since it reflects the paper's status.

Comments on the paper

1 comment

We thank both reviewers for their insightful and constructive feedback. The reviewers found the paper to be "very well-written", creating a "sweep of insights" that "likely move the state of linear attention for tabular foundation models forward".

There were concerns regarding the writing of discussion and conclusion, the presented evaluation, and the relevance of the investigated TFM setting. Following the reviewer's guidance, we were able to address the writing and evaluation and updated the main paper accordingly. In an effort to provide more evidence that our insights transfer, we validated our interventions across a range of new settings in Appendix E and referred to them in the main paper. Ultimately, there seems to be a fundamental disagreement with reviewer HXWn19 on what constitutes an acceptable research contribution. Simplifying state-of-the-art TFMs to researchable methods is an ongoing effort by in the community. For example, the reviewer proposed nanoTabPFN as a state-of-the-art research baseline so we would like to highlight specifically that the insights from our interventions transfer to nanoTabPFN and the state-of-the-art TabICLv2 prior. For the detailed arguments against and in favour of our approach, we would like to refer the AC to the discussion following our initial response here.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.