- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper investigates the theoretical properties of a simple restricted softmax attention model. Specifically, it examines whether such a model can learn to predict the location of a label transition if trained using gradient descent, under narrow data distribution assumptions. The paper proves that under infinite data, gradient descent approaches the optimal solution in the limit. It also establishes that gradient descent achieves bounded estimation error on finite data, with the bound tightening with more training data. The authors then perform synthetic experiments that validate the correctness of the theory.
Critical strengths
- Although I am not a theoretical ML researcher, the technical work that went into producing the theory seems substantial.
- To the best of my understanding, the theoretical results are correct and accurately capture the behaviour of the restricted softmax attention model in the scenario under investigation.
Critical weaknesses
- Unclear practical motivation. As I am not a theoretical ML researcher, I will not place strong judgement on this, as I am unaware of the standards/norms in this research area. However, I am unable to discern how the theoretical results could be of use to practitioners. The paper weakly motivates the research via in-context learning, transformer learning on tabular data and amortised estimation; however, I am struggling to see how the insights produced around the narrow problem scenario investigated port to these motivating problems. The theory pertains to direct supervision of an extremely restricted two-parameter softmax attention model to predict a one-dimensional transition threshold, where the threshold is the explicit training target. As far as I can tell, this has very little to do with practical transformer learning, or even simple one layer neural networks, which use fully parameterised attention and are trained end to end with a task loss. The ICL framing is also weak -- the problem can be simply cast as supervised learning from what I can tell.
- Presentation could be improved. As mentioned above, the motivation of the problem is loosely linked to the actual research contribution, making reading the paper quite confusing. Furthermore, although the structure of the paper is laid out at the end of the introduction, it is confusing to follow section to section -- in my opinion, how results depend on each other should be better signposted. The experimental figures are difficult to parse: text is far too small; legends and axis labels use locally unexplained notation; and captions rely heavily on the main body -- they are not understandable in isolation. Good figures should not only state empirical results but also improve/enable better understanding/intuition for the reader.
Questions for authors
Could the authors describe to me a practical scenario in which the theoretical insight in this paper could be of use?
4 comments
Official Comment by Authors
Part 2/2
Our scalar query attention is closely connected to practical architecture
Our scalar head isolates a standard attention operation. A readout query attends to a set of observations, assigning each a dot-product score and pooling the feature values . The column encoder in TabICL [3, Section 3.2] uses the same query-to-set aggregation operation across rows. The two-parameter restriction lets us track how gradient descent changes the balance between label and feature scores and how tightly the head concentrates.
What theoretical insights connect to practical models?
Our analysis reveals that in a noiseless threshold estimation problem, an attention head can learn to concentrate on the closest observation for a relevant class (positive class () vs. negative class ()). This attention concentration is also observed in TabPFN and TabICL, with their most focused attention heads placing 44% and 82% of their attention mass, respectively, on the two observations closest to the threshold at just .
Furthermore, our analysis reveals that an attention head first learns to select the relevant label class, and then concentrates on the observations nearest the threshold within that class. To test whether this persists beyond our restricted model, we trained standard one- and four-layer transformers, with fully parameterized attention and MLP blocks, end to end from scratch on in-context threshold inference (one seed each). Specific heads pass through both phases in this order (Table R9). For example, head 0 of the one-layer model first moves its attention onto the positive class while its attention stays spread out, and only then concentrates near . The same holds when the model must also infer which of four features to split on, where a head ends up concentrating along the split feature only. On the context sizes used in training, the errors of the one- and four-layer models are at most and times that of the Bayes estimator.
Table R9. Transformers trained from scratch. Each entry gives the attention mass on the class the head selects / the attention-weighted distance to , along the split feature for the four-feature model (; uniform attention gives distance ). Each layer has 4 heads; layers and heads are numbered from 0. Each head first selects a class, then concentrates.
| Training step | 1 layer: head 0 (positives) | 4 layers: layer 1, head 2 (negatives) | 4 layers: layer 3, head 2 (positives) | 4 layers, 4 features: layer 3, head 2 (positives) |
|---|---|---|---|---|
| 1,024 | / | / | / | / |
| 2,048 | / | / | / | / |
| 4,096 | / | / | / | / |
| Final | / | / | / | / |
We agree that our paper may not have made this connections sufficiently explicit. We will clarify these connections in the introduction, signpost how the training analysis leads to the statistical guarantee, and enlarge the figure labels and define the plotted quantities in self-contained captions.
References:
[3] J. Qu, D. Holzmüller, G. Varoquaux, and M. Le Morvan. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data. ICML, 2025.
I apologise to the authors for the delayed reply, I have been busy with personal work.
I appreciate the effort that the authors have put into improving the manuscript given reviewer feedback -- the presentation has considerably improved. However, I feel like the additional experiments presented in the rebuttal do not quite address my concerns about practical motivation. My understanding is that the rebuttal illustrates the following two things
- Models like TabPFN that predict Y given X are able to infer implicitly the decision boundary.
- More complete transformer models, trained to explicitly predict the threshold also demonstrate similar attention concentration behaviour as discussed theoretically for the restricted model in this paper.
To me there seems to be a distinct motivational gap. The practical behaviour of interest is one where a threshold is inferred implicitly from inputs and labels, (e.g. TabPFN) where the threshold is never provided as explicit supervision or produced by the model as an explicit output. On the other hand, the theoretical scenario focused on by the paper is one where the decision threshold is provided as the training target. Inferring a threshold from inputs and labels because it is necessary for minimising a classification loss seems quite different to predicting a threshold using direct supervision. To illustrate my point, one (unlikely) way to close the gap would be to demonstrate a mechanism by which a model like TabPFN learns to internally predict the decision threshold value explicitly and then mechanistically use that for the final classification prediction.
I will re-iterate that as I am not a theoretical researcher, I will defer to other reviewers' opinions on the contribution of the paper. However, given the above, I will keep my current score.
Official Comment by Authors
Thank you for the follow-up. We agree that our model is pretrained on the threshold directly, whereas TabPFN and TabICL are pretrained on query labels; at inference, both see only a labeled context. In our setting, however, both objectives require the same computation. The context informs only through the largest positive and the smallest negative observations, so for any prior the posterior predictive is determined by these two boundary observations. A model trained on query labels must therefore locate them, even without producing as an output. Consistent with this, at a query between these observations, the most concentrated heads of TabPFN and TabICL place a large share of their attention on these two observations, far more than uniform attention would. Our theory analyzes how gradient-based pretraining learns this localization with attention. Analyzing pretraining under a query-label objective remains open, and we will expand our discussion of this limitation in the revision.
Official Comment by Authors
Part 1/2We thank the reviewer for recognizing the substantial technical work and the correctness of our theory. We respond to the weaknesses and question below.
In-context estimation differs from conventional supervised learning
We agree that one can cast our problem as supervised learning, with each dataset as a "training example" and the threshold as its label. Indeed, prior work on ICL for statistical estimation concedes this point, calling it "supervised learning of statistical estimators" [2]. The key distinction from conventional supervised learning lies in the structure of each input (a dataset). Its "coordinates" are exchangeable, since the datapoints in each dataset are i.i.d., and different inputs can have different lengths (dataset sizes). Both properties affect the target learning rates, with our result showing how it decomposes into a pretraining term and a fresh-task term depending on the query context size N.
Existing practical models perform in-context threshold inference
In-context threshold inference is practically relevant. MetaTree [4] is a direct example of a transformer trained to select tree splits in-context. Furthermore, state-of-the-art tabular foundation models like TabICL [3] and TabPFN [1] are trained with tree function priors and need to be able to infer thresholds in-context as a subroutine. We have ran new experiments showing that they do so close to optimally, nearly approaching the Bayes benchmark (Table R8), TabICL at every tested context size and TabPFN up to N=1024. At N=256, their errors are 0.00614 and 0.00648, respectively, compared with 0.00604 for the Bayes estimator. Our analysis, albeit in a simplified setting, show how such optimal behavior may be possible, and how it emerges from gradient descent training.
Table R8. Error on our task without fine-tuning. TabPFN and TabICL estimate the threshold as ∫01Pr(Y=1∣x)dx. For our one-head estimator, we report the best saved checkpoint for each N (the final checkpoint gives 0.139 at N=32).
References: