- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper provides an example with in-context learning to compare linear and non-linear attention models and shows that the power behaviour in depth-related AI scaling laws might arise from the softmax non-linearity in the attentions blocks as it learns information at different frequencies (eigenvalues) in parallel, while linear attention can only do it sequentially
Critical strengths
- Mathematically supported claims - Appendix provided clear and understandable proofs
- Nice mechanistic explanation attempt I like the clear toy simulated example to support the claim under controlled settings. If the claim is supported is less clear though (see weaknesses)
- Clear high-level structure - the introduction clearly states a research gap and positions the work among current related literature
Critical weaknesses
- Different data and results presentation for linear and non-linear attention - why do use different fixed values of and for linear and non-linear cases? Also why do you present the results between different cases so differently. I would argue this makes the conclusion impossible, since several things change at the same time. If the claim is that the non-linearity helps to select things - than and which are used to generate the data should stay the same across models. And the plots should be done in the same way - e.g. show loss for both linear and non-linear attention cases and show the residual loss for both, instead of show one for each (Fig 5 & 6) - for me addressing this claim is the main priority
- Clarity: Critical details are sometimes not defined in the main text but appendix only -for example definitions of and are not in the main text, at least semantically should be defined in the captions, otherwise, the Fig 6 d is hard to understand / interpret. Same for the residual loss - its not defined in the caption and vaguely defined in the main text, same for from equation 16 - its not defined in the main text below or after
- Reproducibility: no code or promise of code is provided and optimization hyperparameters are not provided in the text
Questions for authors
- Why was in-context learning task chosen and what could have been the alternatives for it? what would have been pros and cons of the alternatives? why not to look at the training dynamics of the models directly?
- In section 2.2 attention masks are mentioned but nothing in the results depends on the mask, does it? Why was it mentioned?
- Do I get it right that the claim is that in the non-linear attention case for deep networks each network becomes an independent estimator and averaging across layers makes the connection to the central limit theorem? But do we need depth then? Cannot we have layers in parallel then and just average across their predictions and have the same effect?
- What happens with ? Does the effect become stronger or do we need to average across more layers since the task became more hard?
4 comments
Reply to Reviewer 1ux3 (Part 2)
Question 2
We thank the reviewer for this valuable question. We apologize that our explanation was not sufficiently clear and may have caused some confusion.
As in our response to the previous question, our work builds on the prior work of Bordelon et al. (2026) and aims to generalize it. Since that work includes this masking scheme in its model, we retain it here for consistency.
Attention masks do not change the main results and mechanism. Rather, they are in fact necessary for the linear model to learn. It first learns the strong modes and must then subtract this information to expose the remaining weak modes. Without this subtraction, updates targeting the weak modes would interfere with the already-learned strong modes in subsequent layers. Also, the test token needs to grab information from the context. The mask therefore enables both this subtraction and the test token’s access to information from context tokens. Note that for the linear-attention model we keep the mask matrix identical to Bordelon's choice, so that it is well grounded rather than arbitrary.
For the nonlinear-attention model, nonlinearity allows the model to focus directly on the tokens it needs rather than analyzing the intrinsic data structure. It therefore only needs access to the context for the test token to grab information. We use a mask that permits this access and disables context update, which lets us isolate the "selection, ensemble averaging, and parallel learning" mechanism and learn it thoroughly. More details are provided in Appendix C.3.
Additionally, enabling context updates also works for nonlinear models, as shown in Appendix C.3, where we introduce a learnable scalar as the context-to-context mask value to enable context update. This may mix in additional mechanisms, since the model may first adjust the structure of the context information pool to make focusing and grabbing easier. Even in this case, however, we still observe a similar inverse-depth scaling signal on the top-2 task, suggesting that the mechanism studied in the main text is generalizable.
In this paper, we keep these factors separate and clean: disabling context-to-context updates in the nonlinear-attention model allows us to characterize the emergent inverse-depth scaling behavior thoroughly. Incorporating context information updates and understanding the resulting, more complex mechanism is undoubtedly a valuable future direction.
Question 3
We thank the reviewer for raising this valuable question.
Your understanding is correct. However, depth is nonetheless still needed. In our setup, for example, the task resembles the second part of an induction head (copy and compute) (Olsson et al., 2022), whereas the first part—recognizing a previously seen pattern—requires layers and cannot be achieved by these parallel estimators. Related work (e.g., Gurnee et al. (2026)) reports clustering of layers (e.g., Fig. 27). The mechanism introduced in our paper may correspond to one such cluster, while other clusters may serve other functions and computations, which demonstrates the necessity of depth.
Question 4
We thank the reviewer for this question.
Actually, is easier. First, the model bears less burden in learning and distinguishing representations: with = 2, it needs a strong representation to ensure the test token is close to the top-2 tokens and far from the rest, since only the top-2 should be attended to. Second, the target distribution is less sharp to estimate and approaches a near-uniform distribution. In the limit, when equals , the selection ability of softmax becomes irrelevant—the model does not even need softmax to redistribute and focus on the sparse tokens while discarding the rest. Therefore the effect does not become stronger.
Thanks for clear and concise answers. Since my main concerns (W1-W3) were addressed, I have increased the score.
As for the questions, I few a few small follow-ups
Q3 I still do not get why do we need depth then. What does depth give us that the aggregation of individual non-linear estimators could not give us?
Q4 does not your reply contradict to the caption of figure 2, which says "Both tasks model the second half of an induction head: the easy task copies the most relevant token’s label, while the difficult task averages the labels of the top-k relevant tokens." Or am I missing something?
Second Reply to Reviewer 1ux3
We thank the reviewer for raising the score and for the follow-up questions. We clarify the following points.
Q3. Why is depth needed? What does depth provide that an aggregation of individual non-linear estimators cannot?
We give two examples that directly point to the necessity of depth.
- The first part of the induction head. Our work studies the second part of the induction head (copy-and-paste), where the target is clear (e.g., A-B-C-D-A-?, target B). The first part, however, requires "finding the similar pattern": the model must attend not only to the latter "A" but also to the preceding "A-B". This mechanism is more complicated and may require function composition, rather than independent parallel predictors, to estimate the target top-1 distribution in the second part of the induction head.
- A more complex target function. Our work takes a simplified top-k aggregation as the target function, but real settings may involve more complicated functions. For example, the aggregation may be , whereas a more complicated one may be , which involves more complex computation. Finding the two relevant tokens is still essential at the first step and mirrors the mechanism introduced in this paper, but "copy-and-paste" alone cannot estimate this target function. Here we may need to introduce MLPs in the blocks and place them in sequence, so that their sequential composition can perform such nested, more complex function computation, as actual LLMs do. This remains a valuable direction for future studies.
Q4. Does this contradict the caption of Figure 2, which says "Both tasks model the second half of an induction head: the easy task copies the most relevant token's label, while the difficult task averages the labels of the top-k relevant tokens"?
No. The easy task performs top-1 copy-and-paste. It is easy because the attention block can be written as : as long as the relevant token gives the largest logit, the model can simply drive the norms of and to infinity and estimate the target distribution perfectly (focusing only on this token and discarding the others).
For the difficult task with , however, the target distribution can be very sharp and therefore cannot be perfectly estimated by the smooth softmax. The sharper the distribution, the harder the estimation (). When , the target is uniform and a softmax brings no extra benefit (linear attention using only the logits suffices).
Hence our conclusion that for the task is easier than for . The case is easy because the softmax is exactly suited to perfectly estimating this limit, and the task can be solved within one layer.
We hope our response addresses the reviewer's questions.
Reply to Reviewer 1ux3 (Part 1)
We thank the reviewer for the careful reading and professional technical questions. We reply to the questions and concerns point by point below.
Weakness 1
We thank the reviewer for raising this important question. We apologize that our analysis and reporting were not sufficiently rigorous and caused your confusion and concern.
We agree that using consistent fixed values of a and b is important, and we apologize for overlooking this in the original presentation. Following your suggestion, we have updated Figure 3 to show results at fixed a=0 and fixed b=0, consistent with the other experiments. We have moved the original results at fixed a=2 and fixed b=2 to Appendix A.2 as complementary evidence. The proposed mechanism and our main conclusion remain unchanged.
The updated results show that with a = 0 the data contain no strong/weak modes and thus no power-law depth scaling (updated Figure 3b), whereas with a ≠ 0 and b = 0 the spectra-dependent power-law scaling persists (updated Figure 3a). This in fact supports our conclusion more directly, as it provides a clear contrast: for the linear-attention model, no power-law in the data, no power-law in depth scaling; power-law in the data, power-law in depth scaling. In Section 3.1 and Appendix A.4, we further provide detailed analysis showing that each mode is learned with a different characteristic depth (Figure 3c), and that their combination can yield a spectra-dependent power-law depth scaling.
When the characteristic depths of different modes are identical (as in the case a=0), their combination does not yield power-law scaling. Further mathematical details are provided in Appendix A.5. Please refer to the updated PDF.
We also agree that the results should be presented more consistently. We now report the loss for the nonlinear model on the difficult task in Figure 5 and have removed Table 1 as redundant. For the linear model on the difficult task, however, convergence occurs within 2 layers, so the residual loss contains no substantial information or depth scaling. We therefore remain focused on the residual loss and inverse-depth scaling of the nonlinear model in Figure 6.
Weakness 2
We thank the reviewer for this constructive comment. We apologize that our writing was not sufficiently clear and caused difficulties in reading and understanding.
We now define each of these quantities clearly at their first appearance in the main text, and we have also added the corresponding definitions to the figure captions. Please refer to Section 4 and related figures in the updated PDF.
Weakness 3
We thank the reviewer for this valuable comment.
We have already provided the code for the main experiments in the supplementary material and added a subsection documenting reproducibility. The optimization hyperparameters are included in Appendix D.5.
Question 1
We thank the reviewer for this valuable question. We apologize that our explanation of choosing in-context learning was not clear enough.
First, our work builds on Bordelon et al. (2026), who studied linear models for in-context learning and derived results on depth scaling. Our work aims to generalize theirs, and hence we continue to use in-context learning. Moreover, in-context learning has been a widely-adopted choice for the scaling-law literature on in-context learning (Lyu et al., 2025; Lu et al., 2025; Arora et al., 2025; Nguyen and Reddy, 2025).
In-context learning offers two advantages. First, the data covariance provides a natural and widely accepted way to model data structure, in particular strong and weak modes, and in-context learning allows tokens carrying these modes to interact through attention, thereby exposing this structure for the model to exploit. Second, in-context learning in real settings exhibits sparsity—only a few key words are crucial—which matches the setting studied in this paper (e.g., k = 2).
One alternative is next-token prediction during pretraining. This would require other modeling approaches, such as Markov chains, and is considerably more complex. However, the mechanism proposed here can still arise in such settings.
Examining the training dynamics of real models directly may not reveal a clear depth-scaling pattern, because the data structure is not explicitly modeled and key properties such as token interactions and sparsity are absent. We therefore focus on the in-context learning task in this paper.