acceptodds
Under review as a conference paper at ICLR 2027

Emergent Inverse-Depth Scaling From Nonlinearity In Attention

Abstract

Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood. To explain the parameter count scaling, existing theory posits power-law scaling with model depth. In linear-attention models, this scaling is tied to a power-law data spectrum: unable to selectively attend to relevant tokens, these models learn according to global spectral strength, with stronger directions learned before weaker ones. Large language models, however, can be strongly nonlinear. Here, we show that nonlinear attention yields inverse-depth decay of loss across all tested data spectra. Nonlinearity enables attention to focus selectively on relevant tokens, allowing strong and weak spectral directions to be learned in parallel. Similar focusing across layers motivates a connection to the central limit theorem: shared error across layers sets the loss plateau, while aggregation turns layer-specific differences into continued gains with depth. Our findings suggest that depth scaling may arise from nonlinearity in attention, which allows large language models to focus locally and may make the global covariance structure less relevant.

Primary area:
learning theory
Keywords:
scaling laws, depth scaling, nonlinearity, in-context learning, depth utilization

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_1ux3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper provides an example with in-context learning to compare linear and non-linear attention models and shows that the power behaviour in depth-related AI scaling laws might arise from the softmax non-linearity in the attentions blocks as it learns information at different frequencies (eigenvalues) in parallel, while linear attention can only do it sequentially

Critical strengths
  1. Mathematically supported claims - Appendix provided clear and understandable proofs
  2. Nice mechanistic explanation attempt I like the clear toy simulated example to support the claim under controlled settings. If the claim is supported is less clear though (see weaknesses)
  3. Clear high-level structure - the introduction clearly states a research gap and positions the work among current related literature
Critical weaknesses
  1. Different data and results presentation for linear and non-linear attention - why do use different fixed values of aa and bb for linear and non-linear cases? Also why do you present the results between different cases so differently. I would argue this makes the conclusion impossible, since several things change at the same time. If the claim is that the non-linearity helps to select things - than aa and bb which are used to generate the data should stay the same across models. And the plots should be done in the same way - e.g. show loss for both linear and non-linear attention cases and show the residual loss for both, instead of show one for each (Fig 5 & 6) - for me addressing this claim is the main priority
  2. Clarity: Critical details are sometimes not defined in the main text but appendix only -for example definitions of ALA_L and CLC_L are not in the main text, at least semantically ALA_L should be defined in the captions, otherwise, the Fig 6 d is hard to understand / interpret. Same for the residual loss - its not defined in the caption and vaguely defined in the main text, same for L0(a,b)L_0(a,b) from equation 16 - its not defined in the main text below or after
  3. Reproducibility: no code or promise of code is provided and optimization hyperparameters are not provided in the text
Questions for authors
  1. Why was in-context learning task chosen and what could have been the alternatives for it? what would have been pros and cons of the alternatives? why not to look at the training dynamics of the models directly?
  2. In section 2.2 attention masks are mentioned but nothing in the results depends on the mask, does it? Why was it mentioned?
  3. Do I get it right that the claim is that in the non-linear attention case for deep networks each network becomes an independent estimator and averaging across layers makes the connection to the central limit theorem? But do we need depth then? Cannot we have LL layers in parallel then and just average across their predictions and have the same effect?
  4. What happens with k≥2k \geq 2? Does the effect become stronger or do we need to average across more layers since the task became more hard?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer 1ux3 (Part 1)

We thank the reviewer for the careful reading and professional technical questions. We reply to the questions and concerns point by point below.

Weakness 1

We thank the reviewer for raising this important question. We apologize that our analysis and reporting were not sufficiently rigorous and caused your confusion and concern.

We agree that using consistent fixed values of aa and bb is important, and we apologize for overlooking this in the original presentation. Following your suggestion, we have updated Figure 3 to show results at fixed a=0a=0 and fixed b=0b=0, consistent with the other experiments. We have moved the original results at fixed a=2a=2 and fixed b=2b=2 to Appendix A.2 as complementary evidence. The proposed mechanism and our main conclusion remain unchanged.

The updated results show that with aa = 0 the data contain no strong/weak modes and thus no power-law depth scaling (updated Figure 3b), whereas with aa ≠ 0 and bb = 0 the spectra-dependent power-law scaling persists (updated Figure 3a). This in fact supports our conclusion more directly, as it provides a clear contrast: for the linear-attention model, no power-law in the data, no power-law in depth scaling; power-law in the data, power-law in depth scaling. In Section 3.1 and Appendix A.4, we further provide detailed analysis showing that each mode is learned with a different characteristic depth (Figure 3c), and that their combination can yield a spectra-dependent power-law depth scaling.

When the characteristic depths of different modes are identical (as in the case a=0a=0), their combination does not yield power-law scaling. Further mathematical details are provided in Appendix A.5. Please refer to the updated PDF.

We also agree that the results should be presented more consistently. We now report the loss for the nonlinear model on the difficult task in Figure 5 and have removed Table 1 as redundant. For the linear model on the difficult task, however, convergence occurs within 2 layers, so the residual loss contains no substantial information or depth scaling. We therefore remain focused on the residual loss and inverse-depth scaling of the nonlinear model in Figure 6.

Weakness 2

We thank the reviewer for this constructive comment. We apologize that our writing was not sufficiently clear and caused difficulties in reading and understanding.

We now define each of these quantities clearly at their first appearance in the main text, and we have also added the corresponding definitions to the figure captions. Please refer to Section 4 and related figures in the updated PDF.

Weakness 3

We thank the reviewer for this valuable comment.

We have already provided the code for the main experiments in the supplementary material and added a subsection documenting reproducibility. The optimization hyperparameters are included in Appendix D.5.

Question 1

We thank the reviewer for this valuable question. We apologize that our explanation of choosing in-context learning was not clear enough.

First, our work builds on Bordelon et al. (2026), who studied linear models for in-context learning and derived results on depth scaling. Our work aims to generalize theirs, and hence we continue to use in-context learning. Moreover, in-context learning has been a widely-adopted choice for the scaling-law literature on in-context learning (Lyu et al., 2025; Lu et al., 2025; Arora et al., 2025; Nguyen and Reddy, 2025).

In-context learning offers two advantages. First, the data covariance provides a natural and widely accepted way to model data structure, in particular strong and weak modes, and in-context learning allows tokens carrying these modes to interact through attention, thereby exposing this structure for the model to exploit. Second, in-context learning in real settings exhibits sparsity—only a few key words are crucial—which matches the setting studied in this paper (e.g., kk = 2).

One alternative is next-token prediction during pretraining. This would require other modeling approaches, such as Markov chains, and is considerably more complex. However, the mechanism proposed here can still arise in such settings.

Examining the training dynamics of real models directly may not reveal a clear depth-scaling pattern, because the data structure is not explicitly modeled and key properties such as token interactions and sparsity are absent. We therefore focus on the in-context learning task in this paper.

AuthorsOpenReview ↗

Reply to Reviewer 1ux3 (Part 2)

Question 2

We thank the reviewer for this valuable question. We apologize that our explanation was not sufficiently clear and may have caused some confusion.

As in our response to the previous question, our work builds on the prior work of Bordelon et al. (2026) and aims to generalize it. Since that work includes this masking scheme in its model, we retain it here for consistency.

Attention masks do not change the main results and mechanism. Rather, they are in fact necessary for the linear model to learn. It first learns the strong modes and must then subtract this information to expose the remaining weak modes. Without this subtraction, updates targeting the weak modes would interfere with the already-learned strong modes in subsequent layers. Also, the test token needs to grab information from the context. The mask therefore enables both this subtraction and the test token’s access to information from context tokens. Note that for the linear-attention model we keep the mask matrix identical to Bordelon's choice, so that it is well grounded rather than arbitrary.

For the nonlinear-attention model, nonlinearity allows the model to focus directly on the tokens it needs rather than analyzing the intrinsic data structure. It therefore only needs access to the context for the test token to grab information. We use a mask that permits this access and disables context update, which lets us isolate the "selection, ensemble averaging, and parallel learning" mechanism and learn it thoroughly. More details are provided in Appendix C.3.

Additionally, enabling context updates also works for nonlinear models, as shown in Appendix C.3, where we introduce a learnable scalar mm as the context-to-context mask value to enable context update. This may mix in additional mechanisms, since the model may first adjust the structure of the context information pool to make focusing and grabbing easier. Even in this case, however, we still observe a similar inverse-depth scaling signal on the top-2 task, suggesting that the mechanism studied in the main text is generalizable.

In this paper, we keep these factors separate and clean: disabling context-to-context updates in the nonlinear-attention model allows us to characterize the emergent inverse-depth scaling behavior thoroughly. Incorporating context information updates and understanding the resulting, more complex mechanism is undoubtedly a valuable future direction.

Question 3

We thank the reviewer for raising this valuable question.

Your understanding is correct. However, depth is nonetheless still needed. In our setup, for example, the task resembles the second part of an induction head (copy and compute) (Olsson et al., 2022), whereas the first part—recognizing a previously seen pattern—requires layers and cannot be achieved by these parallel estimators. Related work (e.g., Gurnee et al. (2026)) reports clustering of layers (e.g., Fig. 27). The mechanism introduced in our paper may correspond to one such cluster, while other clusters may serve other functions and computations, which demonstrates the necessity of depth.

Question 4

We thank the reviewer for this question.

Actually, k>2k>2 is easier. First, the model bears less burden in learning and distinguishing representations: with kk = 2, it needs a strong representation to ensure the test token is close to the top-2 tokens and far from the rest, since only the top-2 should be attended to. Second, the target distribution is less sharp to estimate and approaches a near-uniform distribution. In the limit, when kk equals PP, the selection ability of softmax becomes irrelevant—the model does not even need softmax to redistribute and focus on the sparse tokens while discarding the rest. Therefore the effect does not become stronger.

Thanks for clear and concise answers. Since my main concerns (W1-W3) were addressed, I have increased the score.

As for the questions, I few a few small follow-ups

Q3 I still do not get why do we need depth then. What does depth give us that the aggregation of individual non-linear estimators could not give us?

Q4 does not your reply contradict to the caption of figure 2, which says "Both tasks model the second half of an induction head: the easy task copies the most relevant token’s label, while the difficult task averages the labels of the top-k relevant tokens." Or am I missing something?

AuthorsOpenReview ↗

Second Reply to Reviewer 1ux3

We thank the reviewer for raising the score and for the follow-up questions. We clarify the following points.

Q3. Why is depth needed? What does depth provide that an aggregation of individual non-linear estimators cannot?

We give two examples that directly point to the necessity of depth.

  1. The first part of the induction head. Our work studies the second part of the induction head (copy-and-paste), where the target is clear (e.g., A-B-C-D-A-?, target B). The first part, however, requires "finding the similar pattern": the model must attend not only to the latter "A" but also to the preceding "A-B". This mechanism is more complicated and may require function composition, rather than independent parallel predictors, to estimate the target top-1 distribution in the second part of the induction head.
  2. A more complex target function. Our work takes a simplified top-k aggregation as the target function, but real settings may involve more complicated functions. For example, the aggregation may be y1+y22\frac{y_1+y_2}{2}, whereas a more complicated one may be f(g(y1,y2))f(g(y_1,y_2)), which involves more complex computation. Finding the two relevant tokens is still essential at the first step and mirrors the mechanism introduced in this paper, but "copy-and-paste" alone cannot estimate this target function. Here we may need to introduce MLPs in the blocks and place them in sequence, so that their sequential composition can perform such nested, more complex function computation, as actual LLMs do. This remains a valuable direction for future studies.

Q4. Does this contradict the caption of Figure 2, which says "Both tasks model the second half of an induction head: the easy task copies the most relevant token's label, while the difficult task averages the labels of the top-k relevant tokens"?

No. The easy task performs top-1 copy-and-paste. It is easy because the attention block can be written as softmax((Wkx)T(Wqx))\mathrm{softmax}((W_kx)^T(W_qx)): as long as the relevant token gives the largest logit, the model can simply drive the norms of WkW_k and WqW_q to infinity and estimate the target distribution perfectly (focusing only on this token and discarding the others).

For the difficult task with 2≤k≤P2\leq k \leq P, however, the target distribution can be very sharp and therefore cannot be perfectly estimated by the smooth softmax. The sharper the distribution, the harder the estimation (k=2k=2). When k=Pk=P, the target is uniform and a softmax brings no extra benefit (linear attention using only the logits suffices).

Hence our conclusion that for k>2k>2 the task is easier than for k=2k=2. The case k=1k=1 is easy because the softmax is exactly suited to perfectly estimating this limit, and the task can be solved within one layer.

We hope our response addresses the reviewer's questions.

Reviewer_4iYfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In a controlled in-context regression setting, the authors compare linear and softmax attention across data spectra. Linear attention shows spectrum-dependent depth scaling on an easy task and plateaus on a top-k task. Softmax attention shows inverse-depth loss decay, which the authors explain by layer averaging and connect to the central limit theorem.

Critical strengths
  • Isolating the attention nonlinearity as the experimental variable is a clean design.
  • The loss decomposition into shared error and disagreement is simple and interpretable.
  • The spectral analysis paints a clear picture of mechanism in the networks.
Critical weaknesses
  • Softmax attention outperforming a model that cannot select tokens, on a selection task, is expected. The order-of-magnitude difference in loss floors mostly reflects a capacity gap.
  • The linear and nonlinear models also differ in their masking and context updates, which confounds the comparison.
  • Neither error bars nor the number of runs are reported, so the robustness of the fitted exponents cannot be assessed.
  • The claim that nonlinear attention learns strong and weak spectral directions in parallel, whereas linear attention learns them sequentially, may be premature. The paper’s own results show linear attention learning weak modes given sufficient depth. The sequential ordering may therefore stem from the data representation, the masking scheme or the dataset rather than from linearity itself. The update equation during training would allow a linear model to learn modes in parallel. The supported conclusion is that the nonlinearity makes learning weak directions easier, not that it enables it.
Questions for authors
  • How dataset-dependent do the authors expect these results to be? If the ability to learn different spectral modes is a major contributor to the improved performance, would it not follow that the depth scaling is itself data-dependent, contrary to the suggestion that nonlinear attention makes the global covariance structure less relevant? The tested spectra seem too narrow to settle this.
  • What concrete, non-obvious prediction does the mechanism make for standard Transformers, and can it be tested?
  • Can the analysis be applied to other areas where attention is used, e.g. graph neural networks? Nonlinear attention is very common there (e.g. graph attention networks), but attention is restricted to local neighbourhoods and deep stacks suffer from oversmoothing. Would the layer-ensemble picture and inverse-depth scaling still be expected?
4 comments
AuthorsOpenReview ↗

Reply to reviewer 4iYf (Part 1)

We thank the reviewer for the careful reading and professional technical questions. We reply to the questions and concerns point by point below.

Weakness 1

We thank the reviewer for this comment. We apologize that our explanation was not sufficiently clear and may have caused some confusion.

We agree that this result is expected. However, the reported irreducible-loss comparison acts only as a complementary reference, not as a main claim. The emphasis of our work is the depth scaling behavior and the underlying learning mechanisms of the linear-attention and nonlinear-attention models on the in-context learning task.

Weakness 2

We thank the reviewer for this comment. We apologize that we had not fully clarified the role of masking and context updates and may have caused some misunderstanding.

First, our setup follows the prior work of Bordelon et al. (2026), which we aim to generalize. Since that work employs this masking scheme, we adopt it as well for consistency.

For the linear-attention model, the attention masks used here are essential to learning. It first captures the strong modes and must then subtract them to reveal the remaining weak modes. Without this subtraction, updates aimed at the weak modes would overwrite the strong modes already learned in earlier layers. The mask enables both the necessary subtraction and the test token’s access to information from context tokens. We keep the mask matrix exactly as in Bordelon's work, so that the choice is principled rather than arbitrary.

By contrast, nonlinearity enables the model to attend directly to the tokens it needs without analyzing the intrinsic data structure. The nonlinear-attention model therefore needs only context access for the test token to gather information. Therefore, the mask used here permits this access and disables context update, thereby isolating the "selection, ensemble averaging, and parallel learning" mechanism. More details are provided in Appendix C.3.

Context updates can also be enabled for nonlinear models (Appendix C.3), where we use a learnable scalar mm as the context-to-context mask value. This may entangle additional mechanisms, as the model can first restructure the context information pool to facilitate focusing and gathering, as analyzed in this section. Even so, we observe a similar inverse-depth scaling signal on the top-2 task, indicating that the mechanism studied here is generalizable.

In this paper, we deliberately keep these factors separate: disabling context-to-context updates in the nonlinear-attention model lets us characterize the emergent inverse-depth scaling behavior in full. Extending the analysis to context information updates is undoubtedly a valuable direction for future work.

Weakness 3

We thank the reviewer for raising this issue. We apologize that our reporting and writing were not sufficiently rigorous and may have caused some concern.

We have added this information to Appendix D.1. For each data spectrum, the experiment was repeated with three different random seeds, and the reported standard errors account for this variability across different runs in addition to the fit residual.

Weakness 4

We thank the reviewer for this comment. We apologize that our explanation was not sufficiently clear and may have caused some misunderstanding.

By "sequentially," we do not mean that the model needs to fully learn the strong modes without even learning anything from the weak modes. Rather, we mean that in the linear model, the shallow layers mainly learn the strong modes, whereas the weak modes are better learned in deeper layers. In the nonlinear model, by contrast, direct focusing on and grabbing information from relevant tokens allows the strong and weak modes to be learned well in parallel. "Parallel learning" emphasizes the ability to learn weak modes as well as strong modes within same number of layers, compared with sequential learning.

Question 1

We thank the reviewer for this valuable question. We apologize that our explanation was not sufficiently clear and may have caused some confusion.

We study the second part of the induction head (copy and compute) (Olsson et al., 2022) in this paper. Regardless of the spectral structure of the data, this task can be solved by directly focusing on the relevant tokens. The nonlinear model therefore does not need to decompose the internal data structure or recover the covariance. Consequently, it is expected that the inverse-depth scaling of the nonlinear model does not depend on the data covariance.

AuthorsOpenReview ↗

Reply to Reviewer 4iYF (Part 2)

Question 2

We thank the reviewer for this valuable question.

Ensemble averaging and clustering in deep models have been widely observed, for example in Gurnee et al. (2026) and Liu et al. (2026). Based on our work, we conjecture that in standard Transformers the mechanism studied here likely corresponds to one of these clusters: a contiguous block of intermediate layers dedicated to focusing on the relevant tokens. We further conjecture that the remaining clusters play other roles—shallower clusters may preprocess the information pool to facilitate focusing in the intermediate layers, while deeper clusters may require few further updates after focusing and thus give rise to attention sinks (Xiao et al., 2024).

We also predict that in looped Transformers, certain intermediate loops implement the same mechanism. These are all valuable and testable directions, and we are currently investigating them.

Question 3

We thank the reviewer for this valuable question. We first apologize that we are not very familiar with the GNN literature and can therefore offer only a brief intuition.

Our mechanism relies on layers disagreeing in a way that averaging can cancel. Oversmoothing pushes node representations toward a common limit, which shrinks that disagreement. In the oversmoothed regime, we would therefore not expect inverse-depth scaling, and additional depth could even be harmful. In shallower GATs, or in architectures designed to mitigate oversmoothing, the signal might plausibly appear.

Reviewer_4iYfOpenReview ↗

Response to authors.

I appreciate the detailed response of the authors. In saying that, I don't find the additional information to be particularly clarifying given the initial questions. Further, statements like: """ Consequently, it is expected that the inverse-depth scaling of the nonlinear model does not depend on the data covariance.""" likely need some followup with experiments or theory. As that would seem to be like a big claim and core to the thesis of the work. I don't see evidence in the responses that this has been clarified. I will not raise my score.

AuthorsOpenReview ↗

Second Reply to Reviewer 4iYF

We thank the reviewer for the response. We believe there's still some misunderstandings. Regarding the statement "Consequently, it is expected that the inverse-depth scaling of the nonlinear model does not depend on the data covariance," we have clearly stated in the paper that the ability to learn different spectral modes arises from a fundamentally different mechanism. The linear model cannot achieve this because it is unable to focus solely on the relevant tokens while discarding the irrelevant ones. As a result, it can only capture the strong modes first and then the weak ones. The nonlinear model, by contrast, does not need to follow this path—it can directly attend to the relevant tokens and perform a "copy-and-compute" operation. This operation has nothing to do with the data spectrum and depends only on the tokens themselves, which leads to the consequent "parallel learning". Experiments in our paper have clarified this contrast. For example, in the easy task, figure 4c has shown that the model has put all the attention to the relevant token, which is totally different with what we've shown in figure 3 and section 3.1 regarding the linear model. We therefore conclude that the inverse-depth scaling does not depend on the data covariance. Is it clear right now?

Reviewer_fpSsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the mechanisms underlying depth scaling in Transformer models, focusing on whether nonlinear attention can explain the emergence of inverse-depth loss scaling. The authors compare simplified in-context learning (ICL) models with linear and softmax-based nonlinear attention under different input and teacher covariance spectra.

The paper considers two retrieval-and-aggregation tasks. In the easier task, the model retrieves the context token most similar to the query and copies its label. In the more challenging task, it averages the labels of the top-(k) most relevant context tokens. The authors show that linear attention exhibits spectrum-dependent depth scaling and can quickly saturate, whereas nonlinear attention exhibits approximately inverse-depth loss scaling across the tested spectral settings.

The paper also investigates extensions that relax some simplifying assumptions, including shared input and label spaces, shared value matrices, and fixed context representations. These experiments provide additional evidence that inverse-depth-like behavior can persist beyond the simplest model.

Critical strengths
  1. Understanding the mechanisms underlying neural scaling laws is an important problem. The paper moves beyond merely observing scaling behavior and attempts to identify an architectural mechanism that could explain it.
  2. The decomposition R_L=C_L+(A_L-C_L)/L provides a useful framework for separating shared prediction error from layer-specific disagreement. It helps clarify why aggregation can improve performance even when the individual layers retain a common error component.
  3. Independently varying the input and teacher covariance spectra provides a useful way to investigate the interaction between data structure and depth scaling.
Critical weaknesses
  1. The identity R_L=C_L+(A_L-C_L)/L is informative, but an inverse-depth law requires additional assumptions about the behavior of C_L and A_L-C_L as depth increases. In particular, the decomposition alone does not establish that the shared-error term converges to a constant or that the disagreement term does so at the relevant rate.
  2. The top-2 experiments provide strong evidence for exponents near one. However, some top-16 experiments yield exponents around (0.79)–(0.88), and the top-16 task with context updates exhibits rapid saturation in the reported experiment.
  3. The scaling exponents are obtained by fitting a power law with a loss floor over a finite range of depths. A high (R^2) establishes that the proposed functional form fits the observed points well, but does not by itself distinguish it from alternative scaling forms or establish asymptotic behavior.
Questions for authors
  1. Could the authors compare models that are identical in architecture, masking, normalization, residual updates, and training procedure, differing only in the attention-score transformation? For example, would a controlled interpolation between raw attention scores and softmax-normalized scores exhibit a systematic transition in the fitted depth-scaling exponent?
  2. Under what sufficient conditions on the layerwise prediction errors does the loss decomposition imply R_L=R_\infty+\Theta(1/L)? Can the authors derive or justify the convergence of the shared-error and disagreement terms, rather than relying primarily on their empirical behavior?
  3. Could the authors compare models that are identical in architecture, masking, normalization, residual updates, and training procedure, differing only in the attention-score transformation? For example, would a controlled interpolation between raw attention scores and softmax-normalized scores exhibit a systematic transition in the fitted depth-scaling exponent?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer fpSs (Part 1)

We thank the reviewer for the careful reading and professional technical questions. We reply to the questions and concerns point by point below.

Weakness 1

We thank the reviewer for this valuable question. We apologize for the misunderstanding we may have caused.

We do not claim that decomposing the loss into these two terms by itself proves the inverse-depth scaling. Our claim rests on rigorously fitting the loss plateau and the reducible term with a power-law function, not on this decomposition. The decomposition serves only for interpretation and for the connection to the central limit theorem. The convergence of CLC_L and AL−CLA_L-C_L can only be observed empirically rather than derived theoretically.

Weakness 2

We thank the reviewer for this point. We believe there may be some misunderstandings, and we’d like to clarify and address the two points in turn.

For the top-16 exponents in the 0.79–0.88 range, we in fact observe the same qualitative behavior: CLC_L and AL−CLA_L-C_L converge gradually, and the loss approaches its plateau at a rate close to L−1L^{-1}. The deviation from one arises because the top-16 task requires greater depth to reach the perfect inverse-depth regime, and we are limited by the computational cost of training deeper networks. As we discuss in the paper in connection with the central limit theorem, a convergence rate of L−1L^{-1} is itself a deep-depth limiting behavior. Observing a closely similar signal to inverse scaling at finite depth without reaching exactly one is therefore expected and reasonable.

For the rapid saturation of top-16 with context updates, the extended analysis in Appendix C.3 is designed to compare a sharp/sparse target (top-2) with a flatter target (top-16). Our conclusion there is that for the sharp target, context updates and the resulting optimization of the information pool bring no substantial gain, whereas for the flat target, context updates do help. This does not mean that the ensemble-averaging mechanism is absent in top-16. Rather, the task is easier and the context updates are more effective for the flat distribution. The ensemble-averaging mechanism is thereby masked.

Weakness 3

We thank the reviewer for this point and we'd like to clarify a few things to resolve the concern.

We apologize that we did not describe the fitting details clearly enough. Our fitting range already extends to models with L=2048L=2048 layers (Appendix D.1), at which depth the model lies extremely close to its loss plateau and the loss has essentially converged. The fit therefore covers a regime that closely approximates the asymptotically deep limit rather than only the shallow region.

As derived in Appendix B, the nonlinear-attention model admits an equivalent parallel structure: each layer acts as an independent estimator, and the final output is their average. Under this interpretation, inverse-depth scaling is the most natural hypothesis, and power-law fitting is therefore appropriate for this structure and consistent with the broader scaling-law literature (Bordelon et al., 2026; Kaplan et al., 2020; Hoffmann et al., 2022; Liu et al., 2026).

AuthorsOpenReview ↗

Reply to Reviewer fpSs (Part 2)

Question 1

We thank the reviewer for this valuable question, and we apologize for the misunderstandings we may have caused. We would like to clarify our design choices as we believe all of them are reasonable and necessary.

Our central claim for this question is that the linear and nonlinear models rely on fundamentally different mechanisms and therefore require different configurations. There is no continuous transition between them: interpolating would either cause training failure or mix multiple mechanisms, obscuring rather than clarifying the comparison.

Our work builds on the work of Bordelon et al. (2026), which considers the linear-attention model and studies its scaling laws. For consistency, we adopt their linear-attention model, which contains no input normalization. For the nonlinear model, we retain the pre-layer RMSNorm for training stability and consistency with real language models.

Our attention-mask design likewise follows the linear-attention setup of Bordelon et al. (2026). For that model, the mask is essential for learning: after capturing the strong modes, the model must subtract their contribution to expose the weaker ones. Without this step, later updates directed at weak modes would interfere with the strong modes already learned. The mask enables both the necessary subtraction and the test token's access to information from context tokens. Note that we use exactly the mask matrix of Bordelon et al., making this choice grounded rather than arbitrary.

The nonlinear-attention model operates differently: nonlinearity lets it select relevant tokens directly, without needing to analyze the data's intrinsic structure. Therefore, only the test token needs access to the context. Hence we disable the context update and retain the other mask entries from the linear model. This configuration allows us to examine selection, ensemble averaging, and parallel learning on their own. More details are provided in Appendix C.3.

We also consider nonlinear attention with context updates in Appendix C.3, using a learnable scalar mm for the context-to-context mask. In that setting, the model may reorganize the context information pool before focusing and gathering, introducing additional mechanisms. Nevertheless, the top-2 task still shows a similar inverse-depth scaling signal, supporting the generalizability of the mechanism analyzed in the main text.

For the main analysis, we disable context-to-context updates in the nonlinear model so that the emergent inverse-depth scaling behavior can be characterized clearly. Studying how context updates interact with this mechanism is undoubtedly a valuable direction for future work.

The training procedure and residual update are identical across tasks and models. Details of the optimization hyperparameters are included in Appendix D.5.

We therefore isolate each factor rather than mixing them, and we do not expect a systematic phase transition from interpolation, since the two mechanisms are fundamentally different.

Question 2

As in our answer to Weakness 1, this decomposition cannot directly guarantee inverse-depth scaling. We do not aim to prove inverse-depth scaling directly through theory. However, we can establish an experiment-based sufficient condition for RL=R∞+O(1/L)R_L=R_\infty+O(1/L): If both CLC_L and AL−CLA_L-C_L increase and converge, and the convergence rate of CLC_L is at least as fast as 1/L1/L, then this is sufficient to yield RL=R∞+O(1/L)R_L=R_\infty+O(1/L). In Appendix D.4, we provide an analysis of the convergence rate of CLC_L and show that it converges at a rate of approximately 1/L1/L, which is consistent with our fitting and inverse-depth scaling conclusion.

We noticed that Question 3 in your original review is identical to Question 1. Could you please clarify whether Question 3 was repeated in error or whether you intended to ask a different question?

AuthorsOpenReview ↗

Reviewer-Author Discussion Period Closing

Dear Reviewer fpSs,

Thank you very much for your time and effort in reviewing.

With the discussion closing in less than 24 hours (Nov. 18 AOE), we want to follow up on our rebuttal. Please let us know whether any of your concerns or questions have been addressed and whether you have follow-up questions.

We’d be grateful if you would revisit your assessment in light of the rebuttal. Deeply appreciated!

Best,

The authors

Reviewer_p82XOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how attention nonlinearity affects depth scaling in synthetic in-context learning tasks with controlled input and teacher spectra. It contrasts an easy regression task, motivated by an idealized exact match, with a harder task whose target averages labels of the top-k context inputs by cosine similarity. Linear attention exhibits spectrum-dependent improvement on the easy task and quickly saturates on the harder task. In the main nonlinear model, loss above a fitted plateau decreases approximately inversely with depth across the tested spectra, with experiments extending to 2048 layers. Mode-wise losses show similar relative improvement across strong and weak spectral directions. The paper relates these observations to selective retrieval and aggregation across layers through an exact loss decomposition, and examines several architectural extensions and their limits.

Critical strengths

The controlled experiments give an informative comparison between spectral learning and selective retrieval. Figure 6 shows continued improvement with nonlinear attention where the linear model has already saturated, and Figure 7 helps explain the difference by examining individual spectral directions. The additional spectra and targets in Appendix D.3 broaden the evidence. These results provide a useful empirical contribution beyond the general observation that averaging predictors can reduce error.

The analysis makes the observations easier to interpret. The linear-model loss-floor predictions agree closely with the numerical estimates in Table 3, and the exact decomposition in Appendix B connects the nonlinear model's loss to shared errors and differences across layers without assuming independent layer errors. The architectural extensions also reveal a meaningful boundary: allowing context updates preserves the inverse-depth trend on the top-2 task but leads to saturation on the top-16 task. This makes the limits of the proposed explanation concrete.

Critical weaknesses

The finite implementation of the easy task is not sufficiently explicit in the paper. Equation 4 describes i.i.d. Gaussian inputs and queries, whereas Equation 5 introduces an idealized exact match motivated by the infinite-context limit. It is unclear how the actual experiments, particularly Figure 4, implement this relationship. This matters for interpreting the near-zero single-layer loss and the comparison with linear attention. A precise data-generation description and a pointer to the relevant configuration would resolve the concern.

The interpretation of the fitted exponent would benefit from a clearer account of its uncertainty. Appendix D.1 gives the fitting objective, and Appendix D.4 provides a useful consistency check on the shared-error term. However, the paper does not make clear which depth points enter each fit, what the reported standard errors include, or how the exponent changes with the fitted plateau and depth range. Since the headline result concerns loss above a fitted plateau, these details matter for judging how strongly the finite-depth evidence supports an inverse-depth law.

Questions for authors

How are the query and target generated in the finite easy-task experiments, particularly Figure 4, and do the linear and nonlinear models use the same construction? Please point to the corresponding data-generation code or configuration, including the context size and input normalization, to clarify how the experiment relates to the i.i.d. formulation and idealized exact match.

Which depth points and fitting ranges are used for the main loss curves, and how sensitive is the estimated exponent to the plateau and depth window? Appendix D.4 is helpful, but it inherits the plateau from the direct loss fit and therefore does not independently determine it.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer p82X

We thank the reviewer for the careful reading and professional technical questions. We reply to the questions and concerns point by point below.

Weakness 1

We thank the reviewer for this constructive comment. We apologize that our explanation was not sufficiently clear and may have caused some confusion.

Unless otherwise specified, all our experiments use top-1 exact match for the easy task and top-kk aggregation for the difficult task, which keeps the task construction symmetric and simple. We have directly pointed this out in Appendix E in our updated paper.

For the linear-attention model on the easy task, our work builds on Bordelon et al. (2026), so we also run experiments under their setup. Specifically, in their work, inputs and queries are drawn i.i.d. from a Gaussian distribution as in Equation 4, and the context length PP is taken to infinity. In practice, we set P=1024P=1024 to approximate this limit. The results are nearly identical to those of the top-1 exact match and do not affect our conclusion. This comparison is provided in Appendix E.

Weakness 2

We thank the reviewer for this constructive point.

For the main loss curves, the fit is performed on the depths L∈[128,2048]L\in[128,2048] for every data spectrum, which not only includes the finite depth regime but also reaches the deep regime to model the asymptotic behavior.

Regarding sensitivity to the plateau and the depth window: once the model is sufficiently deep (e.g. 2048 layers), the loss has essentially converged to its plateau, so the plateau L0L_0 is estimated very precisely during the fitting and, under this condition, the fitted exponent is stable. To demonstrate this quantitatively, we have added a stability analysis to Appendix D.4.

On the standard errors: For each data spectrum, the same experiment was repeated with three different random seeds. The reported standard errors account for both the fit residual and variability across different runs. We have clarified this in Appendix D.1.

Question 1

We thank the reviewer for this valuable question.

For the easy task construction, please refer to the answer to (Weakness 1). Concerning input normalization, for the linear-attention model we follow Bordelon et al. (2026) exactly and apply no normalization, keeping the model purely linear. For the nonlinear-attention model, we apply pre-layer RMSNorm, a standard choice in modern LLMs that improves training stability. This slight difference reflects our aim to stay consistent with both prior work and real-world models and does not affect the mechanism proposed in the paper.

Question 2

We thank the reviewer for this question. Please refer to our answer to (Weakness 2).

AuthorsOpenReview ↗

Reviewer-Author Discussion Period Closing

Dear Reviewer p82X,

Thank you very much for your time and effort in reviewing.

With the discussion closing in less than 24 hours (Nov. 18 AOE), we want to follow up on our rebuttal. Please let us know whether any of your concerns or questions have been addressed and whether you have follow-up questions.

We’d be grateful if you would revisit your assessment in light of the rebuttal. Deeply appreciated!

Best,

The authors

Thank you for the detailed response and for clarifying the task construction, fitting procedure, and uncertainty estimates. These clarifications address my main questions about the experimental setup and reporting. While helpful, they do not substantially change my overall assessment of the paper’s contribution, so I will maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Comment

We thank all reviewers for their careful reading and constructive questions and suggestions. We have addressed each point in our individual responses and made corresponding revisions to the manuscript. We hope our responses resolve the reviewers’ concerns.

To make the significance of our work clearer, we have also expanded the Discussion (Section 6). It now summarizes our main findings and limitations more explicitly and outlines several directions for future research. We hope these additions are helpful, and we thank the Area Chair and reviewers for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.