acceptodds
Under review as a conference paper at ICLR 2027

The Second Layer of Linear Self-Attention Implements a Context-Adaptive Update

Abstract

Recent work interprets transformer depth as iterations of an optimisation algorithm. We demonstrate that context-adaptive computation first occurs at layer two in linear self-attention. We prove that one-layer models cannot implement context-adaptive updates, that two-layer models can, and that, for every finite depth, there are task distributions where context-adaptive updates achieve strictly lower risk than any composition of the same number of fixed optimisation steps. These results identify the first layer where linear self-attention can condition its update rule on the observed context rather than only the training distribution. Depth therefore enables more than additional iterations of a fixed optimisation algorithm. We introduce a framework for measuring update behaviour in trained transformers by comparison with reference optimisation algorithms. The framework is validated on models with known mechanisms and applied across a range of transformer architectures and in-context learning tasks. One-layer models match the best single fixed optimisation step. Two-layer models show a context-adaptive correction, which disappears when the second layer is ablated. Our framework identifies the known two-layer induction transition from the update kernel in associative recall and extends to in-context classification and softmax attention.

Primary area:
optimization
Keywords:
transformers, in-context learning, linear attention, mechanistic interpretability, optimisation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uxJhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies when linear self-attention can do more than behave like a sequence of fixed preconditioned gradient-descent steps. The main theoretical result is that, for arbitrary one-layer LSA parameters, the effective bilinear preconditioner is fixed across contexts, while unrestricted two-layer LSA can make the effective preconditioner depend on the empirical covariance. The two-layer model can then represent a richer class of context-dependent updates. This is better than just conducting two fixed steps on the distribution. The paper then introduces an update-kernel-based diagnostic to measure this behavior in trained models and tests it across LSA, softmax attention, classification, and associative recall.

Critical strengths

The best part of this paper is the formalization of the cross-layer effect. The theory is quite clean and the empirical tests are well matched to the theorems. I think this way of presenting the problem is nice and the intuition behind it can be possibly used in future theory to explore how transformers behave.

The update-kernel analysis is quite nice. It's a very mechanistic way to ask how much a trained model departs from its fixed bilinear update. The ablations are appropriate and fairly convincing mostly. Bypassing or randomizing the second layer does seem to remove the measured effect while applied to LSA.

Critical weaknesses

(1) I think the biggest problem is how the novelty with respect to prior work is framed. The main idea that multiple linear transformer layers can interact to implement something richer than standard preconditioned gradient descent is already present in the literature. Ahn et al., “Transformers Learn to Implement Preconditioned Gradient Descent for In-Context Learning,” already study “multi-layer transformers beyond standard optimization methods” and analyze a more general parametrization. Vladymyrov et al., “Linear Transformers are Versatile In-Context Learners,” directly establish that cross-layer interaction can make the optimization rule adapt to the current context. The novelty of this paper seems to be more specific regarding the exact separation. This paper does briefly mention the Vladymyrov paper, but in a very limited capacity.

So if I am not mistaken, this paper does not really establish for the first time that multiple layers provide richer context-dependent optimization. Rather, it shows that the unrestricted two-layer LSA is strictly richer than the corresponding class of fixed preconditioned steps, while one layer cannot realize the same covariance-dependent preconditioning. The problem is that this exact effect is not what is shown for softmax. We do not see the same sharp separation, but rather a gap becoming positive at layer two. So the exact novel result does not transfer, while the broader qualitative intuition does. However, as mentioned above, the broader intuition seems to be the part that is less novel.

(2) The second major problem is that there is limited analysis of why this specific result actually matters. The paper basically provides a clean separation in unnormalized LSA, establishes it carefully, and has a brief discussion of how related behavior partially transfers to other settings. However, it is not really clear what this tells us about standard transformers, what broader mechanistic conclusion should follow from it, or how this should motivate future work or architecture design.

I am left unsure whether this is an important contribution to our understanding of transformers more generally, or mainly the correct statement of a specific mathematical property of unnormalized LSA. This is especially important because the broader intuition that deeper layers can implement richer context-dependent optimization is already present in prior work.

Questions for authors

(1) Could the authors clarify more precisely what they view as the main novelty relative to Ahn et al. and Vladymyrov et al.? My current understanding is that the new contribution is specifically the minimal-depth/strict-separation result rather than the broader idea of richer cross-layer optimization.

(2) Which part of the theoretical result do the authors expect to carry over beyond unnormalized LSA? In particular, how should the softmax results be interpreted relative to the sharp LSA separation?

(3) What broader implication do the authors see coming from this result? In particular, what does the separation tell us about understanding or designing more realistic transformers?

3 comments

Thank you for your review and summary of the paper. We answer your questions in order.

  1. Your understanding is correct. The new contribution is the minimal depth and the strict separation.
    • Ahn et al. (Section 5, Theorem 4) study algorithms such as GD++ that transform the inputs with the covariance of the context. They show that these algorithms are near-stationary points of the multi-layer training objective.
    • Vladymyrov et al. show that linear transformers trained with varying noise levels learn momentum and noise-dependent rescaling. You are correct that the submission mentioned this paper only briefly. The revised Related Work section describes it in more detail (p. 2).
    • These results do not establish the smallest depth at which a model can compute an update that the same number of fixed preconditioned steps cannot produce. They also do not compare a model with every sequence of the same number of fixed preconditioned steps. A fixed preconditioned step is θ←θ+M(b−Σ^θ)\theta\leftarrow\theta+M(b-\widehat\Sigma\theta). Here Σ^=X⊤X/n\widehat\Sigma=X^\top X/n is the observed covariance of the context inputs and b=X⊤y/nb=X^\top y/n. The matrix MM is the same for every context.
    • We add four results:
    1. A one-layer LSA cannot implement a context-adaptive update for any parameters (Theorem 1).
    2. A two-layer LSA can have a strictly lower risk than every pair of fixed preconditioned steps (Theorem 2). Results 1 and 2 make depth two the smallest depth at which an LSA can implement a context-adaptive update.
    3. For every ll, there are task distributions where ll fixed steps have a positive excess risk, and one context-adaptive step does not (Corollary 3).
    4. Trained two-layer models have a lower held-out error than two jointly optimised fixed steps in all 20 seeds (4.08 against 5.14, Table 2).
    • Result 4 is new in the revised paper. Table 2 also reports a second test where we fit the update kernel ∂y^/∂yi\partial\hat y/\partial y_i of each trained two-layer model with a class that contains the kernel of every pair of fixed steps. The fit leaves 6.1% of the kernel variance unexplained. For two jointly optimised fixed steps, it leaves 0.0%.
  2. We expect the advantage of context-adaptive updates over fixed optimisation steps to carry over. Corollary 3 does not depend on the architecture. However, we do not expect the depth-two threshold to carry over, as it is specific to linear attention (Theorem 1).
    • The revised submission adds an experiment showing that in standard softmax transformers, a single layer already outperforms two fixed optimisation steps (Section 8, p. 9, and Table 3, p. 20), hence agreeing with the two points above.
    • As a result, softmax does not show a sharp change at depth two. This is because a single linear-attention layer cannot implement a context-adaptive update (Theorem 1), whereas a single softmax layer can compute more than a fixed step.
    • This is empirical evidence, however, and as this also introduces feed-foward layers and normalisation, we cannot say which component (or components) are responsible (Section 8, p. 9). The softmax models have one training run per depth (Limitations, p. 10).
    • We removed our claim that linear attention changes more sharply than softmax in the revised paper.
  3. Section 9 now has an “Implications” paragraph (p. 10) explaining how describing each layer as one fixed gradient step undercounts what the layers compute.
    • For LSA, the ‘one layer maps to one optimisation step’ interpretation fails at depth two (Table 2), for softmax, it fails at depth one (Table 3).
    • The Theory section only explains the LSA results.
    • This comparison suggests that depth matters more than size. The two-layer LSA models have 1,156 parameters and the one-layer softmax model has 200,833. The LSA models still predict the query label with a lower mean squared error (4.08 against 4.39, Table 3).
    • We do not make generalised model design recommendations from these results.

I appreciate the more precise characterization of the novelty and the additional empirical comparisons. However, I am still not convinced that establishing this particular depth separation in unnormalized LSA gives us a substantially new understanding of transformer depth, since prior work has already shown that cross-layer interactions can lead to richer, context-dependent optimization.

I do think the proposed diagnostic and fixed-step comparisons are useful contributions, but I am not yet convinced that these, together with the specific theoretical separation, are sufficient to establish the broader significance of the paper. I therefore intend to maintain my current score.

Thank you for the reply and for stating your remaining concern clearly.

We agree that prior work shows that layers can interact to produce richer, context-dependent optimisation. We also agree that the separation in linear self-attention is a specific result on its own.

Our claim about transformer depth rests on the two parts that you describe as useful. Prior work shows that such updates exist. It does not test whether a trained model computes one, against every sequence of the same number of fixed steps. The revised paper does, with two findings.

  1. Trained two-layer LSA models compute an update that two fixed steps cannot produce (Table 2).
  2. This is not specific to unnormalised LSA. In a standard softmax transformer, a single layer already outperforms two fixed optimisation steps (Section 8 and Table 3).

Reading each layer as one optimisation step therefore undercounts the layers of a standard transformer. The framework only needs a model's predictions, so it applies to any transformer.

If a specific result would establish the broader significance for you, please tell us.

Reviewer_QJLnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper analyzes the behavior of linear self attention, beyond the classic single layer setup. The results presented show that multi-layers do not behave simply as multiple fixed gradient steps. The Authors show some theory and some experiment validation to support their claim.

Critical strengths

None

Critical weaknesses

These are raw notes taken while reviewing your submission:

  • The problem you are trying to solve is not clearly defined, there is just a single sentence at line 37, but it's not enough for the reader to understand what you are trying to achieve. What is a "context-adaptive update"? How does it differ from other interpretations of (meta-) optimization in self-attention?

  • The current manuscript is too difficult to read with too many undefined terms and notation:

    • line 59: what is O1fix\mathcal O^{fix}_1?
    • line 107: what are PP and QQ?
    • line 141: what is GG?
    • line 211: what is vv?
    • line 217: what is O2fix\mathcal O^{fix}_2?
    • line 223: what is γ(s)\gamma(s), what is ss?
    • line 236: what are wkw_k?

Note that most of these are not defined even in the Appendix (vv is used at line 601 but never defined)

  • Connected with the point above, too many clearly ai generated sentences/words that make the text too hard to follow. Some examples: "nested oracle ladder", "ladder measures", "capacity control". "layer grafts", "identification result", "instruments", "atoms", "atom context ensemble", "depth-oracle ladder", "oracle-rung placement", "equivalence and superiority tests".

  • Because of the points above, the theory and Section 5, in particular, is for me difficult to assess: the mathematical rigor is completely missing, the theorems are not stated clearly and they mix the main statement with commentary of the result itself.

  • I don't understand the purpose of Table 1. What are you measuring? What is "share"? "gap" between what? what is the message that you are trying to express with this table?

  • Besides grave presentation problems, I cannot identify any novelty wrt to the literature. Mechanicistic analysis of multi-layer transformers is not new and there are many results already in the literature: for example, what is the novelty with respect to Fu et al?

    • line 59: "Single-layer LSA behaves as a fixed bilinear update and matches O1fix\mathcal O^{fix}_1 . Two-layer LSA exhibits an adaptive correction" I cannot classify this as contributions, this is already shown in Oswald et al. and in Fu et al.
Questions for authors

See above

2 comments

Thank you for your notes. The revised paper addresses your points, which we answer below in your order.

  1. We agree. The submission only stated the problem in one sentence. The revised paper now states the question (p.1). We also define the term context-adaptive in Definition 1 (p.4). Prior work asks which algorithm a model resembles, such as gradient descent or Newton's method. We ask whether LL fixed steps can produce what the model does.
  2. We agree completly, the the revised paper now defines all terms:
    • Section 3 defines O^{fix}_l (We cannot get this to render correctly) as ll gradient steps with fixed matrices, θk+1=θk+Mk(b−Σ^θk)\theta_{k+1} = \theta_k + M_k(b - \widehat\Sigma\theta_k).
    • Section 3 defines PP and QQ.
    • Section 4 defines the matrix in a single fixed preconditioned step, GG as y^=xq⊤Gb\hat y = x_q^\top G b.
    • Section 3 defines vv as the lower-left block of QQ.
    • A new paragraph before Theorem 2 defines ss and γ⋆(s)\gamma^\star(s).
    • Appendix A now starts with the notation.
    • Corollary 3 defines the weights ωk=πkκ(sk)\omega_k=\pi_k\kappa(s_k), which were wkw_k in the submission.
  3. We accept that adding the ladder idea might actually have reduced clarity. The revised paper now removes all references to this.
    • We have simplified compound terms where possible and define complex ideas before first use.
    • We believe the remaining terms are used in the literature. See, for example, Feller’s “An Introduction to Probability Theory and Its Applications”, Lehmann and Romano’s “Testing Statistical Hypotheses”. These terms are in the revised version, but have definitions where appropiate.
    • As instruments referred to ‘measurement instruments’, we have switched to measurements for clarity.
  4. We agree with your mixed claim/commentary comment. In the revised Section 5, we separate the two. The revised Theorem 2 now has numbered parts. Proposition 4 states its condition on Σ^\widehat\Sigma in full. A new Proposition 5 covers two fixed steps.
  5. Share and gap are defined in the two preceding paragraphs (Adaptive share and context conditional gap, p.3, now p.4). We accept that the caption did not reiterate the definitions, nor did it make it clear that we were using shorthand, reducing clarity for those going straight to the table. The revised caption defines each column and says what a positive value means.
  6. Von Oswald et al. and Ahn et al. do already show that one optimal layer equals one gradient step. Fu et al. show that deep GPT-2 models behave like Newton's method. They build a transformer that runs kk Newton steps with kk plus a constant number of layers. None of these papers establish the smallest depth at which a model can compute an update that the same number of fixed preconditioned steps cannot produce. We add four results:
    1. A one-layer LSA cannot implement a context-adaptive update for any parameters (Theorem 1).
    2. A two-layer LSA can have a strictly lower risk than every pair of fixed preconditioned steps (Theorem 2). Results 1 and 2 make depth two the smallest depth at which an LSA can implement a context-adaptive update.
    3. For every ll, there are task distributions where ll fixed steps have a positive excess risk, and one context-adaptive step does not (Corollary 3).
    4. Trained two-layer models have a lower held-out error than two jointly optimised fixed steps in all 20 seeds (4.08 against 5.14, Table 2).

The revised contribution states result 4 in place of the sentence that you quote (p. 2). The revised Related work describes von Oswald et al. and Fu et al. and lists results 1 to 3 (p. 2).

Dear Authors, thank you for your reply and the updated PDF. However, I want to point out how different the two revisions are. Fortunately, I kept the previous version and could compare them directly. The changes extend throughout the entire paper, to the point that I'd say it has been completely rewritten. Just to be clear, they are not just small edits or additions to the appendix to clarify some point, it's a different main paper (abstract included). In my view, this exceeds what is expected or acceptable during the rebuttal period. I will discuss this with the other reviewers if necessary but I will keep my score.

Reviewer_p78gOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper looks at the effects of increasing transformer depth on optimisation, and whether it leads to context-adaptive optimisation or more fixed optimisation steps. The main claim is that one-layer linear attention is unable to do context-adaptive optimisation, specifically covariance-dependent preconditioning, but a model with two or more layers can have better loss than two fixed optimisation step.

Critical strengths

The paper explores a really interesting depth-related optimisation setting, and the experiments are mainly solid as well in my opinion, as they explore adaptive optimisation at depth-two and some extensions to other tasks and architectures, which is very much appreciated.

Critical weaknesses

(1) It is unclear to me why the proof of Theorem 2 in Appendix A claims that γ(s)=a1+a2+a1a2s\gamma(s)=a_1+a_2+a_1a_2s, since the derivation preceding this argument states that y^2=−xq⊤(A1⊤+A2⊤+A2⊤Σ^A1⊤)b\widehat y_2=-x_q^\top(A_1^\top+A_2^\top+A_2^\top\widehat\Sigma A_1^\top)b, implying γ(s)=−(a1+a2+a1a2s)\gamma(s)=-(a_1+a_2+a_1a_2s) for isotropic parameters. So the condition should be c02+4c1=(a1−a2)2≥0c_0^2+4c_1=(a_1-a_2)^2\geq0 instead of c02−4c1≥0c_0^2-4c_1\geq0 in the paper, affecting further conclusions relying on this proof.

(2) Section 4 says that c(Σ^+λI)−1bc(\widehat\Sigma+\lambda I)^{-1}b, where c,λc,\lambda are shared across tasks, attains the Bayes floor ... but noise τ\tau varies across tasks, so I can't see how λ\lambda constant across tasks can be a part of the bayes-optimal predictor?

(3) Section 5 says that the experiment shows that the 'measurement', presumably a positive context-conditional gap Rad2−Rfix2>0R_{ad}^2-R_{fix}^2>0, does not come from a composition of two fixed updates. It seems false in the general case (although it may still hold for the experiment in Section 4): consider n=d=1n=d=1, s=x12∈{1,2}s=x_1^2\in\{1,2\} with uniform probability, b=x1y1b=x_1y_1. Two fixed gradient steps of size 1/21/2 mean c=1.5,λ=1c=1.5,\lambda=1, so Rad2−Rfix2=2/51>0R_{ad}^2-R_{fix}^2=2/51>0.

Questions for authors

In general my opinion is that this could be a very nice paper, if the reviewers could address the points listed in limitations. Particularly point (1), if this were to be addressed I would be willing to increase to a 4. My holdout is, as I described, the concern about the bayes optimal predictor formula.

2 comments

Thank you for your very careful review of the paper and the proofs. Your points were all correct, we have updated the proofs in our revision and answer your questions here in your order:

  1. Your observation is correct. The first step used the prediction from Ahn et al., the later seperation step used the coordinate without the minus sign. The revised Section 3 defines the prediction without the minus sign and says that Ahn et al. use the opposite sign.
    • Without the minus sign, γ(s)=a1+a2+a1a2s\gamma(s)=a_1+a_2+a_1a_2s and the condition is c02≥4c1c_0^2\ge 4c_1. The best fit c0+c1sc_0 + c_1s to (s+λ)−1(s+\lambda)^{-1} has slope c1<0c_1 < 0, because (s+λ)−1(s+\lambda)^{-1} decreases, so the condition always holds.
    • With the minus sign, the condition is c02+4c1≥0c_0^2+4c_1\ge 0, as you derived, which can fail. For example with s∈{0.5,1,2}s \in \{0.5, 1, 2\}, equal probabilities, and λ=0.5\lambda = 0.5, it equals −0.21-0.21. The proof (p. 14) now states both cases.
    • The conclusion in Theorem 2 therefore stands. Part (iii) is now more general, holding for every isotropic ensemble in which ss takes three values vs the submitted version, which had one example. It now compares the two-layer model with every predictor with the form xq⊤(C+DΣ^E)bx_q^\top(C+D\widehat\Sigma E)b, with CC, DD and EE fixed matrices. Every pair of fixed steps has this form (p. 6).
  2. Correct. A shared (c,λ)(c,\lambda) is not Bayes optimal when τ\tau varies across tasks, the revised paper removes this claim.
    • We now compute the Bayes risk by integrating over τ\tau (Appendix C). On three evaluation sets, the error of the single step c(Σ^+λI)−1bc(\widehat\Sigma+\lambda I)^{-1}b is 1.08 times the Bayes risk.
    • None of the theorems or measurements relied on this step being Bayes optimal. The submission used the claim to motivate the candidate family. The revised Section 4 now gives other reasons. In the revised Corollary 3, the noise level is fixed, and the step is the Bayes predictor.
  3. Correct again.
    • We also get 2/512/51 on your example. The revised paper now says that a positive share or gap rules out one fixed step and not two (p. 4). Our reference shows the same: with two jointly optimised fixed steps, the gap is +0.150+0.150 (Table 1).
    • The revised paper adds a new test for two fixed steps. Two fixed steps always have an update kernel of the form xq⊤(C+DΣ^E)xi/nx_q^\top(C+D\widehat\Sigma E)x_i/n, with fixed matrices CC, DD and EE (p. 4). We fit this form to each two-layer model’s update kernel. We then report the fraction of the kernel’s variance that the fit leaves unexplained. The new Proposition 5 in the revision shows that this fraction is zero for every pair of fixed steps.
    • The revised paper includes a new Table 2, containing the results of the above experiment, which we ran with 20 seeds. The fit for the trained two-layer models leave 6.1% of the kernel variance unexplained, compared to 0.0% for two jointly optimised fixed steps.

Thank you very much for the corrections. The mathematical revisions seem correct to me.

However, I feel that these new framings and corrections substantially narrow several of your core claims. Now the gap only rules out a single step, the classification results only support depth-1 claims, and the depth-2 transition is now restricted to linear attention. I feel like these results all changes reduce the scope and significance of the contribution rather than provide stronger support for the original claims. I'll therefore keep my rating of 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comments

We thank the three reviewers. Their comments have led to two new results, four corrections and hopefully, a clearer paper.

New

  1. Trained two-layer LSA models outperform two fixed optimisation steps, even when we optimise the two steps jointly. This holds in all 20 training runs (Table 2, p. 8).
  2. In standard softmax transformers, a single layer already outperforms two fixed optimisation steps (Section 8, p. 9, and Table 3, p. 20). The depth-two threshold is therefore specific to linear attention.

Corrections

  1. Sign (p78g). The sign in Section 3 is now fixed. Theorem 2 stands. Its separation now holds on every isotropic ensemble with three atoms (pp. 6 and 14).
  2. Bayes claim (p78g). The single context-adaptive step is not the Bayes predictor when the noise level varies. We removed the claim and computed the Bayes risk. The error of the step is 1.08 times that risk (Appendix C).
  3. Two fixed steps (p78g). A positive share or gap rules out one fixed step and not two, reflected in the revsied paper (p. 4). A new test covers two fixed steps (Proposition 5 and Table 2).
  4. Softmax (uxJh). We removed the claim that linear attention changes more sharply than softmax.

Clarity

  1. Clarity (QJLn). The problem is now made clear in the Introudction (p.1). Definition 1 defines a context-adaptive update (p. 4). All symbols are defined before first use. Terminology is simplified or defined at first use.
  2. Theorems and tables (QJLn). Section 5 separates each statement from its commentary. The caption of Table 1 defines each column (p. 5).
  3. Novelty and implications (QJLn, uxJh). Related work says what Ahn et al., von Oswald et al., Vladymyrov et al. and Fu et al. show and what we add (p. 2). Section 9 has a new “Implications” paragraph (p. 10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.