acceptodds
Withdrawn from ICLR 2027

Softmax Attention is an Exact Gradient Step: Predictable Finite Responses from RoPE Derivatives

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transformers are said to learn in context by gradient descent, but the claim is proven only where pretrained models do not operate: in linear attention, or at weights constructed to make it true. For the softmax attention they use, it survives only as an approximation that linearises the exponential, and whether pretrained models implement gradient descent at all is disputed (Shen et al., 2024). We show that no approximation is needed: every softmax attention head, for any weights, context and query, computes exactly one step of gradient descent. Its output is the uniform average of its values plus the prediction, at its query, of a linear model trained by one gradient step of weighted least-squares regression from keys to centred values. Each example is weighted by the exponential’s divided difference at its attention score, so every weight is positive and rises with the score. As scores shrink, the weights become uniform and the step becomes the linear-attention gradient step of von Oswald et al. (2023). The loss’s curvature vanishes exactly on query edits that leave attention unchanged, and one linear solve finds the smallest edit realising requested attention ratios. The same identity, taken between two states of the head, gives the exact response to any finite edit, including moving a token, whose rotation follows from integrating RoPE’s derivative. In pretrained Qwen2.5 and SmolLM2, the gradient step reproduces every evaluated head to floating-point precision, and across 153,600 executed positional edits, exact responses cut the error of linearised predictions by 73.6–98.3%. Softmax attention does not approximate gradient descent in context. It performs it, exactly, in every head.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, attention mechanisms, rotary position embeddings, attribution patching, finite interventions, key-value cache

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SoM6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an exact representation of softmax attention as a query-conditioned gradient step and derives finite-response formulas for positional and key–value interventions. Experiments demonstrate improved prediction of intervention effects and explore applications to cache management and answer repair.

Critical strengths
  1. The core identities are mathematically sound under the stated assumptions and apply without specially constructed projection weights or a linearized softmax.
  2. The finite-intervention analysis is useful and empirically supported across two model families, particularly the correction for joint key–value interactions.
Critical weaknesses
  1. The gradient-descent interpretation is an algebraic reformulation. Its objective is constructed from the current query and attention scores, so the equivalence alone does not establish an optimization mechanism underlying in-context learning.
  2. Practical benefits remain insufficiently validated. More accurate intervention predictions do not establish better compression or task performance. The paper should demonstrate higher compression at matched task quality, or better task performance under a fixed cache budget.
Questions for authors

What additional, testable insight does the gradient-descent interpretation provide beyond the exact identities? Can the proposed methods demonstrate better task quality at a fixed cache budget, or higher compression at matched quality, under matched optimization procedures?

Reviewer_b5XzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops a mathematical framework for analyzing finite changes to RoPE-based softmax attention using the exponential divided-difference function. The authors derive exact expressions for changes in attention outputs under positional shifts and joint key-value interventions. A central theoretical claim is that every softmax attention head can be represented as a single gradient-descent step on a query-dependent weighted least-squares objective. The paper also derives results for attention-preserving query modifications and applies the framework to intervention ranking and KV-cache restoration. The experiments use pretrained Qwen2.5 and SmolLM2 models. The proposed finite-response formulas reduce downstream prediction error relative to first-order attribution baselines. Additional experiments investigate cache restoration, computational efficiency, and controlled regression tasks.

Critical strengths
  1. Exact characterization of finite attention interventions. The paper provides a useful decomposition of finite attention changes. Particularly, the interaction term between key and value modifications. This makes it possible to identify what is lost when these effects are treated separately. I also found the connection between the finite response and the logarithmic-mean formulation interesting.

  2. Careful empirical validation. The authors evaluate their predictions across two model families and multiple intervention settings with paired bootstrap intervals and numerical consistency checks. The experiments generally support the accuracy of the proposed finite-response formulas. I appreciate the analysis of different RoPE frequency bands.

Critical weaknesses
  1. The interpretation and novelty of the exact gradient-descent result are not sufficiently convincing.

My main concern is with the paper's central claim that every softmax attention head performs an exact gradient-descent step. Although the algebraic identity appears correct the weighted least-squares objective is constructed using coefficients that depend on the current query and attention scores. So, the effective update matrix generally changes across queries rather than representing a shared regression model learned from the context.

The paper also acknowledges that the effective matrix is not unique under different score-shift conventions (Appendix C), and the identity holds regardless of whether the model is pretrained or randomly initialized.

I think this distinction matters for the paper's positioning relative to prior work on in-context gradient descent. Showing that an attention computation admits an exact gradient-step representation is not the same as demonstrating that a pretrained transformer has learned to perform gradient descent on an independently specified task objective. Appendix I partially illustrates this distinction through its comparison with an independent regression learner.

Given how prominently the gradient-descent claim appears in the title, abstract, and introduction, I find the current interpretation somewhat overstated. The algebraic representation is interesting, but its additional conceptual insight beyond existing interpretations of attention is not sufficiently established.

  1. The practical novelty beyond direct attention recomputation is unclear.

The finite-response formulation gives the exact change in an attention head's output, but the same local result can be obtained by directly recomputing attention using the modified keys and values.

This raises the question of what additional practical advantage the proposed calculus provides beyond decomposing that response into interpretable terms. The comparisons with AtP and AtP* demonstrate improvements over approximate attribution, but they do not fully establish the incremental value relative to direct local recomputation.

I also have some concerns about the positional experiments. The first-order Jacobian baseline is evaluated under relatively large RoPE shifts where linear approximations are naturally expected to perform poorly. A comparison that uses exact rotary score changes while retaining a first-order softmax approximation would help isolate the contribution of the softmax correction itself.

Finally, the runtime experiments show that the sparse method is slower than dense recomputation for single-use evaluation and offers relatively modest benefits when preparation is reused. The demonstrated behavioral impact is also limited, with only one adaptively selected answer-repair example.

Overall, I find the mathematical treatment sound and potentially useful but I am not yet convinced that the paper establishes sufficient conceptual or practical novelty beyond existing attention computations and attribution methods.

Questions for authors
  1. Gradient-descent interpretation and novelty.

Since the regression objective in Equation (11) depends on the query and attention scores, what does the exact gradient-step representation tell us about in-context learning beyond an algebraic equivalence? In particular, what evidence would distinguish a model genuinely implementing an in-context optimization procedure from one that simply admits this representation?

  1. Benefit beyond direct recomputation.

Table J.1 already shows that the exact-joint method produces the same results as direct dense attention recomputation. Could the authors clarify what the proposed calculus offers beyond this equivalence, particularly in terms of interpretability, computational efficiency or downstream intervention performance?

  1. Isolating the source of improvement in positional experiments.

Could the authors include a comparison that uses the exact RoPE-induced score changes but applies a first-order approximation only to softmax? This would help distinguish improvements due to handling finite rotations exactly from improvements due to the nonlinear softmax correction.

Reviewer_e1RjOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper derives formulas for how an attention head's output changes after an edit, such as moving a key or swapping in keys and values from another prompt. Existing tools (AtP, AtP$^*$) predicts these changes. All the formulas come from one function, φ1(x)=(ex−1)/x\varphi_1(x) = (e^x - 1)/x. The same function lets the authors rewrite any softmax head's output as the average of its values plus one gradient-descent step on a regression over the context. From this, they argue that softmax attention does gradient descent in context. The paper experiments using Qwen2.5 (0.5B, 1.5B) and SmolLM2-1.7B.

Critical strengths
  • Key-value interaction results are interesting. See W1 why I think this is insufficient as a reason for acceptance
Critical weaknesses
  • W1: An exact algebraic identity is not evidence of learned gradient descent.

    • In the "in-context-learning as gradient descent" context, the model learns one matrix from the context and applies it to every query. In eq. 11, the regression weights cj=φ1(sj)/Zc_j = \varphi_1(s_j)/Z depend on the query's attention scores. So the "learned" matrix ΔMi\Delta M_i changes with every query. But from my understanding, nothing in eq. 11 is learned once from the context and then reused.
    • It holds "for any weights and query" (§3.3), so a random head satisfies it too. Matching pretrained heads to floating-point precision (abstract) only tests the code. Deutch et al. (2024) found that untrained models look as GD-like as trained ones.
    • The authors did test on trained heads (App. I)m but it shows an opposite finding: agreement with one-step GD is 0.82–0.96, against 0.99 for linear attention. Ren & Liu (2024) also treats softmax attention as a model trained on the context (should be cited).
  • W2: The positional result in the abstract uses a baseline that is worse than predicting no change.

    • The 73.6-98.3% gain in the abstract is over the positional Jacobian. In every row of Table 1, the Jacobian is worse than predicting no change (e.g., 0.160 against 0.057 on Qwen Corrupt). Table 1 reports the reduction against no change, but the authors should not use only the Jacobian result in the abstract.
  • W3: The two contributions are weakly connected.

    • The gradient-descent part and the editing formulas share notation only. Theorem 2 is a toy example that does not connect to the rest. The code-repair example is one case, chosen after the fact, with the correct answer supplied.

References

  • Deutch et al. (2024), In-context Learning and Gradient Descent Revisited.
  • Ren & Liu (2024), Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens.
Questions for authors
  • Q1: How were the test layers chosen? Qwen uses {0,14,21}\{0, 14, 21\} out of {0,7,14,21,27}\{0, 7, 14, 21, 27\}. SmolLM2 uses {12,18}\{12, 18\} out of {0,6,12,18,23}\{0, 6, 12, 18, 23\}. Did the choice depend on how well the formula did? How does it do on the other layers?
  • Q2: Against predicting no change, the formula cuts error by 86–92% on SmolLM2, far more than on Qwen. Why is that the case?

Comments on the paper

5 comments

Thank you for the response. I could not see the public responses that you mentioned, so my reply now is based on this rebuttal.

I do think that the key-value interaction results are interesting, but I decided not to list it as the strength because I do not consider it sufficient to outweigh W1. From my understanding, to claim In-context learning as gradient descent, it requires a matrix that is learned only from the context and can be reused in any queries. But, the paper's query-dependent matrix does not fulfill this criteria.

Regarding W3: I also agree that it shares the same identity, but the editing results do not depend on the gradient-descent interpretation. So I think these are two weakly connected stories. I think the paper would be stronger if you commit to one coherent storyline, especially given the title of the manuscript.

Lastly, about the readability: I try to thoroughly follow the equations. But there are quite a lot of equations that may not be necessary to be included in the main text; they make the paper harder to read. Since I could not see the responses to the other reviewers that the authors mentioned, I would like to keep my original ratings.

Thanks for the reply.

About the Strengths: The review form asks for "one or two critical strengths of the paper that support acceptance", and that "the goal is not to enumerate every positive point". The form also allows me to enter None "if no substantive strength can be identified". The interesting key-value findings is outweighed by the W1 and because of that, I cannot call this a "substantive strength"; so I entered "None". This is my judgement, I leave this point for the AC to decide.

W1: The trained/untrained comparison does answer my concern about the random heads. My concern about the query-dependence is still largely unanswered; or at least I see two different versions of the gradient step in your reply to me and to b5Xz. In the reply to me, the step lies in the kernel's feature space which is shared across all queries, but any normalized kernal smoother can be written in this way, right? Also the version of the gradient step you sent in the reply to me does not seem to be present in the paper (correct me if I am wrong). While in the reply to b5Xz, the step is the explicit finite-dimensional one from the eq. 11, which is different (not in an implicit kernel feature space). The finite step in eq. 11 only fits real models if the learning is "recalculated" for each question, instead of being learned once from the context and being reused for any queries.

W2, W3, Q1, and Q2: all resolved. I will change the soundness score accordingly.

Readability: Please submit the revised draft so that everyone can check the changes.

Thank you.

Fairness: Reviewers come to a paper with different levels of familiarity, and my confidence score reflects mine; I do encourage the AC to take that into consideration while noting that the same concerns were brought up by other reviewers. I have to repeat that I found the paper very hard to follow. The main text should contain what is needed to check the main claims and be very concise. At the moment, the main text contains a lot of equations that do not substantively contribute to the main story, and a lot of details are mentioned only in the 35 pages of appendices.

W1: It seems that we largely agree on the facts mentioned, but not necessarily on the conclusions. The exact gradient step is a per-query identity that holds for any head; this is a narrower claim than the ones mentioend in the title, abstract, and introduction. What would resolve my concern is a statement of scope in the abstract and introduction. You might consider removing the abstract's final sentences ("Softmax attention does not approximate gradient descent in context. It performs it, exactly, in every head."). If a title change is possible (maybe not), a narrower title would help too; at the moment, the title promises a more generalizable finding about gradient step which is not present in the paper. I will check these and the readability changes in the revised draft before reconsidering my rating.

W2, W3, Q1 and Q2 are resolved. My rating rests on W1 (which is the paper's main claim) and on readability, and I would like to see the revised draft before changing it. Please aim to have the revised draft before the discussion closes so that I can reconsider my ratings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.