- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents an exact representation of softmax attention as a query-conditioned gradient step and derives finite-response formulas for positional and key–value interventions. Experiments demonstrate improved prediction of intervention effects and explore applications to cache management and answer repair.
Critical strengths
- The core identities are mathematically sound under the stated assumptions and apply without specially constructed projection weights or a linearized softmax.
- The finite-intervention analysis is useful and empirically supported across two model families, particularly the correction for joint key–value interactions.
Critical weaknesses
- The gradient-descent interpretation is an algebraic reformulation. Its objective is constructed from the current query and attention scores, so the equivalence alone does not establish an optimization mechanism underlying in-context learning.
- Practical benefits remain insufficiently validated. More accurate intervention predictions do not establish better compression or task performance. The paper should demonstrate higher compression at matched task quality, or better task performance under a fixed cache budget.
Questions for authors
What additional, testable insight does the gradient-descent interpretation provide beyond the exact identities? Can the proposed methods demonstrate better task quality at a fixed cache budget, or higher compression at matched quality, under matched optimization procedures?
Thank you for the response. I could not see the public responses that you mentioned, so my reply now is based on this rebuttal.
I do think that the key-value interaction results are interesting, but I decided not to list it as the strength because I do not consider it sufficient to outweigh W1. From my understanding, to claim In-context learning as gradient descent, it requires a matrix that is learned only from the context and can be reused in any queries. But, the paper's query-dependent matrix does not fulfill this criteria.
Regarding W3: I also agree that it shares the same identity, but the editing results do not depend on the gradient-descent interpretation. So I think these are two weakly connected stories. I think the paper would be stronger if you commit to one coherent storyline, especially given the title of the manuscript.
Lastly, about the readability: I try to thoroughly follow the equations. But there are quite a lot of equations that may not be necessary to be included in the main text; they make the paper harder to read. Since I could not see the responses to the other reviewers that the authors mentioned, I would like to keep my original ratings.