- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how gradient descent trains a one-layer softmax-attention model to retrieve labels from in-context examples over a fixed set of input prototypes. It shows that training depends on two quantities: the prototypes’ Gram matrix and the covariance of task outputs across prototypes. Under symmetric geometry, the dynamics reduce to a scalar recurrence with an initial routing stage followed by refinement, yielding explicit convergence-time bounds. The paper also examines when correlated representations permit convergence or cause failure. Although the task functions may be nonlinear, the analyzed mechanism is matching queries to context prototypes rather than recovering the underlying function.
Critical strengths
- The paper clearly connects input geometry and task statistics to attention learning dynamics, deriving explicit convergence rates through an exact scalar reduction in the symmetric setting.
Critical weaknesses
- The analysis closely extends Huang et al. [1], retaining the same prototype-matching mechanism and two-stage dynamics. The additional technical contributions are insufficiently distinguished from prior work.
- In the main symmetric setting, task geometry only rescales the loss and effective learning rate, without introducing qualitatively different learning dynamics.
- The appendix reads like ''AI slop": the exposition is difficult to follow and obscures the key mathematical arguments, making the claimed technical advances hard to assess.
[1] Yu Huang, Yuan Cheng, and Yingbin Liang. In-Context Convergence of Transformers. arXiv:2310.05249, 202
Questions for authors
- Which proof steps and qualitative findings cannot be obtained by extending the analysis of [1], particularly in the symmetric setting?
- Does anisotropic task covariance produce qualitatively different learning dynamics beyond scalar rescaling? Can the authors provide a concrete example or theoretical result?
4 comments
A2. Task Shape and Dynamics
In the isotropic symmetric setting, task scale changes the loss and effective learning rate without introducing a new routing mechanism. Our contribution there is the matching stagewise complexity and accuracy-dependent crossover described in A1, which determine when routing or refinement controls the training budget. Anisotropic task shape introduces a different effect: it can change the ordering of attention during loss descent.
Theoretical distinction. Proposition S1 shows that replacing by is equivalent to replacing by along the entire attention trajectory. Under uniform queries, balanced contexts, zero initialization, and fixed , two task matrices give identical first-step attention at positive step sizes if and only if . Under these conditions, a scalar learning-rate adjustment cannot reproduce a change in task shape.
Concrete example. Proposition S2 fixes uniform queries, balanced contexts, and zero initialization. It takes , , with , and . Both priors have trace 15, initial loss , and Lipschitz constant . At the same step size , the first class-1 matching margin is under anisotropy and under isotropy, while both losses decrease. The isotropic trajectory retains the correct ordering at all subsequent iterations.
Additional numerical evidence. Integrating the exact population update from zero at gives the following results. Majority attention means matching attention at least in every class.
| Metric | Anisotropic | Isotropic |
|---|---|---|
| Class-1 matching margin negative | Steps 1-19,263 | Never |
| All classes reach majority attention | Step 23,916 | Step 1,226 |
| First loss at most 0.5 | Step 2,467 | Step 3,571 |
The distractor still outranks the matching prototype at iteration 19,263, when loss has fallen from 2.5 to approximately 0.0804. The dominant contrast can be fitted by a mixture of distractors, so reducing its error need not establish correct retrieval. The weaker independent contrast distinguishes that mixture from the matching prototype. This differs from the isotropic symmetric trajectory, where loss decreases as matching attention increases. Proposition S2 proves the initial reversal. Its persistence and the stopping times above are numerical.
We added Propositions S1-S2 and their proofs in Appendix D.8, explained scale versus shape after Proposition 1, and clarified the isotropic scope in Sections 4-5, the abstract, and the conclusion.
A3. Proof Exposition
We agree that the appendix needed clearer exposition. The analysis must control coupled class gradients, discrete boundary crossings, and matching upper and lower drift bounds. We revised the presentation to explain the role of these arguments:
- Appendix A maps each main result to its dependencies, key argument, and proof location.
- Main proofs begin with their central idea, and longer proofs are divided into named steps.
- Appendix E.4 consolidates the stagewise drift and crossing estimates. Appendix F.1 separates the conversion to accuracy-dependent stopping times. The random-count loss comparison now precedes its first use in Appendix E.3.
The core proofs were developed by the authors. As disclosed in the AI Use Statement, AI tools assisted with drafting, proof auditing, numerical verification, and experiment implementation. We reviewed the AI-assisted work and take responsibility for the final claims and proofs.
Revised manuscript passages are highlighted in blue. We would appreciate reconsideration of the evaluation in light of these clarifications and results.
[1] Yu Huang, Yuan Cheng, and Yingbin Liang. In-Context Convergence of Transformers. ICML 2024.
Finally, if our response resolves your concerns to a satisfactory level, we wonder if the reviewer could kindly consider raising the score of your evaluation. We thank the reviewer again for the helpful comments and suggestions for our work.
I thank the author for the comprehensive response and I increase the score to 2. My main unsolved concern is still about the contribution. For instance, authors claim 'Task scale rescales the effective learning rate,while task shape can change the direction of attention updates', the rescale is eliminated by condition for convergence due to theoretical nessecity, and the task shape decides the margin among tasks, which should definitely change the direction and convergence rate. Overall, I dont find the result surprising enough but I agree it's technically sound.
Thank you for reconsidering and confirming soundness. We agree that the scale and shape statements alone are unsurprising.
Convergence can fail. Proposition 2 has full-rank features and matching examples in every context, so the model can approach zero loss. Yet GD from zero initialization permanently loses the rare class and converges to positive loss for every . Under the stated step-size condition, the loss decreases throughout. Changing only to restores zero-loss convergence, consistent with [1], where imbalance only delays under-represented features. Thus geometry changes whether retrieval is learned, and no scalar learning-rate normalization repairs this failure. Theorems 2–3 provide complementary conditions for global convergence.
Tight routing bounds. At fixed , Proposition 4 establishes matching routing bounds under its symmetry and balanced-count assumptions. The lower bound identifies an actual bottleneck along the analyzed GD trajectory.
Regarding scale, we agree that makes attention trajectories identical. At fixed absolute accuracy, however, the normalized stopping threshold still changes. At and , increasing from 32 to 256 raises the stopping time from 17,456 to 40,457 steps under the curvature-scaled step, while a common step reduces it from 279,284 to 80,914 (Table 5 and Appendix G.3). This illustrates policy-dependent stopping costs, without implying new scale-dependent dynamics.
We hope these clarifications address your remaining concern about the contribution, and we would welcome your view on whether the convergence-versus-failure distinction changes your assessment.
We thank the reviewer for the direct assessment. The prototype-matching mechanism and two-stage picture are shared with [1]. We distinguish the new outcomes, proof ingredients, and sharp complexity results below.
A1. Distinct Results and Proofs
We use the same architecture, merged query-key matrix, and zero initialization as [1]. Three sets of results go beyond that analysis.
(a) Geometry can change the outcome. The imbalanced-feature result in [1] shows that orthonormal features allow every class to converge despite unequal frequencies. Proposition 2 gives a full-rank example with G11=1, G12=3, G22=10, query probabilities p=(100/101,1/101), isotropic task contrast, and both classes present in every context. For every η>0, GD from zero drives rare-class matching attention to zero and Lt→p2>0. Changing only G to I restores zero-loss convergence. The update S+=S−ηGHG couples query classes that evolve independently when G=I. The proof constructs an invariant cone in joint score-gap space where the frequent class's transported gradient dominates the rare class's corrective signal (Appendix D.5).
(b) Coupling requires additional proof ingredients. Theorems 2-3 establish global convergence from zero for near-orthogonal and transitive geometries under their stated correlation bounds. Theorem 2 uses drift estimates on target-distractor tie boundaries and a discrete non-crossing lemma (Lemmas 5-6, Appendix D.6). Theorem 3 uses gradient synchronization, H(t)1=0, to cancel the common Gram direction (Lemmas 7-8, Appendix D.9). These arguments control coupling terms absent when G=I. For arbitrary G≻0, Theorem 1 establishes monotone descent under its step-size condition, with its convergence rate conditional on routing-basin entry.
(c) Symmetry yields sharp stopping times. The scalar recurrence follows by specializing the score-gradient identities, and we do not claim a new two-stage mechanism. Beyond the convergence upper bounds in [1], we bound the decrease of off-target odds Rt and the increase of inverse-squared attention error ut−2 from both sides, including the discrete step that crosses between stages. Under stronger symmetry and exactly balanced contexts, this gives:
On the common uniform specialization G=I and Σf=P⊥, our bound is O(K2/η+K/(ηϵ)), removing the logarithmic factors in [1]. This comparison uses stronger symmetry and different prompt-length conditions. Our random-prompt theorem bounds absolute loss under an accuracy-dependent prompt-length condition, while [1] allows pk=Θ(1/K) and bounds excess loss L−L⋆. We do not claim to subsume their results.
We revised the Introduction's contribution summary, added Table 3 in Appendix B, and added the proof roadmap in Appendix A to distinguish the shared mechanism from these results and arguments.