acceptodds
Under review as a conference paper at ICLR 2027

A Geometry-Dependent Theory of Softmax-Attention In-Context Learning Dynamics

Abstract

Transformers learn new tasks from examples placed in the prompt, yet theory of how gradient descent trains softmax attention is largely limited to linear tasks and orthogonal features. We study one-layer softmax attention trained on in-context regression with nonlinear task functions over non-orthogonal feature prototypes. Our analysis rests on an exact decomposition of the population loss and gradient: the task distribution enters only through the covariance of its outputs across prototypes, and the representation only through its Gram matrix, which transports every gradient from both sides. Task contrast therefore sets the clock of training, whereas representation geometry decides whether and how fast training converges. For every full-rank geometry we prove monotone descent and an inverse-accuracy rate once attention has found the right prototype; a geometry that violates self-similarity can trap training at nonzero loss, whereas small correlations and transitive geometries such as prototypes on a ring converge globally from zero initialization. Under symmetric geometry, training reduces exactly to a scalar recurrence with two laws: coarse routing, whose duration grows quadratically in the number of prototypes, and inverse-square refinement. The ratio of task contrast to target accuracy decides which law is active when the target is met, giving two exact thresholds, a crossover at which refinement dominates training time, and a learning-rate policy that decides whether stronger contrast speeds up or slows down convergence. Exact integrations and finite-sample SGD on the token-level model confirm the predicted boundaries and scaling laws.

Primary area:
learning theory
Keywords:
in-context learning, transformer, softmax attention, training dynamics, optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9h1jOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how gradient descent trains a one-layer softmax-attention model to retrieve labels from in-context examples over a fixed set of input prototypes. It shows that training depends on two quantities: the prototypes’ Gram matrix and the covariance of task outputs across prototypes. Under symmetric geometry, the dynamics reduce to a scalar recurrence with an initial routing stage followed by refinement, yielding explicit convergence-time bounds. The paper also examines when correlated representations permit convergence or cause failure. Although the task functions may be nonlinear, the analyzed mechanism is matching queries to context prototypes rather than recovering the underlying function.

Critical strengths
  • The paper clearly connects input geometry and task statistics to attention learning dynamics, deriving explicit convergence rates through an exact scalar reduction in the symmetric setting.
Critical weaknesses
  • The analysis closely extends Huang et al. [1], retaining the same prototype-matching mechanism and two-stage dynamics. The additional technical contributions are insufficiently distinguished from prior work.
  • In the main symmetric setting, task geometry only rescales the loss and effective learning rate, without introducing qualitatively different learning dynamics.
  • The appendix reads like ''AI slop": the exposition is difficult to follow and obscures the key mathematical arguments, making the claimed technical advances hard to assess.

[1] Yu Huang, Yuan Cheng, and Yingbin Liang. In-Context Convergence of Transformers. arXiv:2310.05249, 202

Questions for authors
  • Which proof steps and qualitative findings cannot be obtained by extending the analysis of [1], particularly in the symmetric setting?
  • Does anisotropic task covariance produce qualitatively different learning dynamics beyond scalar rescaling? Can the authors provide a concrete example or theoretical result?
4 comments

We thank the reviewer for the direct assessment. The prototype-matching mechanism and two-stage picture are shared with [1]. We distinguish the new outcomes, proof ingredients, and sharp complexity results below.

A1. Distinct Results and Proofs

We use the same architecture, merged query-key matrix, and zero initialization as [1]. Three sets of results go beyond that analysis.

(a) Geometry can change the outcome. The imbalanced-feature result in [1] shows that orthonormal features allow every class to converge despite unequal frequencies. Proposition 2 gives a full-rank example with G11=1G_{11}=1, G12=3G_{12}=3, G22=10G_{22}=10, query probabilities p=(100/101,1/101)p=(100/101,1/101), isotropic task contrast, and both classes present in every context. For every η>0\eta>0, GD from zero drives rare-class matching attention to zero and Lt→p2>0L_t\to p_2>0. Changing only GG to II restores zero-loss convergence. The update S+=S−ηGHGS^{+}=S-\eta GHG couples query classes that evolve independently when G=IG=I. The proof constructs an invariant cone in joint score-gap space where the frequent class's transported gradient dominates the rare class's corrective signal (Appendix D.5).

(b) Coupling requires additional proof ingredients. Theorems 2-3 establish global convergence from zero for near-orthogonal and transitive geometries under their stated correlation bounds. Theorem 2 uses drift estimates on target-distractor tie boundaries and a discrete non-crossing lemma (Lemmas 5-6, Appendix D.6). Theorem 3 uses gradient synchronization, H(t)1=0H^{(t)}\mathbf 1=0, to cancel the common Gram direction (Lemmas 7-8, Appendix D.9). These arguments control coupling terms absent when G=IG=I. For arbitrary G≻0G\succ0, Theorem 1 establishes monotone descent under its step-size condition, with its convergence rate conditional on routing-basin entry.

(c) Symmetry yields sharp stopping times. The scalar recurrence follows by specializing the score-gradient identities, and we do not claim a new two-stage mechanism. Beyond the convergence upper bounds in [1], we bound the decrease of off-target odds RtR_t and the increase of inverse-squared attention error ut−2u_t^{-2} from both sides, including the discrete step that crosses between stages. Under stronger symmetry and exactly balanced contexts, this gives:

  • Routing time TI=Θ(K2/(ησf2γ2))T_I=\Theta(K^2/(\eta\sigma_f^2\gamma^2)) for K≥3K\ge3 (Proposition 4). The lower bound establishes an actual bottleneck along the analyzed GD trajectory.
  • Exact thresholds qinit=2K/(K−1)q_{\mathrm{init}}=2K/(K-1) and qtail=8(K−1)/Kq_{\mathrm{tail}}=8(K-1)/K in q=σf2/ϵq=\sigma_f^2/\epsilon, and a cost crossover at q−qtail=Θ(K)q-q_{\mathrm{tail}}=\Theta(K) (Theorem 5). A target can lie in refinement while routing still dominates its cost.

On the common uniform specialization G=IG=I and Σf=P⊥\Sigma_f=P_\perp, our bound is O(K2/η+K/(ηϵ))O(K^2/\eta+K/(\eta\epsilon)), removing the logarithmic factors in [1]. This comparison uses stronger symmetry and different prompt-length conditions. Our random-prompt theorem bounds absolute loss under an accuracy-dependent prompt-length condition, while [1] allows pk=Θ(1/K)p_k=\Theta(1/K) and bounds excess loss L−L⋆L-L^\star. We do not claim to subsume their results.

We revised the Introduction's contribution summary, added Table 3 in Appendix B, and added the proof roadmap in Appendix A to distinguish the shared mechanism from these results and arguments.

A2. Task Shape and Dynamics

In the isotropic symmetric setting, task scale changes the loss and effective learning rate without introducing a new routing mechanism. Our contribution there is the matching stagewise complexity and accuracy-dependent crossover described in A1, which determine when routing or refinement controls the training budget. Anisotropic task shape introduces a different effect: it can change the ordering of attention during loss descent.

Theoretical distinction. Proposition S1 shows that replacing Σf\Sigma_f by cΣfc\Sigma_f is equivalent to replacing η\eta by cηc\eta along the entire attention trajectory. Under uniform queries, balanced contexts, zero initialization, and fixed G≻0G\succ0, two task matrices give identical first-step attention at positive step sizes if and only if η1Σ1=η2Σ2\eta_1\Sigma_1=\eta_2\Sigma_2. Under these conditions, a scalar learning-rate adjustment cannot reproduce a change in task shape.

Concrete example. Proposition S2 fixes uniform queries, balanced contexts, and zero initialization. It takes K=3K=3, G=(I+11⊤)/2G=(I+\mathbf 1\mathbf 1^\top)/2, ΣA=P⊥/2+bb⊤\Sigma_A=P_\perp/2+bb^\top with b=(1,2,−3)⊤b=(1,2,-3)^\top, and ΣI=15P⊥/2\Sigma_I=15P_\perp/2. Both priors have trace 15, initial loss 5/25/2, and Lipschitz constant 30\sqrt{30}. At the same step size 0<η≤1/3480<\eta\le1/348, the first class-1 matching margin is −η/72-\eta/72 under anisotropy and 5η/245\eta/24 under isotropy, while both losses decrease. The isotropic trajectory retains the correct ordering at all subsequent iterations.

Additional numerical evidence. Integrating the exact population update from zero at η=1/348\eta=1/348 gives the following results. Majority attention means matching attention at least 1/21/2 in every class.

MetricAnisotropicIsotropic
Class-1 matching margin negativeSteps 1-19,263Never
All classes reach majority attentionStep 23,916Step 1,226
First loss at most 0.5Step 2,467Step 3,571

The distractor still outranks the matching prototype at iteration 19,263, when loss has fallen from 2.5 to approximately 0.0804. The dominant contrast can be fitted by a mixture of distractors, so reducing its error need not establish correct retrieval. The weaker independent contrast distinguishes that mixture from the matching prototype. This differs from the isotropic symmetric trajectory, where loss decreases as matching attention increases. Proposition S2 proves the initial reversal. Its persistence and the stopping times above are numerical.

We added Propositions S1-S2 and their proofs in Appendix D.8, explained scale versus shape after Proposition 1, and clarified the isotropic scope in Sections 4-5, the abstract, and the conclusion.

A3. Proof Exposition

We agree that the appendix needed clearer exposition. The analysis must control coupled class gradients, discrete boundary crossings, and matching upper and lower drift bounds. We revised the presentation to explain the role of these arguments:

  • Appendix A maps each main result to its dependencies, key argument, and proof location.
  • Main proofs begin with their central idea, and longer proofs are divided into named steps.
  • Appendix E.4 consolidates the stagewise drift and crossing estimates. Appendix F.1 separates the conversion to accuracy-dependent stopping times. The random-count loss comparison now precedes its first use in Appendix E.3.

The core proofs were developed by the authors. As disclosed in the AI Use Statement, AI tools assisted with drafting, proof auditing, numerical verification, and experiment implementation. We reviewed the AI-assisted work and take responsibility for the final claims and proofs.

Revised manuscript passages are highlighted in blue. We would appreciate reconsideration of the evaluation in light of these clarifications and results.

[1] Yu Huang, Yuan Cheng, and Yingbin Liang. In-Context Convergence of Transformers. ICML 2024.

Finally, if our response resolves your concerns to a satisfactory level, we wonder if the reviewer could kindly consider raising the score of your evaluation. We thank the reviewer again for the helpful comments and suggestions for our work.

I thank the author for the comprehensive response and I increase the score to 2. My main unsolved concern is still about the contribution. For instance, authors claim 'Task scale rescales the effective learning rate,while task shape can change the direction of attention updates', the rescale is eliminated by condition η=O(σf2)\eta=O(\sigma_f^2) for convergence due to theoretical nessecity, and the task shape decides the margin among tasks, which should definitely change the direction and convergence rate. Overall, I dont find the result surprising enough but I agree it's technically sound.

Thank you for reconsidering and confirming soundness. We agree that the scale and shape statements alone are unsurprising.

Convergence can fail. Proposition 2 has full-rank features and matching examples in every context, so the model can approach zero loss. Yet GD from zero initialization permanently loses the rare class and converges to positive loss for every η>0\eta>0. Under the stated step-size condition, the loss decreases throughout. Changing only GG to II restores zero-loss convergence, consistent with [1], where imbalance only delays under-represented features. Thus geometry changes whether retrieval is learned, and no scalar learning-rate normalization repairs this failure. Theorems 2–3 provide complementary conditions for global convergence.

Tight routing bounds. At fixed ησf2γ2\eta\sigma_f^2\gamma^2, Proposition 4 establishes matching K2K^2 routing bounds under its symmetry and balanced-count assumptions. The lower bound identifies an actual bottleneck along the analyzed GD trajectory.

Regarding scale, we agree that η∝1/σf2\eta\propto1/\sigma_f^2 makes attention trajectories identical. At fixed absolute accuracy, however, the normalized stopping threshold ϵ/σf2\epsilon/\sigma_f^2 still changes. At K=32K=32 and ϵ=0.01\epsilon=0.01, increasing q=σf2/ϵq=\sigma_f^2/\epsilon from 32 to 256 raises the stopping time from 17,456 to 40,457 steps under the curvature-scaled step, while a common step reduces it from 279,284 to 80,914 (Table 5 and Appendix G.3). This illustrates policy-dependent stopping costs, without implying new scale-dependent dynamics.

We hope these clarifications address your remaining concern about the contribution, and we would welcome your view on whether the convergence-versus-failure distinction changes your assessment.

Reviewer_z45qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies ICL for nonlinear functions over a specific set of (not orthogonal) features. The paper demonstrates that the attention layer learns to restrict attention to tokens within the same class over two phases - the first of is interpreted as learning to identify the matching prototypes and the second is interpreted as supressing the attention on the incorrect ones. This is also inverted to explain how long it would take to converge to a given accuracy level

Critical strengths

I could not find any errors in the paper, and it is well written.

Critical weaknesses
  1. Some of the theorems in the main paper could be less detailed (reserving constants and such for the appendix)
  2. More importantly: I am not sure how significantly different the story told here is from the one in Huang et al. There I believe they consider orthogonal features but ultimately say the same thing - that the attention layer learns to focus its attention on the matching prototypes. This paper claims to generalize to non-linear functions and non-orthogonal features, but I am not sure that this is a significant change in insight.
Questions for authors
  1. What does this say different from Huang et al "In Context Convergence of Transformers"?
  2. Does it matter that we are optimizing a single Q rather than a K and a Q?
1 comment

We thank the reviewer for the careful reading and constructive questions.

A1. Clearer Theorem Statements

Theorems 2-3 now state the structural assumptions, convergence guarantees, and parameter dependence using universal constants. Explicit constants, recurrences, and iteration counts appear in Appendices D.6 and D.9. Proposition 4 and Theorem 5 retain the constants defining their sharp bounds and accuracy thresholds, together with the order-level summaries in Eqs. 22 and 27.

A2. Distinctions from Huang

The prototype-matching endpoint and routing-then-refinement mechanism are shared with [1]. Our contributions explain when training fails to reach that endpoint, how task shape changes routing, and which stage determines the training budget.

(a) Geometry can change the outcome. In [1], imbalance delays convergence under orthogonal features. Proposition 2 gives a full-rank geometry with unequal prototype norms and imbalanced queries where training from zero drives rare-class matching attention to zero and the loss to a positive limit, although both classes appear in every context. Changing only GG to II restores zero-loss convergence. The update S+=S−ηGHGS^{+}=S-\eta GHG explains this difference through coupling between query classes. Theorems 2-3 provide complementary convergence guarantees for near-orthogonal and transitive geometries under their stated correlation bounds.

(b) Task shape can change routing. Proposition S2 compares anisotropic and isotropic priors with the same Gram matrix, trace, initial loss, and step size. Under anisotropy, a distractor outranks the matching prototype after one step while loss decreases. The isotropic trajectory never exhibits this ordering. Additional population simulations at η=1/348\eta=1/348 show that the reversal persists through step 19,263, while loss falls from 2.5 to approximately 0.0804. Proposition S2 proves the initial reversal, and the longer trajectory is numerical. Proposition S1 shows that a scalar learning-rate adjustment cannot reproduce the anisotropic first-step attention pattern.

(c) Symmetry yields sharp training costs. Under stronger symmetry and exactly balanced contexts, we prove matching upper and lower bounds for both stages. For K≥3K\ge3, routing takes Θ(K2/(ησf2γ2))\Theta(K^2/(\eta\sigma_f^2\gamma^2)) steps along the analyzed GD trajectory. The accuracy thresholds and crossover distinguish entering refinement from refinement dominating the training cost. A target can lie in refinement while routing still determines the training budget.

Task nonlinearity alone adds no mechanism on linearly independent prototypes, where every task admits a label-equivalent linear realization. We revised the framing accordingly and added the comparison in Table 3 of Appendix B.

A3. Factorized Optimization

Our Q=WK⊤WQQ=W_K^\top W_Q is trained directly from zero, as in [1]. Training the factors separately changes the optimization.

Writing D=∇QLD=\nabla_Q L, a simultaneous gradient step gives Q+=Q−η(WK⊤WKD+DWQ⊤WQ)+O(η2)Q^{+}=Q-\eta(W_K^\top W_KD+DW_Q^\top W_Q)+O(\eta^2). The factor Gram matrices are not determined by QQ, so the dynamics no longer close in S=V⊤QVS=V^\top QV. Moreover, initializing both factors at zero makes both gradients vanish.

The loss decomposition in Lemma 2 and score-gradient identity in Eq. 9 remain valid for the same attention model under factorization. Extending the convergence guarantees additionally requires controlling factor initialization and conditioning.

The fully trainable two-layer Transformers in Appendix H.8 use separate key and query projections and random initialization. All six baseline runs at K=16,32K=16,32 exhibit a descriptive loss plateau before meeting the functional-routing criterion, and routing times increase with class count. These observations support qualitative persistence beyond the merged model. They do not establish the exact single-layer laws.

Revised manuscript passages are highlighted in blue. We would appreciate reconsideration of the evaluation in light of these clarifications and results.

[1] Yu Huang, Yuan Cheng, and Yingbin Liang. In-Context Convergence of Transformers. ICML 2024.

Finally, if our response resolves your concerns to a satisfactory level, we wonder if the reviewer could kindly consider raising the score of your evaluation. We thank the reviewer again for the helpful comments and suggestions for our work.

Reviewer_TJQ4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes gradient descent training of a single softmax attention layer for regression tasks on fixed correlated prototypes (discrete and defined by a vocabulary). Under several simplifying assumptions, the authors derive the class-score update dynamics that depends on the prototype geometry (Gram matrix) and the loss gradient with respect to the scores. They prove that under certain conditions, the loss is non-increasing, but also demonstrate that large class imbalance combined with a geometry violating self-similarity can make attention to vanish for rare classes. The paper carefully characterizes convergence of the correct class retrieval and incorrect-attention mass evolution under symmetry (equiangular prototypes, isotropic task contrast, etc.), identifying a routing plateau scaling as K^2 followed by an inverse-accuracy refinement stage. The population simulations verify predicted dynamics empirically. Additional experiments test training robustness to correlated geometry, random prompts and imbalance. Under the conditions used for theoretical analysis, the proper strategy is to retrieve the label, but not infer it from function values on other prototypes.

Critical strengths
  • The paper proposes a deep theoretical analysis of in-context regression with nonlinear task priors. It proposes several creative simplifying assumptions that make thorough theoretical analysis possible. The authors propose an exact (under symmetry) and interpretable description of the training dynamics. The analysis also identifies potential failure modes providing minimal demonstrations.
  • Obtained theoretical results are also carefully verified empirically. Population simulations verify the anticipated trajectories and the scaling laws. On top of this, token-level SGD reproduces them and extends them outside the original assumptions. These experiments support the theory within its synthetic retrieval setting.
Critical weaknesses
  • My main concern is that while the paper sometimes makes broader claims about nonlinear in-context learning, the core setting under a given setup simply learns to retrieve a label of the query's prototype, but does not infer a function at an unseen input. Because all inputs, including the query, lie exactly on the prototypes (vocabulary), a task affects the model only through its values at those prototypes. And since the prototypes are linearly independent, any set of such values can also be produced by a linear function (as a special case). So every nonlinear task distribution has a linear equivalent that leads to exactly the same training dynamics. As a result, the nonlinearity matters for the dynamics only through the spread of outputs across prototypes. The authors acknowledge this, but the paper is nevertheless framed as a contribution to nonlinear in-context learning. This reduces the main advancement over prior work (Huang, 2024) to a study of non-orthogonal geometry, two-sided stage-wise bounds, and the accuracy phase diagram.
  • It is anticipated for theoretical results, but should still be mentioned that the strict simplifying assumptions (including a single softmax layer, no attention to labels, symmetry, etc.) make it difficult to reason about practical implications of these results for in-context learning, especially in deeper Transformer models.
Questions for authors
  • What does the original nonlinear framing establish beyond linear tasks on non-orthogonal features?
  • Have any of the paper's practical conclusions (a mechanism for ICL plateaus, their dependence on the number of classes, learning-rate policy, etc) been tested outside this model (with learned representations or deeper Transformers)?
2 comments

We thank the reviewer for identifying the framing and empirical scope that needed clarification.

A1. Nonlinear Task Scope

Task nonlinearity alone introduces no distinct dynamics on the prototype support. Since G≻0G\succ0, the linear function x↦x⊤VG−1Ffx\mapsto x^\top VG^{-1}F_f reproduces every task's prototype values. This replacement preserves the prototype-label distribution and training dynamics, although it need not preserve the original global Lipschitz bound.

We revised the abstract, Introduction, Section 2, related work, and conclusion to frame the paper around learning a shared retrieval rule, without claiming nonlinear function inference at unseen inputs.

A2. Core Contributions

Our contributions concern whether retrieval succeeds, how routing develops, and the cost of reaching a target accuracy. They also apply to label-equivalent linear priors.

(a) Geometry changes training outcomes. Proposition 2 gives a full-rank geometry with unequal prototype norms and imbalanced queries where training from zero drives rare-class matching attention to zero and the loss to a positive limit, despite available matching examples. Changing only GG to II restores zero-loss convergence. Theorems 2-3 give complementary convergence guarantees under their stated correlation bounds.

(b) Task shape changes routing. Proposition S2 compares anisotropic and isotropic priors with the same GG, step size, tr⁡Σf\operatorname{tr}\Sigma_f, and initial loss. Under anisotropy, a distractor outranks the matching prototype after one step while loss decreases. The isotropic trajectory never exhibits this ordering. This establishes a qualitative routing difference that scalar rescaling cannot reproduce.

(c) Symmetry yields sharp complexity. The two-stage mechanism is shared with Huang et al. [1]. Under stronger symmetry and exactly balanced contexts, our matching upper and lower bounds establish the K2K^2 routing bottleneck and inverse-accuracy refinement cost along the analyzed GD trajectory. The accuracy thresholds and crossover determine which stage controls the training budget. A target can lie in refinement while routing still dominates its cost.

The revised contribution summary and Appendix B distinguish these results from the shared mechanism.

A3. Beyond One Layer

We added 18 runs with fully trainable two-layer causal Transformers and three four-layer runs, all using SGD (Appendix H.8, Figure 13, Table 8). Input projections, attention and value maps, FFNs, and readout are trained jointly.

Plateaus and class count. All six baseline runs at K=16,32K=16,32 exhibit a descriptive loss plateau before meeting the functional-routing criterion, measured through output sensitivity to context labels. Across K∈{4,8,16,32}K\in\{4,8,16,32\}, per-seed log-log fits of routing time give a mean exponent of 1.491.49 with sample standard deviation 0.100.10. Class-count dependence therefore persists, while the measured exponent does not confirm the single-layer K2K^2 law.

Task scale and rate. At K=8K=8, quadrupling label variance at a fixed step size reduces median routing time from 950 to 450 updates, but increases the time to held-out half-MSE 0.010.01 from 2,200 to 3,600. Dividing the step size by four gives routing time 900, close to baseline. These results show that faster retrieval formation need not imply faster attainment of a fixed absolute accuracy. Label scaling changes both Transformer inputs and targets, so the exact trajectory-rescaling identity of the fixed-feature model does not apply.

Representations and depth. Representation measurements confirm that the features evolve, and coherent label interventions support selective use of the supplied matching labels. All three four-layer runs also learn selective retrieval at K=8K=8.

We incorporated these results into the abstract, experimental discussion, and limitations. They support qualitative persistence in the tested architectures with fixed raw prototypes and synthetic labels. They do not establish the exact single-layer laws in deeper networks or validate the mechanism on natural-language tasks. Revised manuscript passages are highlighted in blue.

[1] Yu Huang, Yuan Cheng, and Yingbin Liang. In-Context Convergence of Transformers. ICML 2024.

Finally, if our response resolves your concerns to a satisfactory level, we wonder if the reviewer could kindly consider raising the score of your evaluation. We thank the reviewer again for the helpful comments and suggestions for our work.

I would like to thank the authors for their thorough response.

The suggested reframing (how attention learns to retrieve labels) resolves one of concerns I had. I also appreciate new experiments with fully trainable multi-layer Transformers. From what I understand, they suggest that the main qualitative picture predicted by the theory was not an artifact of a simpler model (existence of a plateau in the training trajectory before retrieval emerges and longer training with more classes).

On the other hand, the observed Transformer results match the single-layer theory only qualitatively. And the novelty relative to Huang is still mainly limited to studying the role of correlated features, accuracy regimes and sharper bounds.

I maintain my positive assessment (weak accept), but I do not yet think that the results are significant enough to justify a higher score.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We thank the reviewers for their feedback. We summarize below our main contributions and revisions for the AC and reviewers, highlighting the distinctions from the prototype-matching mechanism shared with Huang et al.

  • Convergence versus failure. Non-orthogonal geometry can cause GD to permanently lose a rare class despite matching examples in every context. Changing only the geometry restores convergence (Proposition 2). Theorems 2–3 provide complementary conditions for global convergence.

  • Task shape changes routing. Anisotropy can make a distractor outrank the matching prototype while population loss decreases. This ordering difference cannot be removed by scalar learning-rate rescaling (Propositions S1–S2).

  • Sharp training costs. Under the stated symmetry and balanced-count assumptions, matching upper and lower bounds establish routing and refinement costs. The accuracy thresholds and crossover determine which stage dominates the training budget (Proposition 4 and Theorem 5).

We have focused the framing on learning retrieval over prototype support, clarified the linear-equivalence limitation and comparison with Huang et al., simplified the main theorem statements, and reorganized the appendix with a proof roadmap and explanations of the key arguments.

We also added experiments with fully trainable two- and four-layer Transformers on synthetic retrieval tasks (Appendix H.8). The two-layer results support delayed selective retrieval, class-count dependence, and learning-rate-policy effects with learned representations. The four-layer runs also learn selective retrieval. These experiments test the qualitative conclusions beyond the analyzed architecture.

We believe these results provide substantive insight into when retrieval is learned and what training it requires, and that the revisions and additional evidence support acceptance. We welcome further questions and would be happy to address any remaining concerns.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.