acceptodds
Under review as a conference paper at ICLR 2027

Stagewise Learning of Balanced Orthogonal Mixtures by Finite-Width Two-Head Transformers

Abstract

Understanding how attention learns relationships between tokens is central to interpreting Transformers. We study context–signal correspondence learning in balanced orthogonal mixtures. Prior work introduced a three-stage population gradient-flow protocol for this setting. Each input contains three tokens: a context, its matching classification signal, and a distractor. The prior analysis relies on cancellation identities specific to two mixture components (), and its finite-width guarantee uses a sufficient width that grows as the initialization scale decreases. Under the same training setting, we prove high-probability convergence of logistic risk to zero for every fixed number of mixture components in a finite-width two-head softmax Transformer. For fixed , bias, admissible initial readout-training duration, and confidence, the sufficient width is uniform over sufficiently small positive initialization scales and independent of the target risk. Our analysis combines joint Gram dynamics and relative perturbation estimates to control multidimensional attention growth, and handles asynchronous ReLU crossings along the actual random trajectory. We establish feature separation at the end of attention training. Subsequent readout training yields permanent activation of all ReLU gates and exact logistic-risk and max-margin readout asymptotics. For , we further quantify attention-probability gaps, including one-sided cubic lower bounds. Finally, successful learning does not require monotone improvement of the bias-augmented feature margin. At fixed admissible parameters, including a positive initialization scale, the probability of successful learning with an initial margin decrease remains bounded away from zero for all sufficiently large widths. With high probability, the margin nevertheless increases throughout the remaining attention-training interval after a delay that vanishes with width at fixed confidence.

Primary area:
learning theory
Keywords:
Transformers, Softmax attention, Training dynamics, Gradient flow, Feature learning.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ouizOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies context–signal correspondence learning in balanced orthogonal mixtures. Similar setting has been analyzed for the case with two mixture components (K = 2), and its finite width guarantee uses a sufficient width that grows as the initialization scale decreases. Under the same training setting, this paper shows high-probability convergence of logistic risk to zero for every fixed number K ≥ 2 of mixture components in a finite-width two-head softmax Transformer.

Critical strengths

The paper contains comprehensive numerical results.

Critical weaknesses
  1. The paper is very difficult to read. No proper discussion about the motivation for the setup. From the paper, it is difficult to understand the importance of the problem studied. The introduction directly jumps to the considered setup, without elaborating why this setup is important and how this simple setup is connected to the practice.
  2. The theorems are presented without proper discussion. It is very difficult to parse what these theorems mean. Not clear what is new and what challenges the authors have to face to extend K=2 to K≥ 2. No insights were provided.
Questions for authors
  1. Why this setup is important? How is this setup connected to practice? What insights can we obtain by studying this setup?
  2. What is the meaning of these theorems?
  3. What are the main challenges when extending K=2 to K≥ 2? What is the novelty in the proof?
2 comments

We appreciate your observation that motivation and theorem interpretation need clearer exposition. We will open with the example, explain the inherited model and protocol together, and state the insights before the technical machinery.

(1) Why this setting matters

How does training make one token relevant to another? In pronoun resolution or selecting information for a question, context identifies what matters. Our setting isolates this operation: each input has context, its matching classification signal, and a distractor.

For example, [c1,+v1,−v2][c_1,+v_1,-v_2] has label +1+1, whereas [c2,−v2,+v1][c_2,-v_2,+v_1] has label −1-1. Context identifies the relevant signal. Balance and orthogonality remove frequency and geometric-correlation effects, making the formation of this correspondence accessible to a complete training analysis.

The proof explains how the representation develops. Stage 1 aligns readouts with signal directions. Stage 2 freezes those actual readouts and trains key/query factors on conflict-conditioned population loss. Signal queries acquire context-key scores that give different context contributions in the two heads, creating separating features. Stage 3 freezes these features; remaining inactive gates return, and the readout converges in direction to the hard-margin classifier determined by the learned representation.

The sufficient width stays fixed as initialization shrinks, even though the attention-training interval grows logarithmically. The learned-feature margin determines the exact coefficient in asymptotic risk decay. The analysis also identifies a transient geometric decrease followed by sustained growth. These results explain what attention training changes and how the representation governs the classifier.

(2) What the four theorems mean

Theorem 4.1: At every fixed K≥2K\ge2, one finite-width Gaussian initialization learns through all three prescribed stages with high probability. For fixed bias, an admissible Stage 1 duration, and confidence, a single sufficient width works at each sufficiently small positive initialization scale. It also stays independent of the target risk.

Theorem 4.2: Features produced by attention determine the limiting classifier. If v∗v_* is the minimum-norm point of the labeled feature hull and γ=∥v∗∥\gamma=\|v_*\|, then W(t)/log⁡t→v∗/γ2W(t)/\log t\to v_*/\gamma^2, tR3(t)→γ−2tR_3(t)\to\gamma^{-2}, and tW˙(t)→v∗/γ2t\dot W(t)\to v_*/\gamma^2. Feature geometry determines the readout direction and asymptotic speed.

Theorem 4.3: For fixed K≥3K\ge3, the joint learning event yields signed endpoint attention comparisons. First-, second-, and third-order gaps resolve the token interactions behind separation.

Theorem 4.4: Successful learning can begin with a negative initial right derivative of the augmented feature margin on an event of positive probability. On the learning theorem's high-probability event, this margin grows after a width-dependent delay.

(3) What changes beyond two components

At the leading reference scores, context-attention probabilities for the two signal queries on a positive conflict satisfy p+,1∣2(0)−p+,1∣3(0)=2(K−2)z/(9K)+O(z2)p^{(0)}_{+,1|2}-p^{(0)}_{+,1|3}=2(K-2)z/(9K)+O(z^2). Here zz is the reference contrast. The first-order term vanishes at K=2K=2 and is present for K≥3K\ge3. The separation argument combines the distinct query contributions. The top Gram-growth space also has dimension K(K−1)/2K(K-1)/2 per head, so fluctuations occupy several fastest-growing directions.

The full joint-Gram analysis retains both self-Grams. At fixed KK and confidence, the Stage 1 readout error is O(ω2/m)O(\omega^2/\sqrt m). The full Gram error is O(ω2eΛKt/m)O(\omega^2e^{\Lambda_Kt}/\sqrt m) over Stage 2, where ΛK=2μ/[9(K−1)]\Lambda_K=2\mu/[9(K-1)] and μ\mu is the reference alignment. At reference contrast c0c_0, this gives endpoint error O(c0/m)O(c_0/\sqrt m) and initialization-uniform width. Directed gate-normal bounds handle asynchronous returns and connect separation to permanent activation. We will organize the proof overview around these obstacles and estimates.

(4) Evidence supporting the mechanism

Across 200 trajectories with paired seeds, two widths, and five initialization scales, normalized Stage 1 errors have medians 0.078 and 0.074. Every relative Gram-error maximum on the 257-point grid occurs at launch; the endpoint residual after linear transport is at most 0.285% of the transported initial error. A complete K=3K=3, width-4096 certificate validates the path from initialization to invariant-tail entry and vanishing risk. We will present these findings in the numerical section. The supplement contains per-seed data, technical notes, and the runnable certificate.

Thanks for the clarifications. However, the issues still remain and the paper still needs significant changes. Hence, I maintain my rating.

Reviewer_6R52OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the three-stage population gradient-flow protocol of Yang et al. (2025) for a two-head softmax Transformer with factorized key/query matrices on balanced orthogonal mixtures. The main result extends the learning guarantee from two components to every fixed K≥2K\ge2. With high probability over Gaussian initialization, one finite-width network passes through all three stages and achieves logistic risk converging to zero. The sufficient width is uniform over sufficiently small positive initialization scales. Further results characterize readout asymptotics, attention-probability gaps, and initial decreases of a bias-augmented feature margin on successful trajectories. Numerical experiments examine these mechanisms, including at contrasts beyond the displayed sufficient theoretical regime.

Critical strengths

1. A meaningful extension of the finite-width learning guarantee.

Yang et al. (2025) already provide a finite-width analysis for K=2K=2. This paper extends learning to every fixed K≥2K\ge2 and removes the dependence of the sufficient width on the initialization scale ω\omega, within the stated small-initialization regime. The latter permits initialization to shrink without requiring the network width to increase.

2. The analysis connects all stages along an actual random trajectory.

The proof controls the sampled network without imposing exact symmetry or resetting parameters between stages. It handles a leading Gram eigenspace of dimension K(K−1)/2K(K-1)/2 per head and asynchronous ReLU transitions. In particular, the Stage 1 interface estimate ∥W1−Wˉ∥=O(ω2ηm)\|W_1-\bar W\|=O(\omega^2\eta_m), together with the relative Gram estimate in (26)–(27), controls errors over an attention-training interval that grows as log⁡(1/ω)\log(1/\omega). The subsequent permanent-activation argument connects the learned features to risk convergence and the readout asymptotics.

Critical weaknesses

The extension retains restrictive assumptions, and its scaling with KK remains unclear.

The paper extends the learning guarantee to every fixed KK, while retaining most of the data and training assumptions of Yang et al. (2025): balanced orthogonal components, three-token inputs, and a prescribed three-stage population gradient flow with attention trained only on conflicting samples. These assumptions are clearly stated and help isolate correspondence learning. However, the extension remains within the same controlled framework, and the paper provides limited evidence about whether the learning mechanism persists under more general data distributions or training procedures.

In addition, although Appendix G explains the selection and dependencies of the constants, their dependence on KK remains partly implicit. In particular, the paper does not provide explicit bounds on admissible choices of c0c_0 and ρK\rho_K, or on the Stage 3 entry time HKH_K, as KK grows. Thus, the theorem establishes learning for each fixed number of components, but leaves unclear how the sufficient network width and training time scale with that number.

Questions for authors
  1. Assumptions and contrast regime. Which assumptions are essential to the learning mechanism, and which mainly simplify the analysis? Also, is the small sufficient c0c_0 mainly due to conservative estimates, or would extending the guarantee to c0=0.02c_0=0.02, as used in the experiments, require additional arguments?

  2. Dependence on KK. As the number of components KK increases, how do the sufficient network width and training time change? Could the authors give rough estimates by making the dependence of c0c_0, ρK\rho_K, and HKH_K on KK explicit?

  3. Connections to related work. How does the analysis in Appendix C compare with the small-initialization analysis of Li et al. [1]? Which parts use similar arguments, and which require additional work to handle softmax attention or the transitions between training stages? Please also clarify how the assumptions and learning guarantees differ from the mixture-of-Transformers result in [2].

[1] Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning. ICLR, 2021.

[2] Hongbo Li, Qinhang Wu, Sen Lin, Yingbin Liang, and Ness B. Shroff. Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems. arXiv:2510.27004, 2025.

1 comment

We appreciate your questions about the role of assumptions and quantitative dependence on KK. We will clarify which assumptions supply the proof tools and present the explicit dependence alongside the width bound.

(1) Assumptions and contrast regime

Context-dependent relevance and factorized key/query scores define the mechanism. Three tokens—context, relevant signal, distractor—isolate the learning operation. Balance and orthogonality provide reference symmetries, an explicit growth spectrum, and a separating geometry. Zero readouts and a shared positive bias give initial alignment and gate transitions. Prescribed stages define the trajectory, with attention trained on conflicts where signal signs differ and context identifies the label.

Relative Gram propagation requires a reference growth bound, a quadratic remainder bound, and control of readout forcing. Directed gate crossing requires a common strict normal-velocity sign across adjacent activation patterns. The invariant-tail argument applies to frozen features with a positive separation and an unsigned gate-normal reserve.

The core budget covers Stage 1 accuracy, Gram propagation, separation, gate returns, and invariant-tail entry. Coordinate positivity and higher-order attention comparisons have additional tolerances. Write ηm=8log⁡[(64K2+16K)/δ]/m\eta_m=\sqrt{8\log[(64K^2+16K)/\delta]/m}. Resolving cubic attention gaps requires Cscc0ηm≤ΔK(c0)C_{\rm sc}c_0\eta_m\le\Delta_K(c_0), while core learning transfers a positive separating margin and the tail-entry reserves. The supplement defines the full tolerance ρlearn\rho_{\rm learn}.

Small contrast bounds control these remainders and transfer errors. For larger contrasts such as c0=0.02c_0=0.02, we propose a trajectory-based verification route: enclose the reference flow, check endpoint margin, control crossings, and verify invariant-tail entry. We have implemented this route for K=3K=3, m=4096m=4096, b=1/2b=1/2, μ=0.13\mu=0.13, c0=0.1c_0=0.1, and ω2=1/16000\omega^2=1/16000.

The certificate carries the actual Stage 1 readout into Stage 2, stops at the reference first-hit clock T∗=558.972…T_*=558.972\ldots, and certifies all 12 returning gates. At a local Stage 3 time 102010^{20}, the validated trajectory enters an arrival box certified to lie in the invariant-tail region, with γ2>3.4279×10−4\gamma^2>3.4279\times10^{-4}, κ>7.2046×10−3\kappa>7.2046\times10^{-3}, and Lyapunov functional below 0.011782 times its entry threshold. Validated continuation and the invariant-tail lemma give permanent activation from time 2048 and exact risk/readout limits. The package includes the reproducible certificate and 15,669 record checks. We will present this instance beside the learning guarantee.

(2) Dependence on K

At fixed b,μb,\mu, the growth rate is ΛK=2μ/[9(K−1)]\Lambda_K=2\mu/[9(K-1)]. The reference-growth, bootstrap, and gated-separation bounds admit an explicit choice cref=a(b,μ)K−3c_{\rm ref}=a(b,\mu)K^{-3}. The full core contrast also incorporates closed-tube geometry and directed-return thresholds. The reference clock lies between ΛK−1log⁡(c0/ω2)\Lambda_K^{-1}\log(c_0/\omega^2) and ΛK−1log⁡(4c0/ω2)\Lambda_K^{-1}\log(4c_0/\omega^2).

The sufficient core width is m≥max⁡{4K,8ρlearn−2log⁡[(64K2+16K)/δ]}m\ge\max\{4K,8\rho_{\rm learn}^{-2}\log[(64K^2+16K)/\delta]\}. The geometric transfer radius is c02/(139968K2)c_0^2/(139968K^2); its width contribution is O(K8(1+Cw)2c0−2log⁡(K/δ))O(K^8(1+C_w)^2c_0^{-2}\log(K/\delta)), where CwC_w is the Stage 1 interface constant. The complete minimum also includes directed-return and tail-entry radii. From a strict all-active reference state of norm at most BretB_{\rm ret}, an explicit additional tail time UU satisfies log⁡(1+U)=O(K10(1+Bret)3/c06)\log(1+U)=O(K^{10}(1+B_{\rm ret})^3/c_0^6). The supplement derives these expressions and assembles the common clock over the reference family. We will state these dependencies beside the width bound.

(3) Connections to cited works

Li, Luo, and Lyu (2021) use matrix dynamics, linearization, and semigroup estimates to analyze small-initialization escape. Their Theorem 5.3 assumes a simple top eigenvalue. Our joint Gram retains both self-Grams and the key/query cross-block; its top growth space has dimension K(K−1)/2K(K-1)/2 per head. We control arbitrary perturbations there together with forcing from the actual preceding readout stage. The sharp interface ∥W1−Wˉ∥=O(ω2ηm)\|W^1-\bar W\|=O(\omega^2\eta_m) closes the relative-error budget over T∗=O(log⁡(1/ω))T_*=O(\log(1/\omega)), yielding initialization-uniform width.

Li et al. (2025) study learned routing and transformer-expert specialization, with noise tokens, direct attention-score updates, and a router trained throughout their three stages. Their guarantee uses Ω(Nlog⁡N)\Omega(N\log N) experts for NN mixture components. Our result follows factorized key/query training within two heads and characterizes the learned-feature hard-margin limit. We will expand the related-work comparison around these differences in learned structure, parameterization, and guarantees.

Reviewer_Se5gOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work considers a problem of learning mixtures of linear classification, where each sample consists of 3 tokens : a context, a matching signal that determines the label, and a distracting signal. It studies the three-stage population gradient-flow training protocol of a two-head softmax attention model with mixtures of K≥2K\geq 2 components and proves convergence of the logistic loss to 0 for some finite model width.

Critical strengths

The problem of learning mixtures of linear classifications can be an illustrative setup of how attention learns connections between tokens both from the training dynamics perspective and from the found solution interpretation perspective.

Critical weaknesses

The set up can be reduced to memorizing labels of the finite dataset, with the task of learning labels for 4K(K−1)4K(K-1) deterministic triplets of vectors. Since it is known that large enough MLP is capable of doing that, the convergence of loss to 0 by itself is less interesting ; and no comparison between the considered attention model width and a simpler architecture size is made.

The presentation is very hard to follow. Concepts and variables are used before they are introduced (for example, "reference" and "wrong-gate" in line 199, W1W^1 in line 241, notation Zh(X)Z_h(X) or Zh,iZ_{h, i} is inconsistent). No interpretation of the theoretical results is given.

Numerical evidence show that the training with the staged algorithm converges, but do not clarify the intuition behind the theoretical claims. There is no evidence that training simultaneously the attention and readouts with GD would result in the similar stages duration as the considered staged algorithm. No practical insights are drawn from the theoretical results.

Questions for authors

For clarification : could the authors clarify how the reference model is defined and trained ? How does the training trajectory of the reference look like ? What is the solution to which the model converges after training ? is it the one described in (Yang et al. 2025) ?

What do the following phrases mean ?

  • l.199 : "downward wrong-gate crossing"
  • l.245 : "classical factor"
  • l.673 : "fragile set/fragile bit"
3 comments

We appreciate your request to explain the learning mechanism and reference trajectory more clearly. We will organize the exposition around how training creates context-dependent relevance, what the reference means, and what the results show.

(1) Why attention differs from an MLP

How does training make one token relevant to another? Consider [c1,+v1,−v2][c_1,+v_1,-v_2] with label +1+1 and [c2,−v2,+v1][c_2,-v_2,+v_1] with label −1-1. The signal vectors are the same; context determines which signal carries the label. This isolates an elementary operation used in attention-based language models: learning context-dependent relevance. Balance and orthogonality let us trace its development without frequency or geometric-correlation effects.

The contribution is a complete training analysis connecting random initialization, learned token interactions, feature separation, and the final classifier. An MLP applies affine maps and nonlinearities to concatenated coordinates. Attention learns bilinear scores xjTWK,hTWQ,hxℓx_j^T W_{K,h}^T W_{Q,h}x_\ell, normalizes them over keys, and uses the resulting input-dependent weights to aggregate tokens. We identify how signal queries' context-key scores grow and create different context contributions in the two heads, producing separation.

The factor width mm is the key/query representation dimension; the number of heads stays two. The model has 4md+2d4md+2d trained parameters. For comparison, a width-ww two-layer ReLU MLP on three concatenated dd-dimensional tokens, with hidden biases and a linear output, has w(3d+2)w(3d+2) parameters. In our theorem, mm controls concentration of joint Grams around a deterministic reference. For fixed KK, bias, an admissible Stage 1 duration, and confidence, one sufficient width works at every sufficiently small positive initialization scale. Smaller initialization lengthens the attention phase as O(log⁡(1/ω))O(\log(1/\omega)) while this width remains fixed. This identifies how finite-width fluctuations can be controlled throughout representation learning. We will explain this role of width alongside the architecture comparison.

(2) The reference, its trajectory, and the limiting solution

The reference has three explicitly defined phases. Stage 1 starts from zero readouts with zero scores, hence uniform attention, and trains the readouts on the full population until T1T_1. Its endpoint is Wˉ=(μcΣ+βvΣ,μcΣ−βvΣ)\bar W=(\mu c_\Sigma+\beta v_\Sigma,\mu c_\Sigma-\beta v_\Sigma), where cΣ=∑icic_\Sigma=\sum_i c_i, vΣ=∑iviv_\Sigma=\sum_i v_i, β=b/2+2μ\beta=b/2+2\mu, and T1T_1 determines μ\mu.

Stage 2 freezes Wˉ\bar W and evolves the closed joint-Gram ODE from Cˉh(0)=ω2I4K\bar C_h(0)=\omega^2 I_{4K} under conflict-conditioned population loss. Its context-signal contrast grows to c0c_0; the first hit defines T∗(ω)T_*(\omega). The actual network freezes its own Stage 1 endpoint W1W^1 and trains for this same reference duration. Stage 3 freezes the learned features and resumes full-population readout training. Inactive gates return, all gates become permanently active, and risk tends to zero.

The limiting direction is determined by the learned features: if aX=y(Z+(X),−Z−(X))a_X=y(Z_+(X),-Z_-(X)), v∗v_* is the minimum-norm point of their convex hull, and γ=∥v∗∥\gamma=\|v_*\|, then W(t)/log⁡t→v∗/γ2W(t)/\log t\to v_*/\gamma^2 and tR3(t)→γ−2tR_3(t)\to\gamma^{-2}. We inherit Yang et al.'s model and protocol; their finite-width guarantee covers K=2K=2. Our analysis gives every fixed K≥2K\ge2, initialization-uniform sufficient width, and these exact learned-feature asymptotics. The further results resolve signed attention gaps and transient changes in the augmented feature margin. We will present the reference construction and comparison together.

(3) Terminology and notation

We will replace "downward wrong-gate crossing" with "the opposite-label head's gate switches from active to inactive." "Classical factor trajectory" means a continuously differentiable key/query trajectory satisfying the ODE pointwise; Stage 2 has fixed gate signs throughout its prescribed interval. "Fragile set/bit" will become "switching gates/activation indicators." We will define W1W^1 at the stage interface and write Zh,i=Zh(Xi)Z_{h,i}=Z_h(X_i) at first use.

(4) Evidence and interpretation

Across 200 trajectories, normalized Stage 1 error has medians 0.078 and 0.074 at widths 4096 and 262144. Every relative Gram-error maximum on the 257-point grid occurs at launch. A validated K=3K=3, width-4096 trajectory connects initialization, all gate events, and invariant-tail entry, establishing permanent activation and vanishing risk. The accompanying supplement gives the data, derivations, and runnable certificate.

Yang et al.'s Figure 1 reports that readout alignment precedes attention movement in simultaneous training. We will explain how this observation motivates the prescribed protocol, and organize the numerical section around the stage-interface and relative-error estimates above.

Thank you for answering my questions. There are things that are still unclear:

(1) It is still not clear to me, how the factor width mm of an attention model (even the reference) is related to the width of an MLP with comparable performance on this task.

(2) Could you clarify the order of T1T_1 and T∗(ω)T_*(\omega) in terms of problem dimension dd?

(3) Thank you, I believe it will be beneficial to the clarity of the work.

(4) The proposed reorganization of the numerical section still leaves the question of practical relevance of the findings open. Would you observe some similar staged behavior in a more realistic setup? How could your findings be applied there?

Since my initial concerns were not fully addressed I keep my score.

Thank you for the follow-up. We now give an explicit width comparison, the dimension dependence of the clocks, and a concrete way to use the mechanism in richer training settings.

(1) Width and comparable task performance

A two-layer ReLU MLP with 2K2K hidden units can classify every input type and every token permutation correctly. Let s=∑ℓ=13xℓs=\sum_{\ell=1}^3x_\ell, ui=⟨ci,s⟩u_i=\langle c_i,s\rangle, ri=⟨vi,s⟩r_i=\langle v_i,s\rangle, and [a]+=max⁡{a,0}[a]_+=\max\{a,0\}. An explicit realization is g(X)=∑i=1K([ui+ri−1]+−[ui−ri−1]+)g(X)=\sum_{i=1}^K([u_i+r_i-1]_+-[u_i-r_i-1]_+). For X=[ck,yvk,ϵvj]X=[c_k,yv_k,\epsilon v_j], only the kk-th summand is nonzero, and it equals yy. Scaling the output by AA gives logistic risk log⁡(1+e−A)\log(1+e^{-A}). The sum over tokens is absorbed into the hidden-layer weights on the concatenated input. Thus this gives a concrete width-2K2K representation with zero classification error and arbitrarily small logistic risk.

For the reference attention model, m=4Km=4K realizes the isotropic joint Gram ω2I4K\omega^2I_{4K} exactly. The prescribed reference dynamics then produce separating features and vanishing risk under the paper's parameter choices. For the Gaussian-initialized network, the sufficient width is m≥max⁡{4K,8ρK−2log⁡[(64K2+16K)/δ]}m\ge\max\{4K,8\rho_K^{-2}\log[(64K^2+16K)/\delta]\}. Here width controls fluctuations around the reference while the head count remains two.

This comparison locates the contribution in learning dynamics: for every fixed KK, we prove how random key/query factors acquire context-selective interactions, how the resulting features become separable, and which classifier the readout approaches. The sharp Stage 1 interface and relative Gram bound allow the same sufficient width as initialization shrinks and the attention-training horizon grows. We will add the explicit MLP construction and distinguish representation width from the width used to control the random trajectory.

(2) Dependence of T1T_1 and T∗T_* on dd

The ambient dimension dd and the intrinsic dimension 2K2K play different roles. At fixed KK, the dynamics stay in the span of the context and signal vectors, so both clocks are independent of ambient dd. Writing the admissible Stage 1 duration through its alignment parameter μ\mu gives T1=K[b/2+(e2b−1)/4+4μ+(4e2b/5)(e5μ−1)]T_1=K[b/2+(e^{2b}-1)/4+4\mu+(4e^{2b}/5)(e^{5\mu}-1)]. Thus T1=Θ(K)T_1=\Theta(K) at fixed bb and μ\mu. Also, 9(K−1)2μlog⁡c0ω2≤T∗(ω)≤9(K−1)2μlog⁡4c0ω2\frac{9(K-1)}{2\mu}\log\frac{c_0}{\omega^2}\le T_*(\omega)\le\frac{9(K-1)}{2\mu}\log\frac{4c_0}{\omega^2}. If d=2Kd=2K denotes intrinsic dimension, these become T1=Θ(d)T_1=\Theta(d) and T∗=Θ(dlog⁡(c0/ω2))T_*=\Theta(d\log(c_0/\omega^2)) at fixed bb, μ\mu and admissible parameters, retaining the dependence on c0c_0 in the logarithm. Adding orthogonal ambient coordinates changes neither clock. We will state this directly beside the stage definitions.

(3) Exposition

We will implement the terminology and notation changes, and introduce the reference, actual stage endpoints, and learned-feature limit before using them in the theorem discussion.

(4) More realistic settings and practical use

A concrete image-based counterpart appears in Yang et al. (2025), Appendix A.5 and Figure 8: MNIST digit-1 and digit-2 images replace the classification signals, with random context vectors; they report learning under both joint training and three-stage training. Their Figure 1 resolves the readout-first, attention-later ordering under joint GD on the synthetic task. We will discuss these experiments together with the mechanism our results explain.

For jointly trained models with image or text features, the mechanism suggests a concrete diagnostic: track signal/readout alignment, attention gaps between context-relevant and distractor tokens, and loss on examples that require context. This tests whether improved context-dependent prediction accompanies selective token interactions after initial readout alignment. Varying initialization at fixed width then tests how attention onset shifts as the initial scores shrink. Our theory supplies a quantitative account of this sequence in the controlled model, including the logarithmic attention clock and width-uniform error control. We will make these observables and their connection to model interpretability explicit in the revised discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.