acceptodds
Under review as a conference paper at ICLR 2027

How Multi-Head Transformers Specialize for In-Context Learning: A Theoretical Analysis of Training Dynamics and Generalization

Abstract

In-context learning (ICL) enables transformers to infer task-specific mappings from demonstrations in the prompt. A theoretical understanding of how multiple attention heads learn to specialize during training remains limited. We study the training dynamics and generalization of a multi-head softmax transformer on compositional ICL tasks, where each input consists of multiple latent components that are independently mapped to corresponding output components. We jointly analyze the training of attention and value parameters, and show that their coupled gradient-based dynamics amplify weak initial differences between heads into complementary specialization: each head learns to match and retrieve information according to a different latent component. We establish finite-step convergence and show that the learned model generalizes to unseen combinations of component mappings and to out-of-domain pattern families. Our theoretical insights are validated on synthetic and real experiments.

Primary area:
learning theory
Keywords:
In-context learning, Transformers, Multi-head attention, Training dynamics, Feature learning, Gradient-based optimization, Softmax attention, Sample complexity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_EpHmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a theoretical understanding of how attention heads in multi-head softmax Transformers develop specialized roles during training on compositional in-context learning (ICL) tasks. The authors further show how the learned model generalizes to unseen combinations of component mappings and out-of-domain pattern families.

Critical strengths
  1. The paper's presentation of the task formulation, introduction and comparison of related work, and theoretical results to be very detailed and clear, making the paper highly accessible to readers.

  2. The theoretical results are solid and comprehensive under the considered setting.

  3. The research motivation is meaningful, and the theoretical modeling of the problem is well-grounded.

Critical weaknesses
  1. The paper's formulation of compositional tasks appears somewhat restrictive. Specifically, the different latent components are completely independent of each other, and the overall output is simply obtained by concatenating the outputs of their respective mappings.

  2. Following the first point, the finding that "the required context length scales only linearly in the number of patterns..." appear to be closely related to the highly separable structure of the two components in the task formulation. The authors may need to further discuss this connection and clarify to what extent these conclusions depend on the independence of the components.

  3. I am particularly curious about the extent to which the initialization scheme in Eq (13) affects the theoretical conclusions and numerical experiments. Although the authors state that the strength of this initialization can be controlled, the initialization itself introduces a non-negligible prior preference for the two attention heads.

    • Specifically, each head is initialized to attend almost exclusively to information from a particular input component. This may create substantial differences in attention scores and gradient signals associated with the two components from the very beginning of training. The authors could further discuss the influence of this initialization on the emergence of head specialization.

    • Additionally, if possible, it would be helpful to clarify whether attention heads can still develop specialized roles under initialization schemes without such predefined preferences, such as random initialization or other unbiased diagonal initialization schemes.

  4. There are several missing references in the manuscript. Specifically, some equation references appear to be missing in Proposition 1 (line 323), lines 415–416 and line 421.

Questions for authors

see Weakness.

4 comments
AuthorsOpenReview ↗

Response to Weakness 1

We thank the reviewer for this comment. We would like to clarify the input representation, motivate our modeling assumptions, and present new experimental results addressing the reviewer's concern.

  • The input features are not separated in the observed coordinates. Although the two latent input components are sampled independently, their representations need not occupy separate input coordinates, since the orthonormal matrix DxD_x in Eq. (7) can mix them.

  • Independent feature mappings provide a tractable model of compositional tasks. This assumption allows us to theoretically characterize how gradient descent induces complementary specialization across attention heads. It is motivated by tasks in which different latent features contribute differently to the output, such as verb stems and grammatical tense in translation or object shape and color in image classification. Similar compositional structures have been studied in prior work (Lake and Baroni, 2018). We acknowledge that our model is a simplification of real-world feature interactions.

  • New experiments demonstrate robustness to coupled output channels. To investigate specialization beyond independent output mappings in synthetic data, we introduce a coupling parameter λo\lambda_o such that each output channel depends on both input components:

\begin{bmatrix} g_\mu^\tau(\mu)+\lambda_o g_e^\tau(e)\\\\ g_e^\tau(e)+\lambda_o g_\mu^\tau(\mu) \end{bmatrix}.$$ We evaluate $\lambda_o \in \\{0,0.1,0.2,0.3,0.4,0.5\\}$ using a two-head Transformer and report results averaged over three random seeds. The new Figure 8 (Appendix A.3) shows that complementary head specialization persists across all tested coupling strengths. Specifically, Head 1 continues to attend preferentially to matching $\mu$ patterns, while Head 2 attends preferentially to matching $e$ patterns, including on unseen tasks. [See Figure 8.](https://imglink.cc/cdn/_1hmZVpPjq.png) These findings demonstrate empirical robustness to the considered output-channel coupling, while extending the theoretical guarantees to more general feature interactions remains future work. References B. Lake, M. Baroni, Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks, 2018
AuthorsOpenReview ↗

Response to Weakness 2

We thank the reviewer for this insightful comment.

  • We agree that the separable task structure is essential to the context-length improvement established in our analysis. In particular, the independence of the two components allows the output to be recovered through separate component-wise retrieval, rather than requiring a context example that matches the query in both components simultaneously. However, this structure alone does not guarantee the improved context-length scaling. The model must also learn from training data to exploit it.

  • Under our uniform sampling setting, a context example matches both components of the query with probability (1/M21/M^2), whereas matching either individual component occurs with probability (1/M1/M). Consequently, a joint-pattern matching mechanism requires a context length of order (O~(M2)\tilde O(M^2)), as established for single-head models in (Li et al, 2024a), while a component-wise retrieval mechanism can recover the two output components from different context examples with only (O~(M)\tilde O(M)) context length.

  • Our theoretical contribution is to establish how this more efficient retrieval mechanism emerges through training of multi-head attention. Specifically, we show that the coupled gradient dynamics of the attention and value parameters lead to complementary head specialization, where each head learns to match one input component and retrieve its corresponding output component. Thus, while the compositional structure makes the context-length improvement possible, the learned specialization enables the Transformer to realize this advantage.

We have clarified in the revision that the (O~(M)\tilde O(M)) context-length guarantee relies on the independent component structure and the resulting component-wise matching probabilities. For tasks with dependent or nonseparable components, these probabilities and the corresponding retrieval mechanism may change, and our current theoretical guarantee does not directly extend to such settings. Specifically, we added the following in the Conclusion of the revised version:

“ Focusing on compositional tasks, this paper shows how learned complementary head specialization enables efficient component-wise retrieval, achieving improved context-length scaling compared with joint-pattern matching in single-head models.

Our theoretical guarantees rely on the separability of the component mappings, and extending the analysis to more general, interdependent task structures remains an important direction for future work.”

AuthorsOpenReview ↗

Response to Weakness 3

  • This is a valid point. We agree that equation (12) introduces an initial preference for different input components. This initialization is considered to break the symmetry between attention heads so that we can investigate how their preferences are amplified during joint training. However, the theoretical guarantees do not require the exact structured initialization in Eq. (12).

  • Our result still holds when the initialization contains large Gaussian noise. We show in corollary 1 that our theoretical results still hold when the initialization contains additional large Gaussian noise. Specifically, we initialize the attention matrices as W‾h(0)=Wh(0)+W~h\overline{W}_h(0) = W_h(0) + \widetilde{W}_h where Wh(0)W_h(0) is the same structured initialization matrix in (12) and W~h\widetilde{W}_h is the Gaussian noise matrix with entries i.i.d. From N(0,σ2/dx)\mathcal N(0,\sigma^2/d_x). Importantly, this condition does not require the Gaussian perturbation to be small in Frobenius norm relative to the structured initialization. Specifically, we show that for a fixed noise level in the noise matrix, as long as θ\theta in (12) is not too small, greater than the lower bound in (25), then our Theorem 1 still holds with minor changes in some constants that do not affect the order-wise result. For example, when \theta chooses the lower bound in (25), ∥W~h∥F≥∥Wh(0)∥F\|\widetilde{W}_h\|_F \ge \|W_h(0)\|_F scales as dxM32log⁡(M/δ)\frac{d_x}{M^{\frac{3}{2}}\sqrt{\log(M/\delta)}}. Thus, when dx≥M32+ϵd_x \geq M^{\frac{3}{2}+\epsilon} for any ϵ>0\epsilon>0, the Gaussian perturbation can have a substantially larger Frobenius norm than the structured initialization Wh(0)W_h(0).

  • The initial preference is needed for theoretical analyses. Similar symmetry-breaking initializations have been considered in theoretical analyses of head specialization. For example in Chen et al. (NeurIPS 2024, Assumption 3.3), it is assumed that each attention head initially prefers a specific relative position and then the paper analyzes how training reinforces these preferences.

  • Moreover, our initialization does not assume that heads are already specialized. In fact, the initial softmax attention on context tokens with the same pattern as the query for each head is close to uniform (1/M1/M). For example, for M=20M=20 and θ=0.1\theta = 0.1, the softmax score on μ\mu-aligned tokens of head 1 is approximately 5.55.5\\% while uniform attention would be 55\\%. Furthermore, value parameters are initialized at zero and output of the heads are the same for both channels initially. Our analysis shows how training amplifies these weak attention preferences, achieves value specialization and drives attention towards concentration on the relevant context examples.

  • Finally, our experiments with purely Gaussian random initialization, i.e. θ=0\theta=0 investigate a setting beyond the assumptions of our theoretical analysis. The observed complementary specialization suggests that a prescribed initial component preference is not essential for specialization to emerge empirically. Indeed, a realization of purely Gaussian random initialization may introduce small asymmetries between attention heads, which may be amplified during training into complementary specialization. Establishing convergence from arbitrary random initialization remains outside the scope of our current theoretical guarantees.

References

Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Unveiling induction heads: Provable training dynamics and feature learning in transformers. NeurIPS, 2024

AuthorsOpenReview ↗

Response to Weakness 4

Thank you for pointing out the missing references. They are all corrected in the new version.

Reviewer_Be3xOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how different attention heads develop distinct roles during training and how this specialization supports generalization in in-context learning (ICL). The authors consider compositional tasks in which each input contains two latent components, with an independent input–output mapping applied to each component. The theoretical analysis focuses on a one-layer, two-head softmax attention model in which the attention parameters and structured value parameters are trained jointly.

The main result is that weak component-wise preferences introduced at initialization are amplified during training, leading to complementary specialization: each head learns to attend to examples matching a different input component and retrieve the corresponding output component. The paper characterizes this process through attention concentration, component-wise attention margins, and specialization of the value parameters. Under the stated assumptions, it establishes finite-step convergence and generalization to unseen combinations of component mappings. In particular, the training tasks need to cover individual component-wise input–output correspondences rather than every possible combination of mappings. The analysis also gives a context-length requirement that scales linearly with the number of patterns per component, up to logarithmic factors.

The analysis is further extended to structured OOD pattern shifts, and experiments on synthetic data and on CIFAR-10/DTD image features show trends consistent with the theory.

Critical strengths
  1. A concrete account of how head specialization develops during training. The paper analyzes the coupled evolution of attention and structured value parameters, describing specialization both through attention patterns and at the parameter level. Explaining how initial component-wise preferences are amplified and connected to each head's output-retrieval role is a valuable contribution to understanding multi-head ICL.

  2. A meaningful connection between specialization, generalization, and context efficiency. The analysis links component-wise specialization to generalization across unseen task combinations and quantifies the required context length. Extending the analysis to structured OOD shifts further strengthens the contribution. The synthetic experiments support the predicted trends, while the image-feature experiments provide additional evidence of head specialization.

Critical weaknesses
  1. The conditions and probabilistic scope of the generalization guarantees need clarification. A finite context may contain no example sharing a particular component with the query. The statement that the guarantee holds for "all prompts" therefore needs to be reconciled with the context-coverage conditions used in the proof. It would also help to state the appendix's query-balanced mini-batch sampling assumption explicitly in the main theorem. These clarifications matter for understanding precisely which training and test settings the guarantees cover.

  2. The scope of the results depends on structured initialization and parameterization. The theory uses an initialization aligned with the component structure and a restricted value parameterization. Explaining the role of these assumptions more explicitly would clarify the distinction between the mechanism established by the analysis and behavior that might also occur under more general initializations and architectures.


Minor Weaknesses

  1. L177: Please correct "THEORETICAL RESUTLS" to "THEORETICAL RESULTS" in the title of Section 3.
  2. Around L323 (the introductory sentence of Proposition 1), L352, L415–416, L421, and L2333: Please replace the unresolved equation references, shown as "(??)" or "equation ??", with the correct numbers.
  3. L413 and L415: "Figure 3(a)" and "Figure 3(b)" appear to refer to Figure 1(a) and Figure 1(b), respectively. Conversely, "Figure 1(a)" and "Figure 1(b)" at L422–423 and L425 should refer to Figure 3(a) and Figure 3(b).
  4. Around L378, the title of Figure 1(b): Please correct "Effect of heta" to "Effect of θ\theta".
  5. L360–361: A verb is missing in "whether the learned model Ψ^\hat{\Psi} beyond …". Please add a verb such as "generalizes" after Ψ^\hat{\Psi}.
  6. Around L377, below Eq. (21): Please correct "an constant" to "a constant".
  7. L1973: "beta" appears as literal text and should be rendered as β\beta.
  8. L1599–1600: Please remove the duplicated word in "equation equation 258".
Questions for authors
  1. Should the generalization guarantee be understood as a high-probability result over independently sampled test prompts? Could the necessary context-coverage conditions and the query-balanced mini-batch sampling assumption be stated explicitly in the theorem? This would make the scope of the guarantee clearer.

  2. How does the default setting θ=0\theta=0 in the synthetic experiments relate to the theoretical requirement θ>0\theta>0? Is this experiment intended to demonstrate behavior beyond the theoretical assumptions? Clarifying this connection would help assess the role of initialization and the generality of the specialization mechanism.

7 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1

Thank you for pointing out this ambiguity. Our analysis indeed explicitly accounts for the probability that a training prompt does not contain matching context examples. Under uniform sampling, for any fixed training prompt, the probability that at least one query component has no matching context example is at most ((1−1/M)L≤e−L/M(1-1/M)^L \leq e^{-L/M}). Since training uses (BB) prompts per iteration over (TT) iterations, applying a union bound gives a total failure probability of at most (2BTe−L/M2BT e^{-L/M}). By choosing (L≥Mlog⁡(2BT/δ)L\geq M\log(2BT/\delta)), this probability is at most (δ\delta). Thus, with probability at least (1−δ1-\delta), all training prompts contain matching context examples for both components. The in-domain and out-of-domain test prompts are also required to satisfy the context length condition which implies with high probability there exists query-matching tokens in the context for both components. We have clarified in the revision that the guarantee holds on this high-probability event over all the prompts.

AuthorsOpenReview ↗

Response to Weakness 2

  • We agree that equation (12) introduces an initial preference for different input components. This initialization is considered to break the symmetry between attention heads so that we can investigate how their preferences are amplified during joint training. However, the theoretical guarantees do not require the exact structured initialization in Eq. (12).

  • Our result still holds when the initialization contains large Gaussian noise. We show in corollary 1 that our theoretical results still hold when the initialization contains additional large Gaussian noise. Specifically, we initialize the attention matrices as W‾h(0)=Wh(0)+W~h\overline{W}_h(0) = W_h(0) + \widetilde{W}_h where Wh(0)W_h(0) is the same structured initialization matrix in (12) and W~h\widetilde{W}_h is the Gaussian noise matrix with entries i.i.d. From N(0,σ2/dx)\mathcal N(0,\sigma^2/d_x). Importantly, this condition does not require the Gaussian perturbation to be small in Frobenius norm relative to the structured initialization. Specifically, we show that for a fixed noise level in the noise matrix, as long as θ\theta in (12) is not too small, greater than the lower bound in (25), then our Theorem 1 still holds with minor changes in some constants that do not affect the order-wise result. For example, when \theta chooses the lower bound in (25), ∥W~h∥F≥∥Wh(0)∥F\|\widetilde{W}_h\|_F \ge \|W_h(0)\|_F scales as dxM32log⁡(M/δ)\frac{d_x}{M^{\frac{3}{2}}\sqrt{\log(M/\delta)}}. Thus, when dx≥M32+ϵd_x \geq M^{\frac{3}{2}+\epsilon} for any ϵ>0\epsilon>0, the Gaussian perturbation can have a substantially larger Frobenius norm than the structured initialization Wh(0)W_h(0).

  • The initial preference is needed for theoretical analyses. Similar symmetry-breaking initializations have been considered in theoretical analyses of head specialization. For example in Chen et al. (NeurIPS 2024, Assumption 3.3), it is assumed that each attention head initially prefers a specific relative position and then the paper analyzes how training reinforces these preferences.

  • Moreover, our initialization does not assume that heads are already specialized. In fact, the initial softmax attention on context tokens with the same pattern as the query for each head is close to uniform (1/M1/M). For example, for M=20M=20 and θ=0.1\theta = 0.1, the softmax score on μ\mu-aligned tokens of head 1 is approximately 5.55.5\\% while uniform attention would be 55\\%. Furthermore, value parameters are initialized at zero and output of the heads are the same for both channels initially. Our analysis shows how training amplifies these weak attention preferences, achieves value specialization and drives attention towards concentration on the relevant context examples.

  • Finally, our experiments with purely Gaussian random initialization, i.e. θ=0\theta=0 investigate a setting beyond the assumptions of our theoretical analysis. The observed complementary specialization suggests that a prescribed initial component preference is not essential for specialization to emerge empirically. Indeed, a realization of purely Gaussian random initialization may introduce small asymmetries between attention heads, which may be amplified during training into complementary specialization. Establishing convergence from arbitrary random initialization remains outside the scope of our current theoretical guarantees.

References

S. Chen, H. Sheen, T. Wang, and Z. Yang. Unveiling induction heads: Provable training dynamics and feature learning in transformers. NeurIPS, 2024

AuthorsOpenReview ↗

Response to the value parameterization concern

  • We thank the reviewer for raising this point. Structured parameterizations are commonly adopted in theoretical analyses of ICL to make the training dynamics analytically tractable. In particular, several existing works fix the value parameters and analyze only the training of attention parameters (Li et al., 2025; Huang et al., 2024; Shen et al., 2025), while other studies consider joint attention–value training under structured model parameterizations (Chen et al., 2024a; He et al., 2025; Zhang et al., 2025).

  • We follow the latter approach by considering block-scalar value matrices with constrained coefficients. Importantly, we do not fix the value parameters or assume their specialized values. Instead, the value coefficients are initialized at zero and trained jointly with the attention parameters. Our analysis establishes that these coefficients converge to complementary specialized solutions through the coupled training dynamics.

  • We agree that the block-scalar parameterization restricts the scope of our theoretical guarantees, and we have clarified this limitation in the paper. However, our analysis goes beyond fixing the value matrix during training by explicitly characterizing how complementary value specialization develops during training.

References

H. Li, S. Lu, P. Chen, X. Cui, and M. Wang. Training nonlinear transformers for chain-of-thought inference: A theoretical generalization analysis. ICLR, 2025

Y. Huang, Y. Cheng, and Y. Liang. In-context convergence of transformers. ICML, 2024

W. Shen, R. Zhou, J. Yang, C. Shen, On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixtures. ICML, 2025

S. Chen, H. Sheen, T. Wang, and Z. Yang. Training dynamics multi-head softmax attention for in-context learning: Emergence, convergence, and optimality. COLT, 2024a

J. He, X. Pan, S. Chen, and Z. Yang. In-context linear regression demystified: Training dynamics and mechanistic interpretability of multi-head softmax attention. ICML, 2025

Y. Zhang, A. Singh, P. E. Latham, and A. M. Saxe. Training dynamics of in context learning in linear attention. ICML, 2025

Reviewer_KGDgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how multi-head Transformers learn complementary specialization for compositional in-context learning. The authors consider a two-head softmax attention model, where each input contains two components with separate input-output mappings. They analyze the joint training of attention matrices and constrained value coefficients, and establish finite-step convergence, head specialization, and generalization to unseen task mappings. They further provide an OOD generalization bound for structured transformations of the training patterns. Synthetic experiments and a compositional task constructed from CIFAR-10 and DTD features are used to support the analysis.

Critical strengths
  • The paper studies an interesting question: how different attention heads develop complementary roles during training, and how this relates to ICL generalization.

  • The finite-step analysis of coupled attention and value updates is technically involved. The characterization of both attention concentration and value specialization is useful within the considered setting.

  • The synthetic experiments track quantities directly related to the theory, including attention mass, component-wise margins, and OOD prediction error, which is appreciated.

Critical weaknesses
  • The specialization is strongly guided by the setup. Eq. (13) initializes the two heads using the true component subspaces, while Eq. (10) already separates the output components and only learns four scalar value coefficients under simplex constraints. To my understanding, the theory explains how a given head-to-component preference is amplified, but not how complementary roles are discovered from ordinary random initialization. Corollary 1 only allows small Gaussian perturbations around the structured initialization. The synthetic experiments with θ=0\theta = 0 are useful, but this case is not covered by the theorem. How essential are the prescribed initialization and the constrained value parameterization?

  • The role of Condition 1 is unclear. With the scalar-identity value maps in Eq. (10) and orthogonal output patterns, changing a task bijection only permutes the output dictionary within each component. For the same input prompt, this leaves the squared loss and all parameter gradients unchanged. Would training on a single task already produce the same optimization trajectory as training on the MM covering tasks? If so, the generalization to unseen mappings is largely built into the parameterization. The authors should explain what the correspondence coverage condition contributes, and what is actually learned from observing multiple tasks.

  • There is a problem with the probability statements in the generalization results. Proposition 1 claims that Eq. (16) holds for any prompt. However, a finite context can contain no example matching the query’s μ\mu-component, with probability ((1−1M)L>0(1-\frac{1}{M})^{L}>0). In this case, the matching set is empty and the left-hand side of Eq. (16) is exactly 1, regardless of the learned parameters. The theorem needs an explicit probability statement or coverage condition for the test context. The probability over training prompts alone cannot justify this claim.

  • The experimental validation of the connection between specialization and generalization is not sufficient. The real-data experiment reports attention concentration and attention margins, but not the corresponding prediction accuracy on unseen task mappings. The task also explicitly concatenates separately extracted CIFAR-10 and DTD features, with contexts constructed to provide matches for each factor. There is no single-head baseline or ablation that changes the specialization behavior. Can a capacity-matched single-head model achieve similar prediction performance? Figure 2 measures training success, which does not directly verify the claimed generalization or context-length advantage.

Minors:

  • Several equation references remain as “(??)”, including those in Proposition 1, Remark 5, and Section 4.1.

  • The references to Figures 1 and 3 are swapped in Section 4.1.

  • The context length is 30 in Appendix A.3 but 100 in Table 2. Please clarify which setting is used for Figure 5.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Weakness 1

  • We agree that equation (12) introduces an initial preference for different input components. This initialization is considered to break the symmetry between attention heads so that we can investigate how their preferences are amplified during joint training. However, the theoretical guarantees do not require the exact structured initialization in Eq. (12).

  • Our result still holds when the initialization contains large Gaussian noise. We show in corollary 1 that our theoretical results still hold when the initialization contains additional large Gaussian noise. Specifically, we initialize the attention matrices as W‾h(0)=Wh(0)+W~h\overline{W}_h(0) = W_h(0) + \widetilde{W}_h where Wh(0)W_h(0) is the same structured initialization matrix in (12) and W~h\widetilde{W}_h is the Gaussian noise matrix with entries i.i.d. From N(0,σ2/dx)\mathcal N(0,\sigma^2/d_x). Importantly, this condition does not require the Gaussian perturbation to be small in Frobenius norm relative to the structured initialization. Specifically, we show that for a fixed noise level in the noise matrix, as long as θ\theta in (12) is not too small, greater than the lower bound in (25), then our Theorem 1 still holds with minor changes in some constants that do not affect the order-wise result. For example, when \theta chooses the lower bound in (25), ∥W~h∥F≥∥Wh(0)∥F\|\widetilde{W}_h\|_F \ge \|W_h(0)\|_F scales as dxM32log⁡(M/δ)\frac{d_x}{M^{\frac{3}{2}}\sqrt{\log(M/\delta)}}. Thus, when dx≥M32+ϵd_x \geq M^{\frac{3}{2}+\epsilon} for any ϵ>0\epsilon>0, the Gaussian perturbation can have a substantially larger Frobenius norm than the structured initialization Wh(0)W_h(0).

  • The initial preference is needed for theoretical analyses. Similar symmetry-breaking initializations have been considered in theoretical analyses of head specialization. For example in Chen et al. (NeurIPS 2024, Assumption 3.3), it is assumed that each attention head initially prefers a specific relative position and then the paper analyzes how training reinforces these preferences.

  • Moreover, our initialization does not assume that heads are already specialized. In fact, the initial softmax attention on context tokens with the same pattern as the query for each head is close to uniform (1/M1/M). For example, for M=20M=20 and θ=0.1\theta = 0.1, the softmax score on μ\mu-aligned tokens of head 1 is approximately 5.55.5% percent while uniform attention would be 55% percent . Furthermore, value parameters are initialized at zero and output of the heads are the same for both channels initially. Our analysis shows how training amplifies these weak attention preferences, achieves value specialization and drives attention towards concentration on the relevant context examples.

  • Finally, our experiments with purely Gaussian random initialization, i.e. θ=0\theta=0 investigate a setting beyond the assumptions of our theoretical analysis. The observed complementary specialization suggests that a prescribed initial component preference is not essential for specialization to emerge empirically. Indeed, a realization of purely Gaussian random initialization may introduce small asymmetries between attention heads, which may be amplified during training into complementary specialization. Establishing convergence from arbitrary random initialization remains outside the scope of our current theoretical guarantees.

References

Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Unveiling induction heads: Provable training dynamics and feature learning in transformers. NeurIPS, 2024

AuthorsOpenReview ↗

Response to the value parameterization concern

  • We thank the reviewer for raising this point. Structured parameterizations are commonly adopted in theoretical analyses of ICL to make the training dynamics analytically tractable. In particular, several existing works fix the value parameters and analyze only the training of attention parameters (Li et al., 2025; Huang et al., 2024; Shen et al., 2025), while other studies consider joint attention–value training under structured model parameterizations (Chen et al., 2024a; He et al., 2025; Zhang et al., 2025).

  • We follow the latter approach by considering block-scalar value matrices with constrained coefficients. Importantly, we do not fix the value parameters or assume their specialized values. Instead, the value coefficients are initialized at zero and trained jointly with the attention parameters. Our analysis establishes that these coefficients converge to complementary specialized solutions through the coupled training dynamics.

  • We agree that the block-scalar parameterization restricts the scope of our theoretical guarantees, and we have clarified this limitation in the paper. However, our analysis goes beyond fixing the value matrix during training by explicitly characterizing how complementary value specialization develops during training.

References

Hongkang Li, Songtao Lu, Pin-Yu Chen, Xiaodong Cui, and Meng Wang. Training nonlinear transformers for chain-of-thought inference: A theoretical generalization analysis. ICLR, 2025

Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. ICML, 2024

Wei Shen, Ruida Zhou, Jing Yang, Cong Shen, On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixtures. ICML, 2025

Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Training dynamics multi-head softmax attention for in-context learning: Emergence, convergence, and optimality. COLT, 2024a

Jianliang He, Xintian Pan, Siyu Chen, and Zhuoran Yang. In-context linear regression demystified: Training dynamics and mechanistic interpretability of multi-head softmax attention. ICML, 2025

Yidi Zhang, Aditya Singh, Peter E. Latham, and Andrew M. Saxe. Training dynamics of in context learning in linear attention. ICML, 2025

AuthorsOpenReview ↗

Response to Weakness 2

We thank the reviewer for this observation and agree that Condition 1 is unnecessary.

  • We have removed Condition 1 from Theorem 1. Our convergence and head specialization results do not require the training tasks to cover all possible bijections. In fact, the results hold for any nonempty set of training tasks, including a single task.

  • The convergence analysis does not depend on the number of distinct training tasks, and it only depends on the total number of training prompts. We originally considered multiple training tasks to emphasize context-dependent prediction, where the same query input can have different outputs across tasks. As the reviewer correctly points out, the loss and gradients are invariant to the specific task bijections. Consequently, removing Condition 1 does not affect the convergence proof or the characterization of complementary head specialization.

  • Structural assumptions are common in theoretical analyses of Transformer training and generalization. We agree that design choices such as model parameterization and prompt construction can affect generalization to unseen mappings. However, similar structural assumptions appear in prior theoretical work. For example, Assumption 3.3 in [Chen et al., 2024a] imposes distinct positional preferences across attention heads at initialization. This assumption enables the learning of the underlying causal structure in [Chen et al., 2024a]. Assumption 2.1 and Definition 3.1 in [Chen et al., 2024b] require different tasks to correspond to orthogonal feature subspaces and the initialization weights of different heads to align with these subspaces, thereby giving each task a preferred head. Moreover, Condition 3.2 in [Li et al., 2025] ensures that each input pattern contributes uniformly to the labels across all training tasks, preventing training-induced bias between the inputs and labels that could impair generalization. These assumptions, like ours, provide a tractable setting for analyzing specific learning mechanisms and their generalization properties.

References

Hongkang Li, Songtao Lu, Pin-Yu Chen, Xiaodong Cui, and Meng Wang. Training nonlinear transformers for chain-of-thought inference: A theoretical generalization analysis. ICLR, 2025

Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Unveiling induction heads: Provable training dynamics and feature learning in transformers. NeurIPS, 2024a

Siyu Chen, Heejune Sheen, Tianhao Wang, Zhuoran Yang. Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality. COLT 2024b.

Reviewer_N47NOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper explores how multihead transformers perform a hierarchical in-context task, and show that different heads will specialise to different components necessary for solving the task. The paper seems to focus on the case of two heads and two components, with each head specialising to each component.

Critical strengths

The paper investigates a very interesting question, namely the specialisation of different attention heads to components of an ICL task, by providing necessary extensions to other studies available in the literature. Table 1 was a particularly useful summary that I liked, clearly detailing the overlap and extensions of this work compared to other related works.

Critical weaknesses

While nothing stands out to me as being immediately mathematically unsound or incorrect, this paper is unfortunately extremely hard to evaluate due to it's current presentation. I do believe the topic and analysis have merit, and for this reason am inclined to increase my score during the rebuttal period. That said, as it stands currently, this paper is very difficult to understand past the introduction, due to (1) many undefined equations/references (2) extremely dense notation and (3) (to my current reading) insufficiently-discussed initial claims. I discuss these further as follows:

(1) The formatting of this paper is incomplete and must be corrected, there are many undefined equations and references that appear in the proofed document, the ones that I saw are on lines 323, 352, 415, 416, and 2333.

(2) I think I can understand schematically what many of the setup / quantities are trying to discuss, but the notation is (in my opinion) quite impenetrable. At some point there are so many symbols that have been introduced very quickly inline that it makes the notation hard to follow. I think this paper could benefit substantially from its main results and propositions being stated in an informal "notationally light" way and then restated formally in the appendix for the interested reader. Furthermore, key quantities such as this "coherence" measure used in the plots are hard to pin down what they actually mean, as they are introduced very quickly (inline in the text too) without much other discussion.

(3) The introduction promised results about the logarithmic reduction of context length necessary to solve the ICL task. I think this is a potentially very neat finding and deserves more exposition, but again the discussion on this seems limited and furthermore the log factor for M in the main result is again quite hidden in a complicated inline formula.

Questions for authors

As listed in the limitations, I have no specific questions other than the request for a rewrite to improve clarity.

EDIT: the clarity has improved and I am increasing my score from 2 to 3.

1 comment
AuthorsOpenReview ↗

Response to Comments

Thank you for these constructive suggestions. We agree that the presentation can be improved, particularly in the notation and explanation of the main results. We will address the comments as follows.

  1. Undefined references and formatting. We have corrected the undefined equations and figure references, including those identified by the reviewer, and checked the cross-references throughout the manuscript.

  2. Notation and clarity.

  • We have added a table of the main notations and explain the key quantities more clearly. See Table 2 in the revised paper.

  • We have added a clear interpretation of OOD coherence in the revised paper. In particular, the OOD coherence, (ρood\rho_{\mathrm{ood}}), measures the level of orthogonality of columns of matrix (CrC^r). Smaller coherence means better separation between these patterns and leads to a stronger OOD generalization bound.

  • We removed Condition 1 per our discussion with reviewer KGDg. This will help to improve and simplify the presentation.

  1. Context-length scaling.
  • We agree that this result deserves more explanation. Under uniform sampling, a context example matches one component of the query with probability (1/M1/M) but matches both query components simultaneously with probability (1/M21/M^2). Our analysis shows that complementary head specialization enables different heads to focus on different components of the context examples which leads to a sufficient context length of (L=O(Mlog⁡(CM2BT/δ))L=O(M\log(CM^2BT/\delta))). However, for a single head model we would need context examples that match the query in both components which would require (L=O~(M2)L = \widetilde{O}(M^2)).

  • We have moved the discussion on the improvement of the context length condition (previously remark 5) to remark 3 in the revised paper and presented it right after Theorem 3.1 to improve the flow of the content.

  • We have included a discussion on how the context length improvement is achieved through complementary head specialization and how Proposition 3.1 shows this specialization is established. This will further clarify the results presented in Proposition 3.1.

  1. Additional experiments

We would like to point out that we have added new experiments during the rebuttal to further strengthen our paper based on other reviewers’ suggestions. We now evaluate the impact of context length on generalization (Figure 5 in the revised paper), comparison with single-head attention with the same capacity (Figure 6 in the revised paper), and an additional experiment that each output channel depends on both input components (Appendix A.3, Figure 8). Specifically,

  • The impact of context length on generalization. We train a two-head model on synthetic data with context length, L=[20,40,80,160,240,320,400]L = [20,40,80,160,240,320,400] and measure the generalization loss on unseen tasks. We repeat this experiment for a different number of patterns for each channel, i.e. M=[20,40,60]M=[20,40,60]. This experiment is intended to show the effect of context length on generalization loss. The results show with L=O~(M)L=\widetilde{O}(M), generalization loss becomes arbitrarily small. See Figure 5.

  • Comparison of single-head and two-head attention of the same capacity. Using real data, we train a two-head and a single-head model and measure the joint accuracy for predicting the labels of both channels during training. The total attention dimension and architecture of the MLP layer and training settings for both models are the same. This experiment is intended to show the advantage of using two heads for compositional tasks. See Figure 6.

  • A new experiment where each output channel depends on both input channels. To investigate specialization beyond independent output mappings in synthetic data, we introduce a coupling parameter λo\lambda_o such that each output channel depends on both input components:

\begin{bmatrix} g_\mu^\tau(\mu)+\lambda_o g_e^\tau(e)\\\\ g_e^\tau(e)+\lambda_o g_\mu^\tau(\mu) \end{bmatrix}.$$ We evaluate $\lambda_o \in \\{0,0.1,0.2,0.3,0.4,0.5\\}$ using a two-head Transformer and report results averaged over three random seeds. The new Figure 8 (Appendix A.3) shows that complementary head specialization persists across all tested coupling strengths. Specifically, Head 1 continues to attend preferentially to matching $\mu$ patterns, while Head 2 attends preferentially to matching $e$ patterns, including on unseen tasks. [See Figure 8.](https://imglink.cc/cdn/_1hmZVpPjq.png)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.