- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the gradient descent dynamics of implicit chain-of-thought (iCoT) learning in a one-layer multi-head transformer on the parity task. The analysis shows how one head picks up each new relevant input bit while earlier heads keep their learned choices. Under the stated conditions, direct training on final answers fails. They also provide numerical results to support the theoretical findings.
Critical strengths
-
The analysis of multi-head learning dynamics which illustrates the competition among attention heads during training is very interesting and technically novel. Most of previous theoretical work has paid relatively little attention to interactions between different heads while they studied the multi-head setup. This paper helps explain how multiple heads learn together and how their assignments evolve during training.
-
I appreciate the clear comparison with closely related work e.g., Huang et al. (2026) and Tsilivis et al. (2026). The authors directly explain how their results differ from previous studies, which helps clarify the technical contribution of this paper.
Critical weaknesses
The main weakness is that, based on how the paper frames its contribution, the advance over prior ICoT results like Huang et al. (2026) seems somewhat limited. The paper extends the analysis to multi-head softmax attention and removes assumptions such as parameter rounding, but introduces other conditions on the initialization of attention heads and the stopping rule between stages. While the multi-head analysis provides useful insights into the learning dynamics, it is unclear how much broader understanding of ICoT these results provide beyond the setting studied in this paper.
Questions for authors
-
The paper removes parameter rounding but introduces new conditions on the bos initialization and stage transitions. Could the author clarify what additional understanding of ICoT does this provide compared with earlier work? (see the weakness)
-
The multi-head analysis requires one new head to be recruited at each stage while earlier heads keep their assignments. I found in Section G.5, it shows some different learning patterns in experiments, where heads forms cancelling pairs and changes their assignments. Can the theory be extended to explain these behaviors?
-
Can the approach be extended to handle longer inputs or more relevant bits without retraining from scratch?
2 comments
Thank you for the detailed response. It helped clarify the paper's contributions relative to prior work. While I sitll feel the broader insights somewhat limited, I believe the paper makes a sufficiently distinct contribution. I therefore maintain my recommendation for acceptance.
We thank the reviewer for the thoughtful and constructive assessment. We appreciate their recognition of the technical novelty of our multi-head learning dynamics analysis and the clarity of our comparison with prior work. We clarify the additional insights our results provide and address the reviewer’s questions below.
W1 & Q1: Additional understanding beyond earlier ICoT results.
Our central contribution is explaining how ordinary GD acquires new computations while preserving earlier ones under coupled, nonconvex dynamics. Huang et al. (2026) use rounding to keep previously trained layers unchanged, while Tsilivis et al.’s (2026) within-stage analysis reduces to linear regression. In our setting, each deletion exposes a missing bit with an order-one normalized-gradient advantage (Lemma 3). Competition among idle heads recruits one head and suppresses redundant recruitment (Lemma 9), while previously recruited heads retain their assignments despite continuing to train. Establishing acquisition and retention jointly is the key technical challenge.
The initialization and stopping conditions specify the starting state and when the supervised target changes. The initialization is independent of the secret and permits independently sampled BOS scales (Corollary 1). The stopping rule determines stage transition timing without rounding, resetting, or freezing learned parameters. Theorem 1 proves that these updates reach the stopping conditions in polynomial time.
The analysis also explains why initialization and curriculum design matter. Exact head symmetry prevents specialization (Proposition 2), and specified diffuse random initializations lead to exponentially slow learning when H=Ω(d) (Proposition 3). For dense parity, Corollary 2 isolates the curriculum’s benefit through a comparison with direct population GD under the same architecture and initialization. Corollary 3 establishes an obstruction for the specified right-to-left schedule. Appendix G.4 further illustrates how premature deletion can cause failure that longer training resolves. These results identify concrete mechanisms behind initialization and schedule sensitivity. The quantitative conditions in Theorem 1 are sufficient rather than necessary.
Q2: Cancelling pairs and changing assignments.
Our current analysis explains part of this behavior. At sharp attention assignments, two heads selecting the same key contribute xj2=1, so a correct prefix prediction can include heads attending to irrelevant keys. Lemma 5 (Appendix B.2) allows precisely this possibility: its assignment condition requires that the keys selected an odd number of times match the target support. It therefore establishes local stability and sharpening near correct configurations containing cancelling pairs.
The remaining challenge is to explain how GD forms and reorganizes these pairs across stages. Theorem 1 tracks previously idle heads moving from BOS to newly required bits while previously recruited heads retain their assignments. Cancelling-pair trajectories can instead involve several heads changing assignments together. Moreover, Figures 6–7 show that the model can underfit one stage and recover later, so it may need to learn multiple missing bits in the next stage. Extending the proof would therefore require tracking the parity represented collectively by groups of heads and analyzing transitions between these configurations. A full convergence guarantee for these dynamics remains open, and we will discuss it in the limitations section of the revision.
Q3: Longer inputs or more relevant bits without retraining from scratch.
For more relevant bits within the existing input dimension, the curriculum already provides a way to continue training without reinitialization. To extend a learned k-bit prefix to k′>k, with H≥k′, we can continue through k′−k additional stages using the corresponding prefix labels. Previously recruited heads retain their assignments, while unused heads learn the additional bits. Theorem 1 guarantees this continuation from checkpoints satisfying the nonfinal stopping conditions: these ensure that earlier assignments are sufficiently stable and unused heads remain ready to learn. Thus, the guarantee concerns continuation from suitable curriculum checkpoints, rather than adaptation from any completed model.
For longer inputs, new positions require additional attention parameters. Reusing the learned logits and initializing only the new ones is a natural extension, but adding keys also changes softmax normalization and may disrupt the conditions needed for further recruitment. Our current theorem assumes a fixed input dimension, and a guarantee for this length extension is left to future work.