- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates how two-layer transformers trained on algorithmic tasks can converge to one of two mechanistically distinct solution classes depending on their weight initialization. The problem statement is: transformers either learn a "single-stage" solution that doesn't meaningfully use the second attention layer and fails to generalize to longer sequences - or a "two-stage" solution that leverages both attention layers and length-generalizes perfectly. The authors demonstrate they can steer models toward either solution by adjusting the relative initialization scale between layers. They identify two key mechanisms explaining the two-stage solution's superior generalization: layer-wise specialization across gap sizes and a "hinting mechanism" in which first attention layer provides contextual information to narrow the second layer's search space. This generalization relationship is task-dependent, the authors show that for for minimum-finding the simpler single-stage solution actually generalizes better, challenging existing ideas about simplicity bias in neural networks. The paper shows examples of how neural networks can implement multiple algorithms for the same task, with different solutions exhibiting different out-of-distribution behavior depending on their internal mechanisms. I find the results interesting and also important.
Critical strengths
The submission demonstrates that two-layer transformers trained on sorting tasks converge to one of two mechanistically distinct solution classes depending on weight initialization, with dramatically different length-generalization behaviors. Only the two-stage solution generalizes to longer sequences. It challenges ideas about the simplicity bias in neural networks by showing that depending on tasks complexity-generalization relationships exist. It shows that for sorting, the more complex two-layer solution generalizes better, while for minimum-finding, the simpler single-layer solution actually generalizes better. Furthermore, and third, it provides detailed mechanistic interpretability identifying specific mechanisms (layer-wise specialization across gap sizes and a hinting mechanism where the first layer narrows the second layer's search space) that explain the two-stage solution's superior generalization, while also demonstrating practical control over which solution emerges by adjusting the relative initialization scale between layers
Critical weaknesses
The paper primarily focuses on sorting, with only brief extensions to minimum-finding and coin denomination tasks. The authors explicitly say that they have to leave a deeper study of these other tasks to future work. Additionally, the phenomena may be specific to the one-head per layer setup, as multi-head models eliminate the length-generalization failure of single-stage solutions. I would appreciate more dteails about this.
While the paper tests 4-layer transformers, the authors acknowledge they don't attempt to characterize three- and four-layer solutions mechanistically, noting that "the mechanisms identified for two layers do not generalize neatly to more layers".
Some models at vocabulary size 1024 deviate from expected patterns, showing a seed-dependent behaviour rather than vocabulary-size-dependent, maybe the findings are not fully consistent across all conditions?
The paper explicitly leaves several important questions unanswered: whether cooperative structures can be characterized in deeper models, whether solutions can be predicted reliably early in training, and whether the same seed-dependent selection occurs in larger models trained on natural data.
Questions for authors
Your experiments use relatively small vocabularies (up to 1024) and sequence lengths (training on length 16), which may not reflect real-world transformer scales. The simplified architecture with one attention head per layer and no positional encodings differs from standard transformer architectures used in practice. Please comment on this.
Your paper shows that training length affects which solution emerges non-monotonically, and models trained at small lengths have worse length-generalization. This creates a practical constraint where training configuration significantly impacts outcomes. How can this be solved?
2 comments
(C1) Thanks, the distinction drawn is correct and supported by your paper. The classification into single-stage and two-stage solutions is defined operationally via second-attention ablation, the length-generalization gap is measured behaviorally, and the steering result is read off the ablation-defined class fraction. None of these depend on the MLP1-only residual-stream property, so the outliers do not threaten the headline findings. So I am happy with the response. Imaybe add a one-line statement in the paper identifying how many checkpoints violate which property (Appendix I provides this: 5/17 for Claim 2, 17/17 for Claim 1)
(C2) I accept the reframing: Appendix E.1 does establish at depth 4 that (i) solution type varies across seeds (four types rather than two), (ii) lowering the scale of layers 2–4 steers 14 of 50 models to a one-layer solution, and (iii) only solutions using at least two layers length-generalize. My original comment conflated the absence of mechanistic characterization with the absence of preliminary evidence, and the former was never claimed to hold at every depth.
(C3) Ok, thanks for the clarifiaction. Characterizing these as future work is appropriate; they are out of scope for a two-layer mechanistic study. I note that two quantities measured in the paper — first-layer attention on the input at gap 1 and second-attention candidate-set size — correlate strongly with final length-generalization and are natural starting points for an early-prediction probe. This is an observation, not a requirement.
Final assessment. The response resolves most of the three addressed points without altering the paper's core claims. The residual items are minor presentation matters. I hold my review score.
Author Response
We thank the reviewer for their kind feedback. We have addressed the reviewer's concerns and questions below.
(C1) “Some models at vocabulary size 1024 deviate from expected patterns, showing a seed-dependent behaviour rather than vocabulary-size-dependent, maybe the findings are not fully consistent across all conditions?”
(Response): We thank the reviewer for studying the appendix closely. The deviation here concerns one simplifying assumption that is used in our analysis as opposed to the main results (the classification into single-stage and two-stage models, the length-generalization gap, and the steering result), which we would like to emphasize none of which depend on this assumption. In appendix F, we study the dependence of the second attention’s scores on components of the residual stream and find that it can be computed from only the first MLP’s output, but find that there are rare checkpoints where this claim does not hold. We believe that these rare checkpoints also fit with the theme of the paper of variation between seeds.
(C2) “While the paper tests 4-layer transformers, the authors acknowledge they don't attempt to characterize three- and four-layer solutions mechanistically, noting that "the mechanisms identified for two layers do not generalize neatly to more layers".”
(Response): In the appendix, we test the paper’s central claims at 4 layers, namely that a model given more layers than the task needs can use a varying number of them depending on the seed, that this can be controlled via modifying the initialization scale, and that the solutions only using one layer still fail to length-generalize. We show that the three of these claims carry over to deeper models, using 4 layers instead of 2. In the paper, the mechanisms we describe for two layer models are part of our explanation for why the difference generalization arises in the cleanest case, but we don’t claim that they explain how cooperation between layers occurs at every depth. With more layers, since there are more ways to divide the work, we expect that the mechanisms would be correspondingly more varied and more complex, which we preferred to report honestly rather than suggesting that the two-layer explanations we describe apply exactly to deeper models.
(C3) “The paper explicitly leaves several important questions unanswered: whether cooperative structures can be characterized in deeper models, whether solutions can be predicted reliably early in training, and whether the same seed-dependent selection occurs in larger models trained on natural data.”
(Response): We listed these open questions in the paper because we see them as next steps for future work, and we appreciate the reviewer highlighting them.