- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how QK initialization scale affects gradient propagation and training in transformer architecture. The main idea is to derive a gradient-norm recurrence for softmax attention and Pre-Norm blocks, and show that large QK initialization amplifies input gradients as , which yields as upper reference. For small nonzero initialization, QK gradient RMS scales linearly with the weights, but Adam and Muon still make large updates. Empirical evaluations are conducted on 100 layer attention stacks and a 124M language model.
Critical strengths
-
The motivation is clear. Prior work mostly analyzed the forward pass at initialization, and this paper connects the attention concentration transition to backward gradient amplification.
-
The theory matches the measurements at initialization. Figure 2 and 3 show that the predicted backward gains match observations across QK scales and depth.
-
Section 5 gives a simple explanation of why small nonzero initialization still learns. QK gradients shrink at the same rate as the weights, and Adam and Muon normalize the update size, so the weights still move quickly. It also clearly seperates this from exactly zero initialization, where QK gradients are zero and the weights never move. (Figure 1)
Critical weaknesses
-
The training experiments are limited to a small model size (124M) and one context length (T=4096), with no scaling study across either dimension. The theory concerns initialization, but the controlled experiments in Figure 3 use attention-only stacks without MLP. Although sequence length is central to the claim, it is varied only in the initialization experiments at Appendix F.2. The recipe does not get validated with QK normalization, which is left as future work at Section 7. Since modern language models are much larger, use longer context, and often use QK norm, it is hard to support whether the recipe is practical.
-
The initialization analysis matches with the controlled measurements, but its connection to the language model training results are somewhat less direct. Section 4 derives gradient explosion at large initialization, but Section 6 says that under AdamW and Muon, large initialization hurts since updates are small relative to large weights, so QK learns slowly. Therefore, the explosion analysis seems to explain the SGD case, but not the failure under adaptive optimizers, which are more practically used in modern language models.
Questions for authors
-
Can the authors provide at least a minimal scaling experiment in training? Does the boundary where the loss penalty starts shift with T as predicted by ? Also, is the recipe still valid when QK norm is used?
-
Can the authors isolate the contributions of gradient amplification and slow relative updates to the loss penalty, under AdamW and Muon? Does it remain if the learning rate is tuned for each initialization scale?
4 comments
Response to Reviewer S5p5 (2/2)
Q2. As the reviewer notes, our gradient analysis directly explains the SGD case. Under AdamW and Muon, the small relative updates of large QK weights already provide a mechanism for slow adaptation. We further clarify more explicitly how gradient amplification additionally contributes to slow updates.
Large initialization affects optimizer updates through two mechanisms:
- As explained in the submission, optimizers with gradient normalization produce smaller relative motion for larger weights (Eqs. 22–23).
- Besides, gradient amplification can additionally reduce effective updates through gradient clipping. Standard language-model training uses global gradient clipping (as listed in Table 2 of the submission), which rescales all gradients by a common factor so that their global norm does not exceed 1. Under large initialization, the extremely large gradient entries dominate this norm, so the factor becomes very small and many gradient entries are scaled to near zero, below Adam's (40.40% at , Table E). These measurements suggest that gradient amplification can additionally attenuate AdamW updates, compounding the scaling of the first mechanism.
Furthermore, as the reviewer suggests, we sweep the learning rate for each initialization (Table D) and observe that the large-initialization penalty remains clear even with the learning rate tuned. This indicates that the penalty cannot be removed by learning-rate tuning.
Table D. Validation loss across learning rates (124M). Best value in each column is in bold.
| LR | (baseline) | |||||||
|---|---|---|---|---|---|---|---|---|
| 1e-4 | 3.868 | 3.868 | 3.879 | 3.905 | 3.950 | 3.996 | 4.072 | 5.082 |
| 6e-4$^\dagger$ | 3.405 | 3.411 | 3.439 | 3.433 | 3.438 | 3.471 | 3.542 | 4.045 |
| 1e-3 | 3.359 | 3.363 | 3.368 | 3.370 | 3.402 | 3.425 | 3.477 | 3.982 |
| 1e-2 | 5.962 | 6.024 | 6.078 | 5.928 | 6.055 | 5.998 | 6.161 | 6.223 |
Same runs as Fig. 1(a); mean over three seeds. Other rows use seed 1.
Table E. Gradients at initialization before and after gradient clipping (124M).
| Quantity | baseline | |||||
|---|---|---|---|---|---|---|
| Global gradient RMS (pre-clip) | 2.47 | |||||
| Clipping factor | ||||||
| Entries clipped $< 10^{-8} $ (%) | 11.823 | 1.576 | 0.402 | 8.401 | 40.402 | 57.046 |
We thank the reviewer for the valuable questions. In summary, we have added the results and analysis to the revision: Q1 → App. F.4 and G.3.1–G.3.3; Q2 → Sec. 6, App. F.5 and G.2 (Tables 2 and 4).
We believe the training experiments and full-decoder gradient measurements effectively respond to the reviewer's concerns and strengthen our conclusions. We therefore respectfully hope the reviewer would reconsider their assessment.
[1] A. Giorlandino and S. Goldt. Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation. ICLR 2026. [2] B. He and T. Hofmann. Simplifying transformer blocks. ICLR 2024. [3] A. Trockman and J. Z. Kolter. Mimetic initialization of self-attention layers. ICML 2023. [4] L. Noci et al. Signal propagation in transformers: theoretical perspectives and the role of rank collapse. NeurIPS 2022.
Thank you for the detailed response and the substantial additional experiments. The new results address several of my original concerns. In particular, the 350M and 760M runs (up to 10.5B tokens) show that the large-initialization penalty persists at scale, and Fig. 9 shows that the predicted Q/K gradient behavior qualitatively carries over to the full decoder, not only to attention-only models. The added QK-norm, sequence-length, and learning-rate experiments further support the practical recommendation to favor small, nonzero initialization.
On re-reading, I also agree that the first-step analysis for adaptive optimizers (Eqs. 22–23) connects the theory to AdamW/Muon training more directly than my original review acknowledged.
Some limitations remain. Over the tested lengths the predicted reference moves by only about 4%, so the coarse λ grid, with single-seed runs at the two added lengths, does not establish that the onset shifts with T; this supports the paper's interpretation of the reference as a practical warning level rather than an empirically verified T-dependent onset. With QK-norm, the loss already increases well below the reference (e.g., +0.092 at λ = 0.5, single seed), so the reference seems less informative as an onset for QK-norm models, consistent with the limitation noted in Sec. 7 that the theoretical analysis does not cover QK-norm. Finally, the clipping measurements suggest an additional mechanism for attenuating AdamW updates but do not isolate its contribution to the training-loss penalty, and the learning-rate sweeps cover AdamW and plain SGD but not Muon.
Overall, the response strengthens the paper, and I am raising my rating from Weak Rejection to Weak Acceptance.
Response to Reviewer S5p5's Follow-up Comment
We sincerely thank the reviewer for the careful reconsideration and constructive feedback.
We appreciate the reviewer's recognition of the paper's contributions and acknowledgment that our response has addressed several key concerns. In particular, our gradient analysis provides concrete predictions about the effects of QK initialization; the first-step optimizer analysis and extensive training experiments further connect these predictions to practical training behavior.
Regarding the remaining observations, we would like to emphasize a few points. First, the weak dependence of the upper reference on sequence length is inherent in its theoretical scaling , and, in fact, its usefulness as a warning reference does not require a pronounced shift across lengths. Second, our theoretical characterization of the standard Transformer without QKNorm is an important contribution in its own right; an exact characterization of the QKNorm case indeed requires further theoretical analysis.
We thank the reviewer again for the thoughtful review. We are genuinely encouraged by this reassessment.
Response to Reviewer S5p5 (1/2)
We thank the reviewer for the insightful feedback. In brief:
Q1. We would first note that our initialization analysis is fully supported on its own, with derived theory and empirical validation. The resulting practical recipe has also been validated over more than 300 language-model training runs (124M parameters, 2.6B FineWeb tokens). This training setting is already comparable to or larger than the typical scale of this line of work, for example, [1] train models with width ≤ 600 on TinyStories (≈ 0.5B tokens), [2] train 167M models on up to 2B tokens with context length 128, [3] train models of width ≤ 410, [4] train translation models on ≈ 160k sentence pairs. We therefore believe the original results already provide solid support for our conclusions.
Experiments in broader settings test how generally these conclusions hold. Following the reviewer's suggestions, we have added larger models, additional context lengths, and QKNorm. Across these settings, the large-initialization penalty persists, consistent with the original conclusions.
New runs use AdamW with a single seed; for reference, the standard deviation in the original 3-seed sweep is about 0.02. As in the submission, the baseline corresponds to σqk/d=0.02, and the λ columns use σqk=(λlnT)1/4, where λ=2 is our proposed upper reference. For clarity, parentheses give the gap to σqk=10−5.
Table A. Validation loss across model sizes.
Table B. Validation loss across sequence lengths (124M).
Table C. Validation loss without and with QKNorm (124M).
† Same runs as Figure 1(a); mean over 3 seeds. The QKNorm row uses seed 1.
§ σqk/d=0.02, i.e., σqk=0.554 / 0.640 / 0.784 for d=768 / 1024 / 1536.
We observe that:
Regarding W1's concern about the attention-only stacks in Fig. 3, we have additionally measured initialization gradients in a full Transformer (the 124M model used in our training experiments) in Fig. 9 of the revised submission. The results show that QK gradient norms scale approximately linearly at small initialization, while input and parameter gradients increase sharply beyond the upper reference, consistent with our theoretical predictions.