acceptodds
Under review as a conference paper at ICLR 2027

How Query–Key Initialization Shapes Gradients and Training in Transformers

Abstract

We study how query and key (QK) initialization shapes gradient propagation and training in Transformers. We derive an explicit gradient-norm recurrence for softmax attention and Pre-Norm residual blocks, revealing an asymmetry between large and small initialization scales. Large initialization can cause gradient explosion: the expected input-gradient gain diverges in the long-sequence limit. For normalized, weakly correlated inputs, this yields as a practical upper reference for , where QK weight entries have variance , is the input dimension, and is the sequence length. Small, nonzero initialization preserves a nonvanishing gradient-to-weight ratio, while gradient normalization in Adam and Muon permits substantial updates despite nearly uniform attention. Experiments with -layer attention stacks and 124M-parameter language models support the analysis, with AdamW and Muon exhibiting a broad low-loss plateau down to . These findings motivate a practical recipe: favor small, nonzero QK initialization; and for broader searches, stay below the upper reference .

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformers, attention initialization, training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S5p5OpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how QK initialization scale affects gradient propagation and training in transformer architecture. The main idea is to derive a gradient-norm recurrence for softmax attention and Pre-Norm blocks, and show that large QK initialization amplifies input gradients as ln⁡T\ln T, which yields (2ln⁡T)1/4(2\ln T)^{1/4} as upper reference. For small nonzero initialization, QK gradient RMS scales linearly with the weights, but Adam and Muon still make large updates. Empirical evaluations are conducted on 100 layer attention stacks and a 124M language model.

Critical strengths
  1. The motivation is clear. Prior work mostly analyzed the forward pass at initialization, and this paper connects the attention concentration transition to backward gradient amplification.

  2. The theory matches the measurements at initialization. Figure 2 and 3 show that the predicted backward gains match observations across QK scales and depth.

  3. Section 5 gives a simple explanation of why small nonzero initialization still learns. QK gradients shrink at the same rate as the weights, and Adam and Muon normalize the update size, so the weights still move quickly. It also clearly seperates this from exactly zero initialization, where QK gradients are zero and the weights never move. (Figure 1)

Critical weaknesses
  1. The training experiments are limited to a small model size (124M) and one context length (T=4096), with no scaling study across either dimension. The theory concerns initialization, but the controlled experiments in Figure 3 use attention-only stacks without MLP. Although sequence length is central to the claim, it is varied only in the initialization experiments at Appendix F.2. The recipe does not get validated with QK normalization, which is left as future work at Section 7. Since modern language models are much larger, use longer context, and often use QK norm, it is hard to support whether the recipe is practical.

  2. The initialization analysis matches with the controlled measurements, but its connection to the language model training results are somewhat less direct. Section 4 derives gradient explosion at large initialization, but Section 6 says that under AdamW and Muon, large initialization hurts since updates are small relative to large weights, so QK learns slowly. Therefore, the explosion analysis seems to explain the SGD case, but not the failure under adaptive optimizers, which are more practically used in modern language models.

Questions for authors
  1. Can the authors provide at least a minimal scaling experiment in training? Does the boundary where the loss penalty starts shift with T as predicted by (2ln⁡T)1/4(2\ln T)^{1/4}? Also, is the recipe still valid when QK norm is used?

  2. Can the authors isolate the contributions of gradient amplification and slow relative updates to the loss penalty, under AdamW and Muon? Does it remain if the learning rate is tuned for each initialization scale?

4 comments
AuthorsOpenReview ↗

Response to Reviewer S5p5 (1/2)

We thank the reviewer for the insightful feedback. In brief:

  • Q1 (scale, context length, QKNorm): we have added supplementary experiments; results are consistent with the original conclusions.
  • Q2 (failure under adaptive optimizers): we make the link between gradient amplification and slow relative updates more explicit here and in the revision.

Q1. We would first note that our initialization analysis is fully supported on its own, with derived theory and empirical validation. The resulting practical recipe has also been validated over more than 300 language-model training runs (124M parameters, 2.6B FineWeb tokens). This training setting is already comparable to or larger than the typical scale of this line of work, for example, [1] train models with width ≤ 600 on TinyStories (≈ 0.5B tokens), [2] train 167M models on up to 2B tokens with context length 128, [3] train models of width ≤ 410, [4] train translation models on ≈ 160k sentence pairs. We therefore believe the original results already provide solid support for our conclusions.

Experiments in broader settings test how generally these conclusions hold. Following the reviewer's suggestions, we have added larger models, additional context lengths, and QKNorm. Across these settings, the large-initialization penalty persists, consistent with the original conclusions.

New runs use AdamW with a single seed; for reference, the standard deviation in the original 3-seed sweep is about 0.02. As in the submission, the baseline corresponds to σqk/d=0.02\sigma_{qk}/\sqrt{d}=0.02, and the λ\lambda columns use σqk=(λln⁡T)1/4\sigma_{qk}=(\lambda\ln T)^{1/4}, where λ=2\lambda=2 is our proposed upper reference. For clarity, parentheses give the gap to σqk=10−5\sigma_{qk}=10^{-5}.

Table A. Validation loss across model sizes.

Model10−510^{-5}baseline §^\Sλ=0.5\lambda=0.5λ=2\lambda=2λ=8\lambda=855
124M (2.62B tokens)†^\dagger3.4053.439 (+0.034)3.433 (+0.028)3.438 (+0.033)3.542 (+0.137)4.045 (+0.640)
350M (2.62B tokens)3.1653.173 (+0.008)3.193 (+0.028)3.228 (+0.063)3.397 (+0.232)3.974 (+0.809)
760M (10.5B tokens)2.8932.917 (+0.024)2.902 (+0.009)2.905 (+0.012)3.011 (+0.119)3.612 (+0.720)

Table B. Validation loss across sequence lengths (124M).

Length10−510^{-5}10−310^{-3}baselineλ=0.5\lambda=0.5λ=2\lambda=2λ=4\lambda=4λ=8\lambda=855
T=2048T=20483.4003.405 (+0.005)3.430 (+0.030)3.427 (+0.027)3.439 (+0.039)3.469 (+0.069)3.528 (+0.128)4.026 (+0.626)
T=4096†T=4096^\dagger3.4053.411 (+0.006)3.439 (+0.034)3.433 (+0.028)3.438 (+0.033)3.471 (+0.066)3.542 (+0.137)4.045 (+0.640)
T=8192T=81923.4713.482 (+0.011)3.503 (+0.032)3.509 (+0.038)3.504 (+0.033)3.532 (+0.061)3.614 (+0.143)4.087 (+0.616)

Table C. Validation loss without and with QKNorm (124M).

Setting10−510^{-5}10−310^{-3}baselineλ=0.5\lambda=0.5λ=2\lambda=2λ=4\lambda=4λ=8\lambda=855
w/o QKNorm$^\dagger$3.4053.411 (+0.006)3.439 (+0.034)3.433 (+0.028)3.438 (+0.033)3.471 (+0.066)3.542 (+0.137)4.045 (+0.640)
w/ QKNorm3.3483.350 (+0.002)3.403 (+0.055)3.440 (+0.092)3.470 (+0.122)3.482 (+0.134)3.488 (+0.140)3.535 (+0.187)

†^\dagger Same runs as Figure 1(a); mean over 3 seeds. The QKNorm row uses seed 1.

§^\S σqk/d=0.02\sigma_{qk}/\sqrt{d}=0.02, i.e., σqk=0.554\sigma_{qk}=0.554 / 0.6400.640 / 0.7840.784 for d=768d=768 / 10241024 / 15361536.

We observe that:

  1. Across model scales (Table A), the large-initialization penalty persists at 350M and at 760M with 10.5B tokens.
  2. Across context lengths (Table B), the penalty follows a nearly identical trend: relative to σqk=10−5\sigma_{qk}=10^{-5}, the gap grows from within 0.04 at λ=2\lambda=2 to 0.06–0.07 at λ=4\lambda=4 and 0.13–0.14 at λ=8\lambda=8. This supports (2ln⁡T)1/4(2\ln T)^{1/4} as a useful upper reference.
  3. QKNorm (Table C) reduces the penalty at very large initialization, but smaller initialization still yields clearly lower loss. Thus QKNorm does not eliminate the initialization sensitivity.

Regarding W1's concern about the attention-only stacks in Fig. 3, we have additionally measured initialization gradients in a full Transformer (the 124M model used in our training experiments) in Fig. 9 of the revised submission. The results show that QK gradient norms scale approximately linearly at small initialization, while input and parameter gradients increase sharply beyond the upper reference, consistent with our theoretical predictions.

AuthorsOpenReview ↗

Response to Reviewer S5p5 (2/2)

Q2. As the reviewer notes, our gradient analysis directly explains the SGD case. Under AdamW and Muon, the small relative updates of large QK weights already provide a mechanism for slow adaptation. We further clarify more explicitly how gradient amplification additionally contributes to slow updates.

Large initialization affects optimizer updates through two mechanisms:

  1. As explained in the submission, optimizers with gradient normalization produce smaller relative motion for larger weights (Eqs. 22–23).
  2. Besides, gradient amplification can additionally reduce effective updates through gradient clipping. Standard language-model training uses global gradient clipping (as listed in Table 2 of the submission), which rescales all gradients by a common factor so that their global norm does not exceed 1. Under large initialization, the extremely large gradient entries dominate this norm, so the factor becomes very small and many gradient entries are scaled to near zero, below Adam's ϵ=10−8\epsilon=10^{-8} (40.40% at σqk=5\sigma_{qk}=5, Table E). These measurements suggest that gradient amplification can additionally attenuate AdamW updates, compounding the 1/σqk1/\sigma_{qk} scaling of the first mechanism.

Furthermore, as the reviewer suggests, we sweep the learning rate for each initialization (Table D) and observe that the large-initialization penalty remains clear even with the learning rate tuned. This indicates that the penalty cannot be removed by learning-rate tuning.

Table D. Validation loss across learning rates (124M). Best value in each column is in bold.

LR10−510^{-5}10−310^{-3}0.5540.554 (baseline)λ=0.5\lambda=0.5λ=2\lambda=2λ=4\lambda=4λ=8\lambda=855
1e-43.8683.8683.8793.9053.9503.9964.0725.082
6e-4$^\dagger$3.4053.4113.4393.4333.4383.4713.5424.045
1e-33.3593.3633.3683.3703.4023.4253.4773.982
1e-25.9626.0246.0785.9286.0555.9986.1616.223

†^\dagger Same runs as Fig. 1(a); mean over three seeds. Other rows use seed 1.

Table E. Gradients at initialization before and after gradient clipping (124M).

Quantity10−510^{-5}baselineλ=2\lambda=2λ=8\lambda=8551010
Global gradient RMS (pre-clip)1.04×10−31.04\times10^{-3}1.03×10−31.03\times10^{-3}1.03×10−31.03\times10^{-3}1.53×10−21.53\times10^{-2}2.474.01×1034.01\times10^{3}
Clipping factor8.68×10−28.68\times10^{-2}8.69×10−28.69\times10^{-2}8.72×10−28.72\times10^{-2}5.87×10−35.87\times10^{-3}3.65×10−53.65\times10^{-5}2.24×10−82.24\times10^{-8}
Entries clipped $< 10^{-8} $ (%)11.8231.5760.4028.40140.40257.046

We thank the reviewer for the valuable questions. In summary, we have added the results and analysis to the revision: Q1 → App. F.4 and G.3.1–G.3.3; Q2 → Sec. 6, App. F.5 and G.2 (Tables 2 and 4).

We believe the training experiments and full-decoder gradient measurements effectively respond to the reviewer's concerns and strengthen our conclusions. We therefore respectfully hope the reviewer would reconsider their assessment.

[1] A. Giorlandino and S. Goldt. Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation. ICLR 2026. [2] B. He and T. Hofmann. Simplifying transformer blocks. ICLR 2024. [3] A. Trockman and J. Z. Kolter. Mimetic initialization of self-attention layers. ICML 2023. [4] L. Noci et al. Signal propagation in transformers: theoretical perspectives and the role of rank collapse. NeurIPS 2022.

Thank you for the detailed response and the substantial additional experiments. The new results address several of my original concerns. In particular, the 350M and 760M runs (up to 10.5B tokens) show that the large-initialization penalty persists at scale, and Fig. 9 shows that the predicted Q/K gradient behavior qualitatively carries over to the full decoder, not only to attention-only models. The added QK-norm, sequence-length, and learning-rate experiments further support the practical recommendation to favor small, nonzero initialization.

On re-reading, I also agree that the first-step analysis for adaptive optimizers (Eqs. 22–23) connects the theory to AdamW/Muon training more directly than my original review acknowledged.

Some limitations remain. Over the tested lengths the predicted reference moves by only about 4%, so the coarse λ grid, with single-seed runs at the two added lengths, does not establish that the onset shifts with T; this supports the paper's interpretation of the reference as a practical warning level rather than an empirically verified T-dependent onset. With QK-norm, the loss already increases well below the reference (e.g., +0.092 at λ = 0.5, single seed), so the reference seems less informative as an onset for QK-norm models, consistent with the limitation noted in Sec. 7 that the theoretical analysis does not cover QK-norm. Finally, the clipping measurements suggest an additional mechanism for attenuating AdamW updates but do not isolate its contribution to the training-loss penalty, and the learning-rate sweeps cover AdamW and plain SGD but not Muon.

Overall, the response strengthens the paper, and I am raising my rating from Weak Rejection to Weak Acceptance.

AuthorsOpenReview ↗

Response to Reviewer S5p5's Follow-up Comment

We sincerely thank the reviewer for the careful reconsideration and constructive feedback.

We appreciate the reviewer's recognition of the paper's contributions and acknowledgment that our response has addressed several key concerns. In particular, our gradient analysis provides concrete predictions about the effects of QK initialization; the first-step optimizer analysis and extensive training experiments further connect these predictions to practical training behavior.

Regarding the remaining observations, we would like to emphasize a few points. First, the weak dependence of the upper reference on sequence length is inherent in its theoretical scaling ∝(ln⁡T)1/4\propto (\ln T)^{1/4}, and, in fact, its usefulness as a warning reference does not require a pronounced shift across lengths. Second, our theoretical characterization of the standard Transformer without QKNorm is an important contribution in its own right; an exact characterization of the QKNorm case indeed requires further theoretical analysis.

We thank the reviewer again for the thoughtful review. We are genuinely encouraged by this reassessment.

Reviewer_yXWWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how query–key initialization affects gradient propagation and Transformer training. It derives gradient-norm recurrences and explains why large initialization can amplify gradients, while small, nonzero initialization can still support effective learning under AdamW and Muon. Experiments with deep attention stacks and a 124M-parameter language model support the analysis and motivate practical initialization guidance.

Critical strengths
  1. The paper addresses a concrete training question and connects initialization scale, gradient propagation, and optimizer behavior within a unified analysis.

  2. The distinction between small, nonzero initialization and exactly zero initialization is clearly explained and supported experimentally.

  3. The experiments track gradients, parameter updates, and attention distributions alongside validation loss, providing useful evidence for the proposed mechanisms.

Critical weaknesses
  1. Limited training scale and duration. The language-model experiments use a 124M model trained for approximately 2.62B tokens under AdamW and Muon. Further validation with approximately 1B parameters and 10B training tokens is needed to assess generalization across scales and whether the observed loss differences persist beyond transient initialization effects.

  2. The practical advantage over standard initialization could be clearer. The broad low-loss plateau includes the standard initialization baseline, with limited evidence of consistent improvement from smaller scales. Comparisons with established stabilization methods, such as QK normalization, would further clarify the practical value of the recommendation.

  3. Limited validation of the predicted sequence-length dependence. The proposed QK initialization reference, (2ln⁡T)1/4(2\ln T)^{1/4}, explicitly depends on sequence length. However, the language-model experiments only use (T=4096), leaving unclear whether the initialization scale associated with performance degradation shifts with sequence length as predicted.

Questions for authors
  1. Please consider extending representative initialization settings to approximately 1B parameters and 10B training tokens, with validation-loss gaps tracked throughout training. This would help determine whether the observed differences persist or diminish over time.

  2. Does small QK initialization provide measurable benefits over a well-tuned standard baseline, and do these benefits remain when QK normalization is used?

  3. How well does the proposed sequence-length-dependent reference predict the onset of performance degradation at different context lengths? Repeating the initialization sweep across several lengths would help assess its predictive value.

4 comments
AuthorsOpenReview ↗

Response to Reviewer yXWW

We thank the reviewer for the constructive review and for recognizing our clarity and soundness. In brief:

  • Q1 (scale and duration): we have trained 760M models on 10.5B tokens; the large-initialization penalty persists through long-duration training.
  • Q2 (baseline and QKNorm): we clarify the role of the standard baseline and show that QKNorm does not remove the penalty.
  • Q3 (sequence length): across three context lengths, the penalty rises consistently beyond the predicted reference.

Q1. We would first note that the submission has already included more than 300 language-model training runs, amounting to about 1500 A100 GPU hours, which is a substantial experimental effort. Moreover, the training setting (124M models with 2.6B FineWeb tokens) is in fact comparable to or larger than the typical scale of this line of work: for example, [1] train models with width ≤ 600 on TinyStories (≈ 0.5B tokens), [2] train 167M models on up to 2B tokens with context length 128, [3] train models of width ≤ 410, and [4] train translation models on ≈ 160k sentence pairs. We therefore believe the original results already provide solid support for our conclusions.

Since experiments at larger scale further strengthen the generality of our conclusions, we tried our best to follow the reviewer's suggestion and trained 760M models (i.e., 0.76B) as GPT-3 Large configuration on 10.5B tokens with AdamW. Validation loss throughout training is reported in Table I (also in revised Appendix G.3.2, Table 7 and Fig. 14).

As in the submission, the baseline corresponds to σqk/d=0.02\sigma_{qk}/\sqrt{d}=0.02, and the λ\lambda columns use σqk=(λln⁡T)1/4\sigma_{qk}=(\lambda\ln T)^{1/4}, where λ=2\lambda=2 is our proposed upper reference.

Table I. Validation loss of the 760M model throughout training. For readability, gaps to σqk=10−5\sigma_{qk}=10^{-5} are given in parentheses.

Step (tokens)10−510^{-5}0.7840.784 (baseline)λ=0.5\lambda=0.5λ=2\lambda=2λ=8\lambda=855
5k (2.6B)3.2043.232 (+0.028)3.217 (+0.013)3.232 (+0.028)3.401 (+0.197)6.151 (+2.947)
6k (3.1B)3.1563.179 (+0.023)3.166 (+0.010)3.179 (+0.023)3.327 (+0.171)4.195 (+1.039)
7k (3.7B)3.1233.147 (+0.023)3.133 (+0.010)3.143 (+0.020)3.278 (+0.155)4.087 (+0.963)
8k (4.2B)3.0913.113 (+0.023)3.100 (+0.009)3.108 (+0.017)3.235 (+0.144)4.027 (+0.936)
9k (4.7B)3.0643.088 (+0.024)3.075 (+0.011)3.082 (+0.018)3.203 (+0.139)3.976 (+0.912)
10k (5.2B)3.0383.061 (+0.022)3.049 (+0.010)3.053 (+0.015)3.172 (+0.133)3.912 (+0.874)
11k (5.8B)3.0153.039 (+0.024)3.027 (+0.011)3.031 (+0.016)3.147 (+0.132)3.862 (+0.847)
12k (6.3B)2.9953.018 (+0.023)3.005 (+0.010)3.008 (+0.013)3.121 (+0.126)3.815 (+0.820)
13k (6.8B)2.9742.998 (+0.024)2.983 (+0.009)2.988 (+0.014)3.099 (+0.125)3.772 (+0.798)
14k (7.3B)2.9562.979 (+0.023)2.965 (+0.009)2.969 (+0.014)3.079 (+0.123)3.734 (+0.778)
15k (7.9B)2.9382.963 (+0.025)2.949 (+0.010)2.952 (+0.013)3.061 (+0.122)3.701 (+0.763)
16k (8.4B)2.9252.950 (+0.025)2.934 (+0.010)2.938 (+0.013)3.046 (+0.121)3.676 (+0.751)
17k (8.9B)2.9142.938 (+0.025)2.923 (+0.009)2.926 (+0.012)3.034 (+0.120)3.654 (+0.740)
18k (9.4B)2.9042.929 (+0.025)2.914 (+0.010)2.917 (+0.013)3.024 (+0.120)3.637 (+0.733)
19k (10.0B)2.8982.923 (+0.024)2.908 (+0.009)2.910 (+0.011)3.017 (+0.118)3.623 (+0.725)
20k (10.5B)2.8932.917 (+0.024)2.902 (+0.009)2.905 (+0.012)3.011 (+0.119)3.612 (+0.720)

To compare small and large initialization, let's focus on two representative columns, σqk=10−5\sigma_{qk}=10^{-5} and λ=8\lambda=8. We observe that, throughout training, the gap narrows quickly at first but then increasingly slowly, remaining 0.119 after 10.5B tokens. The penalty is therefore not merely a transient initialization effect.

AuthorsOpenReview ↗

Response to Reviewer yXWW - 2

Q2. First, we would like to clarify that our practical recipe (Sec. 7) is a guideline on the safe range of QK initialization rather than a new state-of-the-art initialization scheme: large initialization beyond the reference needs to be avoided, while small initialization, down to the tested 10−510^{-5}, is safe. It is therefore expected that the standard baseline lies in the low-loss plateau (Table 1 of the submission: common defaults lie below the reference); our analysis explains why this default works and where larger scales become harmful.

Moreover, under the common fixed-std convention (σqk/d=0.02\sigma_{qk}/\sqrt{d}=0.02), σqk=0.02d\sigma_{qk}=0.02\sqrt{d} grows with width and approaches the upper reference (2ln⁡T)1/4(2\ln T)^{1/4} as models become wider (e.g., 1.81 at d=8192d=8192 and 2.22 at d=12288d=12288, vs. 2.02 at T=4096T=4096 and 2.10 at T=16384T=16384). This may inform the initialization of future larger models.

Regarding QKNorm (Table II), we observe that it reduces the penalty at very large initialization, which is expected since it normalizes the scale of queries and keys; however, smaller initialization still achieves clearly lower loss, so QKNorm does not eliminate this sensitivity.

Table II. Validation loss without and with QKNorm (124M).

Setting10−510^{-5}10−310^{-3}0.5540.554 (baseline)λ=0.5\lambda=0.5λ=2\lambda=2λ=4\lambda=4λ=8\lambda=855
w/o QKNorm$^\dagger$3.4053.411 (+0.006)3.439 (+0.034)3.433 (+0.028)3.438 (+0.033)3.471 (+0.066)3.542 (+0.137)4.045 (+0.640)
w/ QKNorm3.3483.350 (+0.002)3.403 (+0.055)3.440 (+0.092)3.470 (+0.122)3.482 (+0.134)3.488 (+0.140)3.535 (+0.187)

†^\dagger Same runs as Figure 1(a); mean over 3 seeds. The QKNorm row uses seed 1.

Q3. We repeat the initialization sweep at two additional lengths, T=2048T=2048 and 81928192, alongside the original T=4096T=4096 (Table III).

We observe that across all lengths, the penalty follows a nearly identical trend: relative to σqk=10−5\sigma_{qk}=10^{-5}, the gap grows from within 0.04 at λ=2\lambda=2 to 0.06–0.07 at λ=4\lambda=4 and 0.13–0.14 at λ=8\lambda=8. This supports (2ln⁡T)1/4(2\ln T)^{1/4} as a useful upper reference.

Table III. Validation loss across sequence lengths (124M).

Length10−510^{-5}10−310^{-3}0.5540.554 (baseline)λ=0.5\lambda=0.5λ=2\lambda=2λ=4\lambda=4λ=8\lambda=855
T=2048T=20483.4003.405 (+0.005)3.430 (+0.030)3.427 (+0.027)3.439 (+0.039)3.469 (+0.069)3.528 (+0.128)4.026 (+0.626)
T=4096†T=4096^\dagger3.4053.411 (+0.006)3.439 (+0.034)3.433 (+0.028)3.438 (+0.033)3.471 (+0.066)3.542 (+0.137)4.045 (+0.640)
T=8192T=81923.4713.482 (+0.011)3.503 (+0.032)3.509 (+0.038)3.504 (+0.033)3.532 (+0.061)3.614 (+0.143)4.087 (+0.616)

†^\dagger Same runs as Figure 1(a); mean over 3 seeds.

We thank the reviewer again for the constructive suggestions. In summary, we have added the above results to the revision: Q1 → App. G.3.2 (Tables 6–7, Fig. 14); Q2 → Sec. 4.1 and App. G.3.3 (Table 8); Q3 → App. G.3.1 (Table 5).

We believe the additional experiments on model scale, training duration, context length, and QKNorm effectively respond to the reviewer's concerns and support our conclusions. We therefore respectfully hope the reviewer would reconsider the assessment in light of the paper's substantial contributions.

[1] A. Giorlandino and S. Goldt. Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation. ICLR 2026. [2] B. He and T. Hofmann. Simplifying transformer blocks. ICLR 2024. [3] A. Trockman and J. Z. Kolter. Mimetic initialization of self-attention layers. ICML 2023. [4] L. Noci et al. Signal propagation in transformers: theoretical perspectives and the role of rank collapse. NeurIPS 2022.

Thank you for the responses and additional experiments. The 760M model trained on 10.5B tokens remains insufficient in scale to support broader recommendations for LLM training. The additional experiments mainly reinforce existing initialization trends and do not convince me that the work provides substantial new theoretical insight beyond prior studies of Transformer initialization and gradient propagation. I maintain my score.

AuthorsOpenReview ↗

Follow-up Response to Reviewer yXWW

Thank you for the follow-up. We have carefully considered your latest comments alongside the original review.

The original review suggested extending the training scale and duration to approximately 1B parameters and 10B training tokens. In response, we have added experiments with 760M (0.76B) models trained on 10.5B tokens, very close to the requested parameter scale and exceeding the requested token count. The follow-up nevertheless calls this scale insufficient. This appears to represent a substantial shift in the criterion. We wonder whether these additional experiments were fully taken into account in the follow-up, and would appreciate it if the reviewer could revisit this assessment. Regarding the latter two points in the original review, we understand these concerns to have been addressed, as the follow-up does not identify remaining issues.

Beyond these points, the follow-up raises a new concern about theoretical novelty. We respectfully disagree with this assessment. Our contributions are clearly stated: explicit QK-scale-dependent gradient recurrences, the transition to backward amplification, and small-initialization gradient scaling and its implications for first-step optimizer updates. Their distinctions from prior work are discussed in detail in Appendix A.1 of the original submission. We acknowledge that these may not be apparent in the current presentation, and we can further highlight them in the main text (Sections 3–5) to make them more prominent. If the concern remains, we would expect the reviewer to provide specific evidence and reasoning to support this assessment.

Reviewer_6ncUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how Q-K initialization, within a transformer, affects optimization with gradient based optimizers. The authors consider small and large scale initializations. Their work shows that in principle it is better to go with a small non-zero initialization and to stay below a certain bound the authors derive dependent on the sequence length.

Critical strengths
  1. The paper is exposited well and the clarity of the paper is good. Furthermore, the authors give a good amount of details for the technical parts of the paper.
Critical weaknesses
  1. It would be good if the authors would have shown how the different initializations affect the optimization trajectory.

  2. The paper is about training transformers but none of the theory actually say anything about the learning dynamics with the different initializations.

Questions for authors
  1. Are you able to say anything about the learning dynamics of the transformers with these different initialization schemes?

  2. Since understanding the exact learning dynamics can be difficult have you tried seeing what happens in the infinite width regime with the neural tangent kernel with regard to such initializations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6ncU

We thank the reviewer for the feedback. We would like to clarify that our analysis does provide concrete, experimentally verified insights into training behavior, and our approach follows a well-established research methodology.

Q1. We would first emphasize that combining initialization theory with training experiments is a classical and well-established approach to understanding neural network optimization. Notable examples are the foundational Xavier [1] and Kaiming [2] initializations: both were derived from analyses at initialization, and their training benefits were validated experimentally, without requiring a complete theory of learning dynamics. This methodology has also been widely adopted for Transformers: Xiong et al. [3] use gradient analysis at initialization to explain the different training stability of Post-LN and Pre-LN Transformers; Noci et al. [4] use signal-propagation analysis at initialization to show how rank collapse hinders training.

Our work follows this established approach and obtains concrete predictions that are verified through training experiments. Specifically, our initialization and first-step analyses predict that: under SGD, small initialization takes small absolute steps (Proposition 5.1, Eq. 21), while large initialization produces amplified gradients (Theorem 4.1); under AdamW and Muon, in contrast, smaller initialization leads to larger relative updates (Eqs. 22–23; Fig. 4). We then test these predictions in training experiments: under SGD, small-init QK attention barely changes, while large initialization can cause divergence (Figs. 11–12 (original)/Figs. 12–13 (revised)); under AdamW and Muon, small-init weights are rapidly updated and the attention adapts quickly, whereas large-init attention stays concentrated (Figs. 5, Figs. 9–10 (original)/Figs. 10–11 (revised)). Thus, the predicted mechanisms are supported by practical training experiments. We therefore believe our theory and experiments provide meaningful conclusions about training behavior.

Regarding W1, the original submission already reported optimization trajectories, including validation loss, gradient and weight norms, relative updates, and attention concentration (Figs. 9–11, Appendix F.4 in the original submission; now moved to Figs. 10–12, Appendix G in the revision). We have made these results more prominent by adding representative training-loss trajectories to the main-text Fig. 5.

Q2. We did not pursue an NTK analysis previously because it typically describes the lazy-training regime [5,6]. Since attention logits are bilinear in WQW_Q and WKW_K, an NTK analysis would suggest a diminished initial QK-kernel contribution at small initialization. However, under AdamW and Muon, small-init QK weights grow by orders of magnitude and attention changes substantially (Figs. 10–11 in the revision), so a fixed-kernel approximation may not capture their behaviour.

We thank the reviewer again for the valuable feedback. In the revision, we have clarified the connection between initialization analysis and training behavior in the Introduction and Sec. 7 (Q1), and moved representative trajectories to Fig. 5 (W1).

We hope these clarifications address the reviewer's concerns, and we believe our work provides substantial insights into Transformer optimization, even without a complete theory of learning dynamics. We therefore respectfully hope the reviewer would reconsider the assessment.

[1] Xavier Glorot and Yoshua Bengio. Understanding the Difficulty of Training Deep Feedforward Neural Networks. AISTATS 2010.

[2] Kaiming He et al. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. ICCV 2015.

[3] Ruibin Xiong et al. On Layer Normalization in the Transformer Architecture. ICML 2020.

[4] Lorenzo Noci et al. Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse. NeurIPS 2022.

[5] Lénaïc Chizat et al. On Lazy Training in Differentiable Programming. NeurIPS 2019.

[6] Jiri Hron et al. Infinite Attention: NNGP and NTK for Deep Attention Networks. ICML 2020.

Thanks for your response and thank you for pointing out those references. While those references are certainly good they do have issues, e.g. Kaiming's paper, where conditions like independence is assumed among weights at initialization which certainly do not hold during training. So I think using them to defend your paper is a way to try to avoid discussion. That being said, I do agree that getting insight about initialization even in idealized settings is useful especially for practical purposes but your practical experiments are at extremely small scales. In the experiments don't you show results for 124M model in the main paper? That's unfortunately not a practical scale people work with, neither is 760M. The community trains much larger models so if your goal is to show some theory at initialization and not go into the theory of training dynamics I would expect a lot more empirical evidence which I don't see.

I thank you for the clarification and your effort in the rebuttal. I will keep my score.

Thank you for the feedback. We have carefully considered your concern, and we believe we now have a clearer understanding of the root of our disagreement.

We understand the reviewer’s request for large-scale experiments. From this perspective, larger scales may always be better. However, there could be a reasonable and broadly accepted range of scales for the research community. In fact, models with roughly 100M–1B parameters are commonly used in recent research on pretraining. For example, [1] studies models with 360M and 660M models; [2] studies 130M–1.2B models; and [3] studies 124M and 774M models. Our experiments at 124M, 350M, and 760M parameters are comparable to this standard range. Therefore, we believe the potential value of our findings can be supported.

We hope the reviewer would assess our experimental scales with reference to the practical scales commonly adopted in the research community, while also giving consideration to the value of our initialization analysis and the empirical evidence consistent with its predictions.

[1] Nikhil Vyas et al. SOAP: Improving and Stabilizing Shampoo using Adam for Language Modeling. ICLR 2025. [2] Kaiyue Wen et al. Fantastic Pretraining Optimizers and Where to Find Them. ICLR 2026. [3] Noah Amsel et al. The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm. ICLR 2026, Outstanding Paper Honorable Mention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.