- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the evolution of normalization weights across Transformer layers and observes depth-dependent changes in their magnitude and direction. Based on these observations, the authors propose LayerRoPE, which generates layer-wise normalization weights from shared parameters using depth-conditioned scaling and rotation. Experiments on language models, deep Transformers, looped models, and Vision Transformers show improvements in training convergence, learning-rate stability, and downstream performance.
Critical strengths
The paper provides extensive experiments across different model scales, depths, and architectures. The improvements in learning-rate stability and depth scaling are particularly interesting. LayerRoPE is relatively simple and introduces almost no additional computational overhead. The results also show improvements across both Pre-Norm and Peri-Norm architectures. The analysis of normalization weights across pretrained LLMs provides an interesting motivation for the proposed method. The transfer results on looped models and ViTs are also encouraging.
Critical weaknesses
-
Normalization weights as depth-positional encoding needs further justification. The authors observe that normalization weights change in magnitude and direction across layers and interpret this as an implicit depth-positional encoding. However, since each Transformer layer already has its own parameters and computational role, such depth-dependent variations are not necessarily evidence of positional encoding. I am not fully convinced that the observed patterns indicate an encoding mechanism rather than differences in the scaling requirements of individual layers. Some additional analysis would help clarify this interpretation.
-
Rotation component. Table 4 shows that magnitude-only scaling already improves performance considerably, while adding rotation provides a smaller additional gain. This raises the question of how much of LayerRoPE's improvement comes from depth-dependent scaling rather than the proposed rotary formulation. A comparison with simpler learnable layer-wise scaling schedules would help clarify the role of rotation.
-
The depth-scaling experiments. The experiments up to 512 layers are conducted with a hidden dimension of 128 and models ranging from approximately 18M to 111M parameters. While these experiments demonstrate interesting behavior at extreme depth, it is unclear whether similar advantages would hold for deeper models with more typical hidden dimensions and parameter counts. The main model-scaling experiments also stop at 1.3B parameters.
Questions for authors
see weakness
5 comments
Author Response (2/2)
A comparison with simpler learnable layer-wise scaling schedules would help clarify the role of rotation.
Ablating against carefully tuned layer-wise scaling in our LayerRoPE form, as well as tuned standard layer-wise scaling approaches, shows that depth encoding is a consistently more useful inductive bias. In LayerRoPE's own form, we remove the depth encoding and instead learn scalars shared across layers or free at every layer (which can express any layer-wise scaling schedule), and carefully sweep 16 initializations of these learned scalars per backbone (Appendix F.2):
| Validation loss at 134M (share of LayerRoPE's improvement) | Pre-Norm | Peri-Norm |
|---|---|---|
| LayerRoPE | 3.0266 (100%) | 3.0228 (100%) |
| − Depth encoding, learned per-layer scalars (best of 5 initializations) | 3.0731 (34%) | 3.0473 (29%) |
| − Depth encoding, learned shared scalars (best of 11 initializations) | 3.0788 (26%) | 3.0530 (13%) |
| − Depth encoding | 3.0921 (7%) | 3.0573 (1%) |
| Baseline + learned per-layer output gate | 3.0905 (9%) | ≡ Baseline |
| Baseline | 3.0969 | 3.0575 |
Peri-Norm's output normalization already gates every block, so its baseline includes any benefit of gating.
Among standard approaches, we compare against LayerScale, SkipInit (the learned scalar gate of ReZero), Scaled Initialization, Depth-Scaled Initialization and Layer-Norm Scaling, each with its learning rate tuned until the optimum is bracketed (Updated in Section 4.6 and Appendix D.3).
| Validation loss | Pre-Norm | Layer-Norm Scaling | Scaled Init. | DS-Init | LayerScale | SkipInit | LayerRoPE |
|---|---|---|---|---|---|---|---|
| 58M | 3.172 | 3.145 | 3.169 | 3.167 | 3.167 | 3.158 | 3.118 |
| 134M | 2.908 | 2.858 | 2.892 | 2.899 | 2.910 | 2.910 | 2.830 |
We observe that none of the standard residual gating and depth-dependent scaling baselines are able to recover LayerRoPE's performance or its learning rate stability (LayerRoPE trains stably at 4–8× higher rates with wider and lower basins). Simple layer-wise scaling approaches thus cannot replicate either its convergence or its learning-rate stability benefits.
The depth-scaling experiments. The experiments up to 512 layers are conducted with a hidden dimension of 128 and models ranging from approximately 18M to 111M parameters. While these experiments demonstrate interesting behavior at extreme depth, it is unclear whether similar advantages would hold for deeper models with more typical hidden dimensions and parameter counts. The main model-scaling experiments also stop at 1.3B parameters.
We agree that our depth scaling experiment, while demonstrating useful extreme depth:width ratios, do not show these extreme depths for wider models. To test typical widths on varying depths, we additionally evaluate a tuned LayerRoPE on DepthBench, which sweeps various width:depth ratios at a fixed 400M parameters.
| Method (DepthBench, 400M) | Mean loss over the five shapes |
|---|---|
| HC | 2.712 |
| mHC | 2.719 |
| Attention Residuals | 2.729 |
| Block Attention Residuals | 2.724 |
| LayerRoPE + Pre-Norm | 2.715 |
| LayerRoPE + Block Attention Residuals | 2.680 |
We observe that a tuned LayerRoPE applied to a Pre-Norm backbone, outperforms mHC and Attention Residuals on average, along with all standard single-stream methods. Further, when applied to Block Attention Residuals, we show that LayerRoPE siginificantly advances the loss-compute Pareto frontier. This is updated in the paper in Section 4.5 (details in Appendix D.4).
In looped models, which apply up to 60 blocks per forward pass at width 1280 (sharing weights across loop iterations), LayerRoPE also improves Parcae at every scale (Section 4.4).
Indeed our model scaling experiments stop at 1.3B, but we do also train every method at 6.7B on a reduced budget and show that LayerRoPE trains stably and outperforms every baseline with better convergence and downstream performance. Nevertheless, we agree, training very deep models at typical widths remains beyond our compute budget.
We thank the reviewer again for their helpful comments, and hope these responses help address their questions.
Thank you for the responses and additional results. My questions have been resolved, and I maintain my positive score.
Author Response (1/2)
We thank the reviewer immensely for taking the time to go through our work, and greatly appreciate their positive comments. We also appreciate the time to address these questions.
We understand the reviewer's concern. We mean depth encoding in a strict sense here: that the magnitude and directional shifts of the norm weights γℓ encode depth, or in other words are depth-conditioned. To make this clear, we have accordingly revised the paper to say depth encoding rather than depth-positional encoding.
To support our claim of depth encoding in LLMs: We now include a comprehensive analysis in Section 2 (detailed in App B.3) that quantifies the depth encoding in norm weights with a standard probing approach. We fit how γℓ varies with depth on all other layers, decode the held-out layer's depth, and compare carefully against a controlled null using a permutation-test. We find that the direction or magnitude of γℓ alone can reliably recover held-out layers' depths, with the direction alone recovering depth to within a median of 1.1 layers (3.8% of depth) across the 16 LLMs (p<10−4 in every model), compared to 11.1 for the null. This is included in Section 2 (Figure 2c), with details in Appendix B.3 (Figures 13–14, Tables 7–8).
The depth encoding is therefore found to be present in all 16 open-source LLMs across 9 families.
We also don’t believe this necessarily contradicts the interpretation of “differences in scaling requirements of individual layers”. This depth-conditioning may well be emergent in order to account for these very scaling requirements with depth, that occur due to either the independent or dependent layer-wise contributions of computational blocks to the stream. Time permitting, we would be very interested in the Reviewer’s thoughts on this. The underlying reasons for this emergence of depth encoding are a curious point, and likely worth studying further.
Whatever its origin, we see that encoding depth into the gains is what matters for training. Giving each layer its own learned scale recovers at most about a third of LayerRoPE's improvement, while removing the depth code loses 93–99% of it (see depth encoding ablation Table below).
We understand the reviewer's concern. To be clear, LayerRoPE proposes both a novel magnitude and rotation depth encoding in the norm weights. The primary novelty is in fact the core idea of depth encoding, whether through magnitude or rotation or both, rather than the rotary formulation alone, with the full LayerRoPE combining magnitude and rotational depth encoding performing the best (Table 5b, formerly Table 4). Rotating with RoPE's spectrum of frequencies also outperforms a single uniform angle (Appendix F.4).