acceptodds
Under review as a conference paper at ICLR 2027

LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition

32 views

Abstract

As data propagates through a Transformer, the norm of its hidden states grows by orders of magnitude with depth, a phenomenon framed as a “curse of depth” and nearly universally treated as a pathology to be suppressed. We take the opposite view. Across 16 pre-trained LLMs from 9 families, spanning dense, mixture-of-experts and hybrid architectures and Pre-, Peri- and Post-Norm designs, we find that this growth reflects an emergent depth-positional encoding, carried by the only learned per-layer gain on the residual stream, the normalization weight : with depth, grows in magnitude and rotates in direction, jointly encoding the layer index. We make this depth-conditioned encoding explicit with LayerRoPE, an implicit analog of RoPE along the depth axis, which replaces all layerwise vectors with a single shared vector and depth-conditioned scalars, at a net reduction in parameters and change in FLOPs. Across a model ladder scaled up to B+ tokens, LayerRoPE consistently outperforms Pre-, Post- and Peri-Norm, Layer-Norm Scaling and DeepNet, reaching Pre-Norm's 1.3B loss with less compute; LayerRoPE is the only approach whose loss improves monotonically as depth scales to 512 layers. It improves learning-rate sensitivity by –, and transfers naively to and consistently improves looped latent models and Vision Transformers. Inspecting its learned schedule inverts the prevailing premise: LayerRoPE does not shrink the residual stream but widens it, damping what each block reads while amplifying what it writes. Depth stability, our results suggest, calls not for suppressing the residual stream, but for depth-conditioned regulation of the computational blocks it feeds.

Primary area:
foundation or frontier models, including LLMs
Keywords:
emergent depth-positional encoding, rotary position embeddings (RoPE), RMSNorm gain, residual stream, curse of depth, transformer depth scaling, activation variance, mitigating variance

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_kZSgOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the evolution of normalization weights across Transformer layers and observes depth-dependent changes in their magnitude and direction. Based on these observations, the authors propose LayerRoPE, which generates layer-wise normalization weights from shared parameters using depth-conditioned scaling and rotation. Experiments on language models, deep Transformers, looped models, and Vision Transformers show improvements in training convergence, learning-rate stability, and downstream performance.

Critical strengths

The paper provides extensive experiments across different model scales, depths, and architectures. The improvements in learning-rate stability and depth scaling are particularly interesting. LayerRoPE is relatively simple and introduces almost no additional computational overhead. The results also show improvements across both Pre-Norm and Peri-Norm architectures. The analysis of normalization weights across pretrained LLMs provides an interesting motivation for the proposed method. The transfer results on looped models and ViTs are also encouraging.

Critical weaknesses
  1. Normalization weights as depth-positional encoding needs further justification. The authors observe that normalization weights change in magnitude and direction across layers and interpret this as an implicit depth-positional encoding. However, since each Transformer layer already has its own parameters and computational role, such depth-dependent variations are not necessarily evidence of positional encoding. I am not fully convinced that the observed patterns indicate an encoding mechanism rather than differences in the scaling requirements of individual layers. Some additional analysis would help clarify this interpretation.

  2. Rotation component. Table 4 shows that magnitude-only scaling already improves performance considerably, while adding rotation provides a smaller additional gain. This raises the question of how much of LayerRoPE's improvement comes from depth-dependent scaling rather than the proposed rotary formulation. A comparison with simpler learnable layer-wise scaling schedules would help clarify the role of rotation.

  3. The depth-scaling experiments. The experiments up to 512 layers are conducted with a hidden dimension of 128 and models ranging from approximately 18M to 111M parameters. While these experiments demonstrate interesting behavior at extreme depth, it is unclear whether similar advantages would hold for deeper models with more typical hidden dimensions and parameter counts. The main model-scaling experiments also stop at 1.3B parameters.

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank the reviewer immensely for taking the time to go through our work, and greatly appreciate their positive comments. We also appreciate the time to address these questions.

Normalization weights as depth-positional encoding needs further justification. The authors observe that normalization weights change in magnitude and direction across layers and interpret this as an implicit depth-positional encoding. However, since each Transformer layer already has its own parameters and computational role, such depth-dependent variations are not necessarily evidence of positional encoding. I am not fully convinced that the observed patterns indicate an encoding mechanism rather than differences in the scaling requirements of individual layers. Some additional analysis would help clarify this interpretation.

We understand the reviewer's concern. We mean depth encoding in a strict sense here: that the magnitude and directional shifts of the norm weights γℓ\gamma_\ell encode depth, or in other words are depth-conditioned. To make this clear, we have accordingly revised the paper to say depth encoding rather than depth-positional encoding.

To support our claim of depth encoding in LLMs: We now include a comprehensive analysis in Section 2 (detailed in App B.3) that quantifies the depth encoding in norm weights with a standard probing approach. We fit how γℓ\gamma_\ell varies with depth on all other layers, decode the held-out layer's depth, and compare carefully against a controlled null using a permutation-test. We find that the direction or magnitude of γℓ\gamma_\ell alone can reliably recover held-out layers' depths, with the direction alone recovering depth to within a median of 1.1 layers (3.8% of depth) across the 16 LLMs (p<10−4p \lt 10^{-4} in every model), compared to 11.1 for the null. This is included in Section 2 (Figure 2c), with details in Appendix B.3 (Figures 13–14, Tables 7–8).

The depth encoding is therefore found to be present in all 16 open-source LLMs across 9 families.

We also don’t believe this necessarily contradicts the interpretation of “differences in scaling requirements of individual layers”. This depth-conditioning may well be emergent in order to account for these very scaling requirements with depth, that occur due to either the independent or dependent layer-wise contributions of computational blocks to the stream. Time permitting, we would be very interested in the Reviewer’s thoughts on this. The underlying reasons for this emergence of depth encoding are a curious point, and likely worth studying further.

Whatever its origin, we see that encoding depth into the gains is what matters for training. Giving each layer its own learned scale recovers at most about a third of LayerRoPE's improvement, while removing the depth code loses 93–99% of it (see depth encoding ablation Table below).


Rotation component. Table 4 shows that magnitude-only scaling already improves performance considerably, while adding rotation provides a smaller additional gain. This raises the question of how much of LayerRoPE's improvement comes from depth-dependent scaling rather than the proposed rotary formulation.

We understand the reviewer's concern. To be clear, LayerRoPE proposes both a novel magnitude and rotation depth encoding in the norm weights. The primary novelty is in fact the core idea of depth encoding, whether through magnitude or rotation or both, rather than the rotary formulation alone, with the full LayerRoPE combining magnitude and rotational depth encoding performing the best (Table 5b, formerly Table 4). Rotating with RoPE's spectrum of frequencies also outperforms a single uniform angle (Appendix F.4).

AuthorsOpenReview ↗

Author Response (2/2)

A comparison with simpler learnable layer-wise scaling schedules would help clarify the role of rotation.

Ablating against carefully tuned layer-wise scaling in our LayerRoPE form, as well as tuned standard layer-wise scaling approaches, shows that depth encoding is a consistently more useful inductive bias. In LayerRoPE's own form, we remove the depth encoding and instead learn scalars shared across layers or free at every layer (which can express any layer-wise scaling schedule), and carefully sweep 16 initializations of these learned scalars per backbone (Appendix F.2):

Validation loss at 134M (share of LayerRoPE's improvement)Pre-NormPeri-Norm
LayerRoPE3.0266 (100%)3.0228 (100%)
− Depth encoding, learned per-layer scalars (best of 5 initializations)3.0731 (34%)3.0473 (29%)
− Depth encoding, learned shared scalars (best of 11 initializations)3.0788 (26%)3.0530 (13%)
− Depth encoding3.0921 (7%)3.0573 (1%)
Baseline + learned per-layer output gate3.0905 (9%)≡ Baseline
Baseline3.09693.0575

Peri-Norm's output normalization already gates every block, so its baseline includes any benefit of gating.

Among standard approaches, we compare against LayerScale, SkipInit (the learned scalar gate of ReZero), Scaled Initialization, Depth-Scaled Initialization and Layer-Norm Scaling, each with its learning rate tuned until the optimum is bracketed (Updated in Section 4.6 and Appendix D.3).

Validation lossPre-NormLayer-Norm ScalingScaled Init.DS-InitLayerScaleSkipInitLayerRoPE
58M3.1723.1453.1693.1673.1673.1583.118
134M2.9082.8582.8922.8992.9102.9102.830

We observe that none of the standard residual gating and depth-dependent scaling baselines are able to recover LayerRoPE's performance or its learning rate stability (LayerRoPE trains stably at 4–8× higher rates with wider and lower basins). Simple layer-wise scaling approaches thus cannot replicate either its convergence or its learning-rate stability benefits.


The depth-scaling experiments. The experiments up to 512 layers are conducted with a hidden dimension of 128 and models ranging from approximately 18M to 111M parameters. While these experiments demonstrate interesting behavior at extreme depth, it is unclear whether similar advantages would hold for deeper models with more typical hidden dimensions and parameter counts. The main model-scaling experiments also stop at 1.3B parameters.

We agree that our depth scaling experiment, while demonstrating useful extreme depth:width ratios, do not show these extreme depths for wider models. To test typical widths on varying depths, we additionally evaluate a tuned LayerRoPE on DepthBench, which sweeps various width:depth ratios at a fixed 400M parameters.

Method (DepthBench, 400M)Mean loss over the five shapes
HC2.712
mHC2.719
Attention Residuals2.729
Block Attention Residuals2.724
LayerRoPE + Pre-Norm2.715
LayerRoPE + Block Attention Residuals2.680

We observe that a tuned LayerRoPE applied to a Pre-Norm backbone, outperforms mHC and Attention Residuals on average, along with all standard single-stream methods. Further, when applied to Block Attention Residuals, we show that LayerRoPE siginificantly advances the loss-compute Pareto frontier. This is updated in the paper in Section 4.5 (details in Appendix D.4).

In looped models, which apply up to 60 blocks per forward pass at width 1280 (sharing weights across loop iterations), LayerRoPE also improves Parcae at every scale (Section 4.4).

Indeed our model scaling experiments stop at 1.3B, but we do also train every method at 6.7B on a reduced budget and show that LayerRoPE trains stably and outperforms every baseline with better convergence and downstream performance. Nevertheless, we agree, training very deep models at typical widths remains beyond our compute budget.


We thank the reviewer again for their helpful comments, and hope these responses help address their questions.

Thank you for the responses and additional results. My questions have been resolved, and I maintain my positive score.

Reviewer_i4CCOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper reports the finding that the residual stream weights γ\gamma in the layer update with PreNorm

ht+1=ht+Fℓ(γℓ⊙N(hℓ))h_{t+1} = h_t + F_\ell( \gamma_\ell \odot N(h_\ell) )

tends to (1) magnify in size and (2) rotate with respect to ℓ\ell across wide range of LLMs. Based on this discovery, this paper reduces the training burden by pre-parametrizing the γℓ\gamma_\ell to express just that, in parcular of form γℓ[j]=γ∗Diag(exp⁡(r(ℓ))exp⁡(iθ(ℓ,j))\gamma_\ell[j] = \gamma * Diag(\exp(r(\ell)) \exp(i \theta(\ell, j )) Expanding the usual update into

$h_{t+1} = h_t + Diag(\gamma_{\ell, out})F_\ell( Diag (\gamma_{\ell, in} N(h_\ell) ) $

Critical strengths

This paper showcases a pattern of very typical Machine Learning Win-- discovering through the investigation of general networks the common parametric pattrens that the strong models are learning with the hardship of careful training, and then lifting that effort by pre-fulfilling the pattern with parametrization. The gain of prior constraining the functional search-space to what is to be eventually learned is computational advantage, as to be expected.

Critical weaknesses

To this reviewer this paper was presented in an unfriendly way because it saved the precise meaning of γ\gamma until late. Yes, it can be inferred from context and I was imagining what is to come as I was reading through the work, but it did require me to re-write the whole model to myself, because the paper neither comes with the architectural figure or the the explicit equation of how γ\gamma appears in the layer block. As one reviewer I am quite against the locally prevalent beliefs that everyone working in ML shall be familiar with specific calling of a variable based on ambiguous agreement, because such things are to change over time---and this paper itself may take part of it in the future. Identifying the exact meaning shall not be a task of a reader.

Also, while this paper offers very interesting discovery and leverages it in a canonical way, the paper does not seem to delve into theoretical maning of this discovery and the additional tricks introduced in the paper---which I wish not to be shoved off as a geek's complaint from a corner, in relation to the questions that I would like to comment separately below.

Questions for authors
  • Although the idea of reparametrization is very natural from the observation, the motivation of γin\gamma_{in}, γout\gamma_{out} is not too well motivated. I read that read/write interpretation of residual computation is the interpretation, but from what observations of the trained LLMs did this come into play, or from what theory-motivated inspiration? I believe the paper shines more if more can be given to why does it work? Why is input-only depth scaling insufficient alone?

  • Did the authors ablate on relating the parametrization of γin\gamma_{in} with the parametrization of γout\gamma_{out}? Has there been any additional parametric inspiration from the observation that γin↓\gamma_{in} \downarrow, γout↑\gamma_{out} \uparrow wrt ℓ\ell seems to work? What happens if γin⊙γout=c\gamma_{in} \odot \gamma_{out} = c? Any other rules?

4 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank the reviewer immensely for taking the time to go through our work, and greatly appreciate their positive comments. We also appreciate the time to address these questions.

To begin, we agree precisely with the Reviewer’s interpretation of the kind of typical “Machine Learning Win” this is. Indeed, we see this as conserving learning capacity by providing the network as an inductive bias the very structures that are emergent and it is forced to learn.

To this reviewer this paper was presented in an unfriendly way because it saved the precise meaning of γ\gamma until late. [...] the paper neither comes with the architectural figure or the the explicit equation of how γ\gamma appears in the layer block.

We thank the reviewer for their articulation of the necessity for the core ideas behind LayerRoPE to be outlined sooner and clearly. (It is indeed not the reader’s task to await clarity on these aspects). To that end, we’ve expended significant effort on presentation:

  1. Figure 1a was updated to clarify the LayerRoPE method and notation right away.
  2. γ\gamma is defined at first use: While we do retain the γ\gamma notation, we make sure to specify it as normalization weights throughout the paper. We also hope the immediate Figure 1a makes this clear to the reader.
  3. Section 3 gives the explicit block update (Equation 2). We also include explicitly how LayerRoPE is applied, for the reader's benefit.
  4. Sections 1 – 3 re-written. We have also worked quite a bit to make the introductory sections (Sections 1--3) easier to follow in our introduction of depth encoding and LayerRoPE. We hope the above would make a positive difference to the reader.

Below, we address the theoretical meaning of the read and write gains in our answers to the two questions.

Although the idea of reparametrization is very natural from the observation, the motivation of γin\gamma_{in}, γout\gamma_{out} is not too well motivated. I read that read/write interpretation of residual computation is the interpretation, but from what observations of the trained LLMs did this come into play, or from what theory-motivated inspiration? I believe the paper shines more if more can be given to why does it work? Why is input-only depth scaling insufficient alone?

We agree, and thank the reviewer for their suggestion on providing more insights into LayerRoPE, and the motivation behind both read and write depth encoded gains.

On the motivation for both γin\gamma_{in} and γout\gamma_{out}:

From observation. Firstly, as we now include in a quantitative analysis in Section 2 (detailed in App B.3) that quantifies the depth encoding in norm weights with a standard probe, we find that depth encoding is present in all models: Pre-Norm (read gains only), Post-Norm (write gains only) and Peri-Norm (both read + write gains), with both output gains and input gains encoding depth well. LayerRoPE therefore naturally motivates from this, and induces depth-encoding on both read γin\gamma_{in} and write γout\gamma_{out} gains.

Ablating why input-only depth encoding is insufficient. We directly ablated read-only and write-only depth-encoding (App. F.3), and show that conditioning only the reads costs performance. Further we show that without the write code, the read only gains no longer learn to damp inputs, and end nearly flat with depth, at 0.95–0.97 of the first layer's gain at 134M (Appendix F.3, Figure 30). Only conditioning both read+write sides, allows LayerRoPE to learn damped reads and amplified writes, enabling the various block I/O regulation and stability under residual variance growth benefits we discuss in the paper.

AuthorsOpenReview ↗

Author Response (2/2)

On why LayerRoPE works. Our ablations identify the improvements directly in the depth encoding itself. Without this depth encoding, we show that LayerRoPE loses 93–99% of its improvement. Depth-independent output gates, scalars (shared or free at every layer) tuned over 16 initializations, recover little of LayerRoPE’s performance (Section 5.2 and Appendices F.1–F.2). Note that free per-layer scalars can express every schedule LayerRoPE learns, so the advantage lies not in expressivity or residual gains/branch scalars, rather in the depth encoding itself, that induces a globally shared depth-dependent trajectory across the network.

Beyond these, the revised paper adds several analyses of how LayerRoPE and the depth code work:

  • Where the depth code lives (Appendix B.3 and C). In pre-trained LLMs about 99% of the dimensions of γℓ\gamma_\ell follow one shared depth trajectory, while the directional shift is concentrated in 1-5% of channels. This motivates generating every γℓ\gamma_\ell from one shared γ\gamma, a scalar magnitude schedule and a rotation.
  • Gating is not the source of LayerRoPE's performance (Appendix F.1). Learned output gains without depth conditioning recover little of LayerRoPE's improvement. On Peri-Norm, which already gates every block's output, 99% of LayerRoPE's improvement comes from its depth conditioning.
  • Other Baselines and Comparisons
    • Section 4.6 ,Appendix D.3: We compare LayerRoPE against five standard baselines with learnable residual gates, depth-dependent scalar scaling: LayerScale, SkipInit (a learned scalar on each branch, the gate ReZero uses), Scaled Initialization, Depth Scaled Initialization and Layer-Norm Scaling, and find that none reproduce LayerRoPE's performance or LR stability.
    • Section 4.5, Appendix D.4.1: We evaluate a tuned LayerRoPE on DepthBench, and see that it outperforms mHC and Attention Residuals on average, at Pre-Norm costs, and significantly advances the compute-loss pareto frontier when applied to a block Attention Residuals backbone.

Did the authors ablate on relating the parametrization of γin\gamma_{in} with the parametrization of γout\gamma_{out}? Has there been any additional parametric inspiration from the observation that γin↓\gamma_{in} \downarrow, γout↑\gamma_{out} \uparrow wrt ℓ\ell seems to work? What happens if γin⊙γout=c\gamma_{in} \odot \gamma_{out} = c? Any other rules?

Inducing γin↓\gamma_{in} \downarrow, γout↑\gamma_{out} \uparrow into the initialization. We test this directly in Appendix E.2.2, with 22 models at 134M. We set among a few parameterizations, the depth params βout=−βin\beta_{out} = -\beta_{in} such that this initializes γin⊙γout=c\gamma_{in} \odot \gamma_{out} = c at every depth. Indeed we find that initializing γin\gamma_{in} and γout\gamma_{out} to be opposed underperforms the default unopposed shared initialization of βout=βin\beta_{out} = \beta_{in} i.e. LayerRoPE's default. LayerRoPE thus trains best when both its reads and its writes start unopposed, damped with depth, and it learns to amplify its writes during training. In fact, if the reads start amplified (βin≥0.5\beta_{in} \ge 0.5), it degrades the model.

Analyzing γin⊙γout\gamma_{in} \odot \gamma_{out}. In Appendix E.1, we measure the effective read ×\times write gains and observe that while LayerRoPE amplifies each block's writes, by 22–4×4\times from the first layer to the last at 58M and 134M and by 66–25×25\times at 1.3B, its reads damp the net amplification by up to 7×7\times, so that the read ×\times write product grows by 1.51.5–3.5×3.5\times and 3.53.5–14×14\times with depth, respectively. Thus, write gains massively amplify residual variance, read gains damp the inputs, maintaining a controlled layer-wise net amplification.

Role of Depth slope β\beta. We exhaustively sweep the depth slope param β\beta of LayerRoPE over 14 values in [−1,1.5][-1, 1.5] at 9 learning rates (126 runs, on the 12-layer model of the depth study; Appendix E.2.1), and find that the slope β\beta significantly affects the quality of convergence in LayerRoPE; with a large effect on the convergence quality as we gradually sweep β\beta towards negative values of -0.5 to -1.0. These greatly improve over our default LayerRoPE results.

We hope these insights and ablations are interesting and useful!


We would also like to highlight other detailed analyses, results we have included to improve the work. These are updated in the General Response, if the reviewer may benefit from seeing this. Additionally, to support LayerRoPE's evaluations across varying width/depth configurations, we evaluated a tuned LayerRoPE on DepthBench. We observed that a tuned LayerRoPE outperforms mHC and Attention Residuals on average, at Pre-Norm costs, and significantly advances the compute-loss pareto frontier when applied to a block Attention Residuals backbone. (Section 4.5, Appendix D.4.1)

We thank the reviewer again for their thoughtful questions, which have improved the paper.

Reviewer_i4CCOpenReview ↗

Thank you for the response

Thank you very much for accomodating my requests regarding the clarity. I have confirmed the changes and the saw that the clarity improved. For the later reference for ICLR submission when the in-midst editings are allowed during the rebuttal, however, I very much appreicate that, for the benefit of the entire community of the reviewers, the authors revise the manuscript in colors so that the reviewers can identify the changes with more ease. I am raising my score.

We appreciate the reviewer's advice on employing color-coded edits in future ICLR submissions. This is noted, and we will be sure to do so.

We thank the reviewer again for their time and effort in reviewing our work, and for providing useful critique and feedback, which has undoubtedly helped improve the paper.

Reviewer_hGQnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LayerRoPE, a depth-dependent parameterization of normalization gains for Transformers. The authors observe that normalization weights in pretrained LLMs often exhibit structured changes in both magnitude and direction across layers. Based on this observation, LayerRoPE replaces independent layer-wise gains with a shared vector modulated by depth-dependent scaling and rotation. Experiments show improvements in training loss, depth scaling, and learning-rate robustness, with additional results on looped language models and Vision Transformers.

Critical strengths
  1. Extensive experimental evaluation. The paper evaluates LayerRoPE across different model sizes, network depths, and learning rates. I particularly appreciate that the authors tune the learning rates for individual baselines rather than relying on a shared setting. The improvements are fairly consistent, and the experiments on looped models and ViTs provide further evidence that the method is not limited to a single architecture.
  2. A simple method with promising practical benefits. LayerRoPE introduces little computational overhead and slightly reduces the number of parameters. The improvements in convergence and learning-rate robustness are encouraging, especially given the simplicity of the modification. The analysis of normalization weights across pretrained LLMs is also interesting and offers a useful motivation for the method.
Critical weaknesses
  1. Insufficient evidence for the necessity of depth-wise rotary encoding.
    LayerRoPE shares similarities with existing residual scaling methods, such as LayerScale [1] and ReZero [2]. Table 4 shows that magnitude-only scaling already provides most of the improvement, while rotation adds a relatively small gain. Without comparisons to simpler methods, such as learnable residual gates or depth-dependent scalar scaling, it is difficult to determine whether the proposed rotary parameterization offers a meaningful advantage over existing approaches.
  2. The proposed mechanism is not fully validated.
    The authors introduce depth-dependent normalization weights as an implicit positional encoding, but their correlation with layer depth does not establish this functional role, especially since the pattern is not consistent across all pretrained models. Similarly, while LayerRoPE increases residual variance and improves convergence, the experiments do not establish a causal relationship between them. The current evidence supports the method's effectiveness more strongly than its proposed explanation.

[1] Touvron et al., Going Deeper with Image Transformers. ICCV 2021.

[2] Bachlechner et al., ReZero is All You Need: Fast Convergence at Large Depth. UAI 2021.

Questions for authors

A controlled comparison with simpler read/write scaling baselines, such as LayerScale-style gates or depth-dependent scalar gains, would help clarify the source of LayerRoPE's improvements and better establish its novelty.

6 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank the reviewer immensely for taking the time to go through our work, and greatly appreciate their positive comments. We also appreciate the time to comprehensively address these questions, and hope that these help conclusively support LayerRoPE’s claims.


Insufficient evidence for the necessity of depth-wise rotary encoding. LayerRoPE shares similarities with existing residual scaling methods, such as LayerScale [1] and ReZero [2]. Table 4 shows that magnitude-only scaling already provides most of the improvement, while rotation adds a relatively small gain. Without comparisons to simpler methods, such as learnable residual gates or depth-dependent scalar scaling, it is difficult to determine whether the proposed rotary parameterization offers a meaningful advantage over existing approaches.

A controlled comparison with simpler read/write scaling baselines, such as LayerScale-style gates or depth-dependent scalar gains, would help clarify the source of LayerRoPE's improvements and better establish its novelty.

We have added both kinds of comparisons.

We compare against five standard baselines with learnable residual gates, depth-dependent scalar scaling, with each carefully LR tuned: LayerScale, SkipInit (a learned scalar on each branch, the gate ReZero uses), Scaled Initialization, Depth Scaled Initialization and Layer-Norm Scaling.

Validation lossPre-NormLayer-Norm ScalingScaled Init.DS-InitLayerScaleSkipInitLayerRoPE
58M3.1723.1453.1693.1673.1673.1583.118
134M2.9082.8582.8922.8992.9102.9102.830

Learnable residual gates and depth-dependent scalar scaling baselines are unable to recover LayerRoPE's performance improvements. The primary novelty in LayerRoPE is in the core idea of inducing an explicit depth-encoding in the norm weights, whether through the proposed magnitude or rotary-style rotation schedules, and not simply its rotary formulation. We make this point clearer in the Introduction as well as Sections 1 and 3 of the paper. Comparing against tuned standard layer-wise scaling and learnable gate approaches like LayerScale, SkipInit, Scaled Initialization & Depth Scaled Initialization show that depth-encoding is a consistently more useful inductive bias, and that simple layer-wise scaling approaches cannot replicate its convergence benefits (or even other LR stability benefits). Note that LayerRoPE also reduces parameters compared to these methods. We have added these additional baseline comparisons to Section 4.6 (Table 4) with details and learning rate sweeps to Appendix D.3.

To clarify the important role depth encoding plays in LayerRoPE’s properties, we ablate the depth conditioning in LayerRoPE directly (below) and observe that removing depth encoding from LayerRoPE loses 93–99% of its performance gains over its Pre/Peri-Norm backbones.

We also then test whether learnable output gates or scalar gains can recover LayerRoPE's performance. We also check whether with learnable per-layer scalars, LayerRoPE without depth conditioning can recover the benefits of depth-conditioning. We find that without depth encoding, neither learnable gates, nor shared scalar gains help, nor per-layer freedom on “LayerRoPE without depth conditioning”, can recover LayerRoPE's performance.

Validation loss at 134M (share of LayerRoPE's improvement)Pre-NormPeri-Norm
LayerRoPE3.0266 (100%)3.0228 (100%)
− Depth conditioning3.0921 (7%)3.0573 (1%)
− Depth conditioning + per-layer scalars3.0819 (21%)3.0546 (8%)
Baseline3.09693.0575
+ Per-layer learnable output gates3.0905 (9%)≡ Baseline
+ Output gate and scalar gains shared across layers3.0881 (13%)not run

We have included several additional ablations to the Appendix. Appendix F.2 ablates layer-wise and shared learnable scalars on “LayerRoPE without depth conditioning” with exhaustive tuning to see if depth encoding could be recovered. In Appendix F.1, similarly, we have added ablations that carefully verify if output gains and global scalars can recover LayerRoPE’s performance. Note that per layer scalars are able to freely express any depth-dependent schedule. We show summarily that none of these are able to recover LayerRoPE’s performance with depth encoding. Also note that Peri-Norm already has a learned gain on every block's output, yet LayerRoPE improves it and 99% of that improvement comes from depth conditioning.

We believe this reliably shows the importance of depth encodings like LayerRoPE, that induce a globally shared depth-dependent trajectory across the network. And that is not recoverable through gains, per layer scalars or otherwise standard baselines. We have updated Sections 5.2, Appendices F.1, F.2 and F.3 in the paper accordingly.

AuthorsOpenReview ↗

Author Response (2/2)

The proposed mechanism is not fully validated. The authors introduce depth-dependent normalization weights as an implicit positional encoding, but their correlation with layer depth does not establish this functional role, especially since the pattern is not consistent across all pretrained models.

Depth can be decoded from the norm weights of every model. We now include a comprehensive analysis in Section 2 (detailed in App B.3) that quantifies the depth encoding in norm weights with a standard probing approach. We fit how γℓ\gamma_\ell varies with depth on all other layers, decode the held-out layer's depth, and compare carefully against a controlled null using a permutation-test. We find that the direction or magnitude of γℓ\gamma_\ell alone can reliably recover held-out layers' depths, with the direction alone recovering depth to within a median of 1.1 layers (3.8% of depth) across the 16 LLMs (p<10−4p \lt 10^{-4} in every model), compared to 11.1 for the null.

The depth encoding is therefore present in all 16 models, Pre-, Peri- and Post-Norm.

On the functional point. We agree that the probe itself shows that depth is encoded across all LLMs, not that the networks use it. However, we provide functional evidence from our proposed depth encoding in LayerRoPE, where we show that simply ablating LayerRoPE's depth conditioning loses 93–99% of its improvement, and neither baselines with depth-dependent scalars, nor ablating with learned gains or scalars can replace it.

Similarly, while LayerRoPE increases residual variance and improves convergence, the experiments do not establish a causal relationship between them. The current evidence supports the method's effectiveness more strongly than its proposed explanation.

We agree that our experiments do not isolate the residual variance itself as the cause of improved convergence. Our claim, as we clarify in Section 6, is that depth stability "depends less on how large the residual stream grows than on whether each block can scale what it reads and writes, to its depth". Therefore, we argue that the residual stream's variance growth need not be treated as a pathology to suppress, rather that through LayerRoPE the wider stream is enabled through its learned regulation of each block's I/O (reads and writes). We now clarify this more carefully in Section 1. We also support this through updated analysis: At 1.3B, LayerRoPE's write gains grow up to 25× across its layers while its read gains shrink, widening the stream (Appendix E, Section 5.1). The damped reads we show contain this write amplification by reducing the net read × write amplification by up to 7× (Appendix E.1). Finally, we also show that removing this depth code from either side breaks this amplification + shrinking I/O capability and costs performance (Appendix F.3, Figure 30).


We would also like to highlight other detailed analyses, ablations we have included to improve the work. These are updated in the General Response, if ther reviewer may benefit from seeing this. Additionally, we would like to add that in order to support LayerRoPE's evaluations on varying width/depth configurations, we evaluated a tuned LayerRoPE on DepthBench. We observed that a tuned LayerRoPE outperforms mHC and Attention Residuals on average, at Pre-Norm costs, and significantly advances the compute-loss pareto frontier when applied to a block Attention Residuals backbone. (Section 4.5, Appendix D.4.1)

We thank the reviewer again for their helpful comments, and hope our efforts with these responses help address their question.

Thanks for the authors' response. I'll keep my score.

Comments on the paper

3 comments

Dear Authors and Reviewers,

As we approach the end of the discussion period, the authors have not yet submitted a rebuttal. While the initial ratings are positive, the concerns raised by the reviewers do not resolve themselves automatically. I urge the authors to engage with the feedback and address the reviewers' questions thoroughly before the deadline.

Reviewers, please keep an eye out for a late rebuttal and respond accordingly. If no response is provided by the close of the discussion period, please re-evaluate whether your scores remain appropriate in light of the unaddressed concerns.

Best,

AC

Dear Area Chair,

We were just about to release a response to the Reviewers.

Since reviewer concerns were largely around supportive analyses and additional baselines/comparisons, we have taken the extra time to include both comprehensively.

We hope the comprehensive response would provide conclusive evidence to support our work.

Regards,

Authors

AuthorsOpenReview ↗

General Response

We thank all three reviewers for all their positive comments. As well as for the time and effort taken in going through our work. We really do appreciate it.

The main requests from reviewers broadly were:

  • comparisons with simpler residual-scaling baselines
  • evidence that depth encoding, rather than per-layer scaling or gating, is behind LayerRoPE's performance
  • evidence that norm weights encode depth beyond a qualitative correlation
  • the motivation for depth encoding, and the use of both read and write side gains
  • a clearer presentation
  • results on varying depth with more typical widths.

We have revised the paper substantially in response:

  1. Quantifying the depth encoding (Section 2, Fig. 2c and App. B.3). We now carefully probe whether a layer's depth can be read from its norm weights alone. We fit how γℓ\gamma_\ell varies with depth on all other layers, decode the held-out layer's depth, and compare against a null through a well controlled permutation test. Across all 16 LLMs, both direction and magnitude of γℓ\gamma_\ell can alone recovers a held-out layer's depth, with direction alone recovering to within a median of 1.1 layers, against 11.1 with shuffled depths (p<10−4p \lt 10^{-4} in every model).
  2. Ablating the depth encoding (Section 5.2, Appendix F).
    • Ablating depth-conditioning in LayerRoPE loses 93-99% of its performance improvement.
    • Gating without depth conditioning, depth-independent scalars (shared across layers or per layer) are all unable to recover LayerRoPE's performance. Despite when free per layer, they can express every schedule LayerRoPE learns.
    • Depth encoding is needed on both the reads and the writes of every block (Appendix F.3).
  3. How LayerRoPE regulates each block (Section 5.1; Appendix E, F.3). Analysis showing LayerRoPE learns to damp what each block reads while amplifying what it writes, widening the residual stream, and only conditioning both sides learns this. We also show that reads damp the growth of the writes (App. E.1), and initializing reads and writes as reciprocals (γin⊙γout=c\gamma_{in} \odot \gamma_{out} = c) underperforms LayerRoPE's default (Appendix E.2.2).
  4. Additional baselines (Section 4.6, Appendix D.3). We add Scaled Initialization, DS-Init, LayerScale and SkipInit (the per-branch scalar gate of ReZero). Each is tuned until its optimum is bracketed. LayerRoPE outperforms all four at 58M and 134M while also training stably at 4–8× higher rates.
  5. Trading width for depth on DepthBench (Section 4.5, Appendix D.4). At 400M parameters and five shapes of varying width/depth ratios, a tuned LayerRoPE applied to a Pre-Norm backbone outperforms mHC and Attention Residuals on average at Pre-Norm's cost. On a Block Attention Residuals backbone, LayerRoPE significantly advances the loss-compute Pareto frontier. It adds no memory I/O, activation memory or pipeline communication to its backbone, and no FLOPs under DepthBench's accounting. | Method (DepthBench, 400M) | Mean loss over the five shapes | |:---|---:| | HC | 2.712 | | mHC | 2.719 | | Attention Residuals | 2.729 | | Block Attention Residuals | 2.724 | | LayerRoPE + Pre-Norm | 2.715 | | LayerRoPE + Block Attention Residuals | 2.680 |
  6. Presentation.
    • Figure 1a now illustrates the LayerRoPE block and its notation.
    • γ\gamma is now carefully introduced at first use as the normalization weight
    • Section 3 includes an explicit block update for LayerRoPE (Equation 2).
    • Section 2 is rewritten around the quantitative analysis of depth encoding, and we now say depth encoding rather than depth-positional encoding.
    • The ablations (above) are all consolidated in Section 5.2 and Appendix F
    • The title now states the contribution more clearly: LayerRoPE: Depth Encodings Enable Deeper, More Stable and Efficient Transformers.

We thank the reviewers again.


We believe our work's finding of an emergent depth encoding in LLMs (across all 16 open-source LLMs from 9 families, dense, MoE, hybrid, and Pre-/Peri-/Post-Norm designs), our proposed explicit form of depth-encoding in LayerRoPE, and its broad benefits to:

  • Compute efficiency

    • LayerRoPE + Pre-Norm outperforms mHC, Attention Residuals at Pre-Norm costs
    • LayerRoPE + Block Attention Residuals sets a new pareto frontier of loss-compute efficiency
    • 3.4x Compute efficiency gain over backbone at 4x Chinchilla
  • Depth scaling upto 512 layers with near-monotonic improvements

  • 3-10x higher learning-rate stability with wider basins

  • Consistent improvements to Looped Latent models and ViTs

We believe are significant scientific contributions and will be of interest to the ICLR community.

Our work would therefore benefit greatly from the reviewer's support in highlighting it to the ICLR community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.