acceptodds
Under review as a conference paper at ICLR 2027

Loop-μP: Residual Scaling and Hyperparameter Transfer in Looped Transformers

Abstract

Looped Transformers have drawn growing attention because looping improves multi-step reasoning and increases the executed depth of a model without adding parameters. They achieve this by reusing a stack of layers for loops. However, increasing the loop count not only destabilizes training but also shifts the best hyperparameters, so they must be searched again for every loop count. Inspired by Depth-P (Yang et al., 2024a), which multiplies each residual branch by for depth so that a learning rate tuned on a shallow untied network transfers to a deep one, we study how the residual branches and the learning rate should scale with the loop count . We find that loops cannot simply be treated as extra depth, because the loops share weights. At initialization, distinct layers add nearly uncorrelated increments to the residual stream, so their sum grows as , whereas a shared layer adds positively correlated increments at every loop, so their sum grows linearly in . For a single layer looped times, we show in the infinite-width limit that multiplying the residual branch by and keeping the learning rate independent of is the only parameterization with both a stable forward pass and maximal residual-stream updates. We then generalize the analysis to a stack of distinct layers looped times to obtain a parameterization we call Loop-P, in which the loop factor combines with the depth factor . Empirically, we find in language modeling that the base learning rate tuned on the smallest model stays best under Loop-P for loop counts up to and executed depths up to layers. These results make Loop-P a practical recipe for scaling looped Transformers.

Primary area:
optimization
Keywords:
looped transformers, weight sharing, residual scaling, training stability, hyperparameter transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_V29mOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies how to scale the residual branch and its learning rate with respect to the loop count in the looped transformer architecture. The main idea is since loops shares weights, residual increments have positive correlation and thus grow linearly in N, unlike L\sqrt{L} in untied architecture. Based on this, proposes Loop-μ\muP which multiplies each residual branch by 1/N on top of Depth-μ\muP and keeps the learning rate indepentdent of N. Emperical evaluations on FineWeb-Edu shows that the learning rate tuned on the smallest model transfers up to N=8, 384 executed layers.

Critical strengths
  1. The motivation is clear and intuitive. Figure 3 shows that residual increments of a looped architecture have positive correlation while untied architecture are near orthogonal, which clearly explains why loops cannot be treated equal as extra depth.

  2. The theory and its supporting experiments are consistent. Section 4.1 checks both predictions of Section 3.2; a bounded forward pass and order-one change after an optimizer step, and only N−1N^{-1} satisfies both conditions. (Figure 2 and 3)

  3. The recipe is practical. Table 1 shows only the residual multiplier have to change from the untied recipe, and Figure 4 shows that a model tuned on N=1 can be looped without further LR sweep. It makes the design simple and effective.

Critical weaknesses
  1. Only a single seed was used for Figure 4 (mentioned in Appendix E.3), and the learning rate grid is somewhat coarse. It seems important since the curves in Figure 4 are quite flat near the optimum, therefore hard to conclude that the best LR truly stays fixed across N. Also the experiments are relatively small scale (10B tokens, 48 layers) so it is unclear whether it transfers to a larger scale.

  2. The residual multiplier depends on N. Running more loops at test-time is one of the main motivations looped architectures have, also mentioned in Section 2 (Geiping et al., 2025). However, this paper does not discuss how Loop-μ\muP performs when the test-time loop count differs from its training.

Questions for authors
  1. What is the seed variance for Figure 4? Does the best LR still holds with multiple seeds and a finer grid near the optimum?

  2. What happens if the test-time loop count differs from training? Should the multiplier be fixed to the training N, or should it be changed?

4 comments
AuthorsOpenReview ↗

Rebuttal 1/n

We thank Reviewer V29m for finding our motivation clear and intuitive, our derivations well aligned with the experiments, and our recipe simple and practical. We reply to each point below.

W1 & Q1: Multi-seed robustness, learning-rate grid resolution, and model scale

(1) Multiple seeds, grid resolution, and flat curves. General Response G2 reports our four-seed results and grid resolution check: the four-seed mean stays within 0.00100.0010 of seed 0 across N∈{1,2,4,8}N \in \lbrace 1, 2, 4, 8\rbrace, Loop-μ\muP beats the best alternative loop factor by 2.32.3–6.56.5 seed stds at every N>1N > 1, and at N=8N = 8 our 1.21.2–1.25×1.25\times grid (matching the finest grid in Tensor Programs V [1]) keeps η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3} optimal over four seeds relative to both neighbors (1.01.0 and 1.5×10−31.5 \times 10^{-3}). Consistently, across all 20 Loop-μ\muP sweeps in the paper and rebuttal (Figure 4 and G3), 19 out of 20 attain their discrete grid minimum at η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3} (and the single exception, width 10241024 at N=2N = 2, is within 0.0013<1σ0.0013 < 1\sigma).

(2) Model scale. We agree that evaluating larger scales is important, as also noted in Appendix D. Within our compute budget, General Response G3 reports two new scaled-up experiments. In a deeper stack (L=96L = 96, about 780M parameters, up to N=8N = 8 loops and 768768 executed layers), η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3} remains optimal across all loop counts. In a wider model (width 10241024 at L=12L = 12), the learning rate tuned at width 768768 and N=1N = 1 remains optimal on the grid for N=1,4,8N = 1, 4, 8, and reusing it at N=2N = 2 incurs a loss difference of only 0.00130.0013 (below the 0.00190.0019 width-768 seed std).

AuthorsOpenReview ↗

Rebuttal 2/n

W2 & Q2: Varying loop count at test time (1/Ntest1/N_{\mathrm{test}} vs. 1/Ntrain1/N_{\mathrm{train}})

(1) Which multiplier to use. Our continuum-limit analysis motivates rescaling the branch multiplier to 1/Ntest1/N_{\mathrm{test}}. Under Loop-μ\muP, NN loops discretize a fixed integration horizon [0,1][0, 1] with step size 1/N1/N (Claim 3, Lemma 5). Rescaling to 1/Ntest1/N_{\mathrm{test}} keeps this total horizon fixed while taking finer steps, whereas keeping 1/Ntrain1/N_{\mathrm{train}} stretches the horizon by Ntest/NtrainN_{\mathrm{test}}/N_{\mathrm{train}} and pushes the final residual norm outside the scale seen during training.

(2) Test at L=12L = 12. Evaluating our fixed-NN Loop-μ\muP checkpoints with additional loops at test time confirms this prediction.

NtrainN_{\mathrm{train}}Test-time multiplierNtest=NtrainN_{\mathrm{test}} = N_{\mathrm{train}}+1 loop+2 loops+3 loops
44Fixed 1/Ntrain1/N_{\mathrm{train}}2.93402.93402.94052.94053.03843.03843.13393.1339
44Rescaled 1/Ntest1/N_{\mathrm{test}}2.93402.93402.9293\mathbf{2.9293}2.9458\mathbf{2.9458}2.9561\mathbf{2.9561}
88Fixed 1/Ntrain1/N_{\mathrm{train}}2.90782.90782.91012.91012.91782.91782.92122.9212
88Rescaled 1/Ntest1/N_{\mathrm{test}}2.90782.90782.9060\mathbf{2.9060}2.9063\mathbf{2.9063}2.9130\mathbf{2.9130}

Keeping 1/Ntrain1/N_{\mathrm{train}} fixed degrades validation loss rapidly as loops are added (rising by 0.2000.200 at Ntrain=4N_{\mathrm{train}} = 4 with 3 extra loops). Rescaling to 1/Ntest1/N_{\mathrm{test}} keeps the loss stable (within 0.0050.005 across 1 to 3 extra loops at Ntrain=8N_{\mathrm{train}} = 8 and within 0.0220.022 at Ntrain=4N_{\mathrm{train}} = 4) and even yields slight improvements for 1–2 extra loops.

(3) Why the gains stop after one or two extra loops. Because our experiments focus on hyperparameter transfer across training runs, each model is trained at a fixed NN, and the final pass specializes to the end of the loop (Figure 5). To keep improving over more extra loops at test time, models need variable-NN training (such as randomized loop counts or intermediate supervision [2, 3, 4]) rather than training at a single fixed NtrainN_{\mathrm{train}}. Importantly, because Loop-μ\muP's learning rate does not depend on NN (γN=0\gamma_N = 0), it fits variable-NN training naturally: when a different loop count NN is sampled on each step or batch, only the forward branch multiplier 1/N1/N changes, with no need to rescale the optimizer learning rate from step to step. We leave it for future work.

Thank you again for the helpful feedback.

References

[1] Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., & Gao, J. (2022). Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466.

[2] Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B. R., Kailkhura, B., Bhatele, A., & Goldstein, T. (2025). Scaling up test-time compute with latent reasoning: A recurrent depth approach. In Advances in Neural Information Processing Systems (Vol. 38).

[3] Zhu, R.-J., Wang, Z., Hua, K., Zhang, T., Li, Z., Que, H., Wei, B., Wen, Z., Yin, F., Xing, H., Li, L., Shi, J., Ma, K., Li, S., Kergan, T., Smith, A., Qu, X., Hui, M., Wu, B., . . . Eshraghian, J. (2025). Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741.

[4] Prairie, H., Novack, Z., Berg-Kirkpatrick, T., & Fu, D. Y. (2026). Parcae: Scaling laws for stable looped language models. arXiv preprint arXiv:2604.12946.

Thank you for the detailed response and the additional experiments. The multi-seed results, the larger-scale sweeps, and the test-time analysis address my questions, and I agree that the grid spacing near the optimum is reasonable. I will maintain my rating of Weak Acceptance.

Thank you again for your constructive feedback!

Reviewer_wqGfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies residual scaling and learning-rate transfer in looped Transformers, which reuse a stack of layers multiple times. It proposes Loop-μ\muP, which scales each residual branch by 1/N on top of an existing depth parameterization and keeps the learning rate independent of the loop count N. The authors establish stability and nontrivial-update results for linear networks trained with SGD, and discuss extensions to nonlinear networks and Adam under additional assumptions. Language-modeling experiments on FineWeb-Edu show that the same base learning rate remains optimal among the tested values across loop counts up to 8 and executed depths up to 384 layers.

Critical strengths
  1. The paper addresses training instability and learning-rate retuning as loop counts increase. Loop-μ\muP provides a simple residual-scaling rule that is easy to apply to looped Transformers. This is a important research and practical question.

  2. The paper derives scaling conditions for linear networks trained with SGD and tests the proposed mechanism through residual-stream measurements and learning-rate sweeps.

Critical weaknesses
  1. The presentation is difficult to follow in places. In the introduction, notation is introduced abruptly without sufficient explanation. For example, the indices rr and ℓ\ell are used without clearly stating that they refer to loop iterations and distinct layers, respectively. Defining the notation when it first appears would make the argument easier to follow.

  2. The incremental contribution over existing depth parameterizations needs clearer justification. The practical recipe mainly adds a 1/N residual multiplier while retaining the existing width- and depth-dependent learning-rate rules. Although establishing transfer across loop counts is meaningful, the paper could better explain which theoretical techniques introduced by weight sharing require new arguments beyond the Depth-μ\muP framework.

  3. The validation of hyperparameter transfer remains limited. All language-modeling experiments use a fixed width of 768, leaving joint transfer across width and loop count untested.

Questions for authors
  1. I find two claims in introduction which maybe strong:

(1) Regarding the statement that 'all N loops receive the same weight update, so the changes it causes in their residual increments are also positively correlated', why does using the same weight update imply positively correlated changes across loops, not negatively? Figure 3 (left) measures the alignment of residual increments at initialization, rather than the alignment of their update-induced changes. Could you clarify the additional conditions required and directly test this claim?

(2) Regarding the statement that 'the increments of L distinct layers are independent', why independent? Since each layer’s input depends on previous layers, independent weights do not generally imply independent increments. Do you mean uncorrelated or asymptotically orthogonal increments at initialization, rather than statistically independent increments? Could you clarify the extent of assumptions?

  1. Which parts of the theoretical analysis require fundamentally new arguments compared with Depth-μ\muP? In particular, could you clarify what is needed to establish an N-independent learning rate after introducing the 1/N residual multiplier?

  2. Appendix I introduces additional assumptions for nonlinear networks and Adam. Which of these assumptions are verified for the Transformer architecture used in the experiments, particularly with attention and RMSNorm, and which remain unverified? How does this affect the scope of the claimed uniqueness of Loop-μ\muP?

  3. All language-modeling experiments use a fixed width of 768. Does the base learning rate tuned at width 768 and one loop remain effective when transferred to a wider model with more loops, following the scaling rules in Table 1?

  4. Compared with alternative scaling rules, does Loop-μ\muP provide a wider range of well-performing learning rates or a smaller validation-loss penalty when transferring rather than retuning the learning rate across loop counts?

6 comments
AuthorsOpenReview ↗

Rebuttal 1/n

We thank Reviewer wqGf for recognizing the value of this problem and the close agreement between our derivations, residual-stream measurements, and learning-rate sweeps. We reply to each point below.

W1: Abrupt introduction of notation and presentation clarity

We revised both the Introduction and Section 3 to define every index where it first appears and cut unnecessary symbols (detailed in General Response G1, and highlighted in blue in the updated PDF). Specifically: (1) the Introduction (p. 2) now defines rr, ℓ\ell, and xrx^r at first use; (2) Section 3 drops four intermediate scale symbols (βN,sN,ρN,χ0\beta_N, s_N, \rho_N, \chi_0), renames the loss to Lt\mathcal{L}_t to avoid colliding with ℓ\ell, and opens Section 3.2 (p. 5) with a plain-English walkthrough before the equations; and (3) Table 2 (p. 16) summarizes all main-text notation in one place.

W2 & Q2: Contribution beyond Depth-μ\muP, new proof arguments, and NN-independent learning rate

(1) What changes compared with depth scaling. In untied networks, the increments of different layers are uncorrelated in the infinite-width limit, so the residual stream grows as L\sqrt{L}. Consequently, an entire interval αL∈[1/2,1]\alpha_L \in [1/2, 1] is both stable and nontrivial, and Depth-μ\muP [1] requires an additional criterion, feature diversity, to select αL=1/2\alpha_L = 1/2. With weight sharing, the increments of different loops are positively correlated, so the residual stream grows linearly in NN. Stability and nontriviality alone then uniquely select (αN,γN)=(1,0)(\alpha_N, \gamma_N) = (1, 0) (Appendix A.2, Claims 1–2).

Practically, this yields a simple 1/N1/N branch multiplier that composes directly with existing width and depth rules, allowing hyperparameters tuned at N=1N = 1 to transfer without modification. By contrast, treating loops as untied depth and applying the Depth-μ\muP factor N−1/2N^{-1/2} causes the forward norm to grow with NN (Figure 2(b)), leads to divergence at high learning rates, and yields higher validation loss (Figure 4(b)).

(2) Where the proofs need new arguments. In Depth-μ\muP, each layer has an independent weight matrix. In a looped network, the same matrix WW (and W⊤W^\top in the backward pass) is reused in every loop, and a single shared update ΔW\Delta W acts across all NN loops, coupling all forward and backward passes. Our proofs track these cross-loop correlations in the infinite-width limit and the N→∞N \to \infty continuum limit. The key technical step is showing that repeated passes do not cancel out. Specifically, internal preactivations remain nonvanishing (Lemma 7), and one shared update changes both the residual state and every internal preactivation by Θ(1)\Theta(1) (Proposition 8, Lemma 11).

In the revision, we expanded the derivations of the shared gradient, the single-step response, and the nonvanishing results in Appendices G–H, and clarified in Appendix A that feature learning requires the shared update to induce Θ(1)\Theta(1) changes in both the residual states and the internal preactivations.

(3) Why the learning rate stays independent of NN. Under 1/N1/N branch scaling (αN=1\alpha_N = 1), the 1/N1/N damping and the NN-fold weight reuse balance out, because the skip connection keeps cross-loop inputs and backpropagated signals positively aligned rather than canceling each other (see Q1 below):

  • For Adam: Adam normalizes each parameter's gradient to Θ(1)\Theta(1) regardless of scale, so an NN-independent learning rate (γN=0\gamma_N = 0) produces a parameter step ΔW\Delta W of the same size as at N=1N = 1 (whereas γN<0\gamma_N < 0 makes the first loop's update blow up with NN). In the forward pass, each branch is damped by 1/N1/N, but the same ΔW\Delta W acts coherently across all NN loops, giving N×(1/N)=Θ(1)N \times (1/N) = \Theta(1) total update to the residual stream (whereas γN>0\gamma_N > 0 makes the update vanish).
  • For SGD: The same balance happens in the backward pass as well—each loop's gradient carries the 1/N1/N branch factor, and summing NN aligned loop gradients gives N×(1/N)=Θ(1)N \times (1/N) = \Theta(1) (so the NN-dependent gradient rescaling factor in Eq. (2) is simply NαN−1=1N^{\alpha_N - 1} = 1). Even the raw gradient scale is therefore independent of NN.
AuthorsOpenReview ↗

Rebuttal 2/n

W3 & Q4: Joint transfer across width and loop count

General Response G3(b) reports this experiment in detail. Scaling the width from 768768 to 10241024 at L=12L = 12 following Table 1, we find that the learning rate tuned at width 768768 and N=1N = 1 (1.25×10−31.25 \times 10^{-3}) remains optimal on the grid at width 10241024 for N=1,4,8N = 1, 4, 8. At N=2N = 2, reusing it incurs a loss difference of only 0.00130.0013, which is smaller than the width-768 seed standard deviation (0.00190.0019). This confirms that the width-independent 1/N1/N loop multiplier combines cleanly with standard width-scaling rules.

Q1: Positive correlation across loops (Q1.1) and wording for untied layers (Q1.2)

(1) Why update-induced increment changes are positively correlated. Thank you for prompting us to clarify the physical intuition here; this positive correlation arises from combining weight sharing with the residual skip connection in the bounded-branch regime (αN≥1\alpha_N \ge 1), which we now highlight explicitly in the Introduction (p. 2). Without a skip connection, repeatedly applying WW could rotate or flip the state across loops, so a shared update ΔW\Delta W could push different loops in opposite directions. With the skip connection xr=xr−1+cf(xr−1)x^r = x^{r-1} + c f(x^{r-1}) and αN≥1\alpha_N \ge 1, every loop input xr−1x^{r-1} retains a positive component of the original input carried by the skip path (⟨xq−1,xr−1⟩>0\langle x^{q-1}, x^{r-1}\rangle > 0), and similarly for the backpropagated signals (⟨bq,br⟩>0\langle b_q, b_r\rangle > 0). Applying the same ΔW\Delta W to positively aligned inputs produces positively aligned changes: under a shared update ΔW∝∑qbq(xq−1)⊤\Delta W \propto \sum_q b_q (x^{q-1})^\top in the linear model, the direct change in loop rr's increment is c ΔWxr−1c\thinspace \Delta W x^{r-1}, so for any two loops r,sr, s, ⟨ΔWxr−1,ΔWxs−1⟩∝∑q,q′⟨bq,bq′⟩ ⟨xq−1,xr−1⟩ ⟨xq′−1,xs−1⟩>0,\langle \Delta W x^{r-1}, \Delta W x^{s-1}\rangle \propto \sum_{q, q'} \langle b_q, b_{q'}\rangle\thinspace \langle x^{q-1}, x^{r-1}\rangle\thinspace \langle x^{q'-1}, x^{s-1}\rangle > 0, and perturbations from earlier loops also propagate through the same shared weights in later loops (Proposition 8, Lemma 11; Remark 2 states the corresponding alignment condition for Adam).

Direct measurement in Transformers. Following the protocol of Section 4.1 (L=12L = 12, N=64N = 64), we measured the mean pairwise cosine similarity across 6464 passes both for the initialized increments xr−xr−1x^r - x^{r-1} and for the one-step changes Δ(xr−xr−1)\Delta(x^r - x^{r-1}) after the first AdamW step, under both the unscaled setup of Figure 3 (left) (αN=0\alpha_N = 0) and Loop-μ\muP (αN=1\alpha_N = 1, branch multiplier 1/N1/N).

Branch scalingModelIncrements at initializationOne-step changes Δ(xr−xr−1)\Delta(x^r - x^{r-1})
Unscaled (αN=0\alpha_N = 0, Fig. 3 left)Looped (N=64N = 64)0.84890.84890.71730.7173
Unscaled (αN=0\alpha_N = 0, Fig. 3 left)Untied (6464 distinct stacks)−0.0001-0.00010.21820.2182
Loop-μ\muP (αN=1\alpha_N = 1, 1/N1/N)Looped (N=64N = 64)0.89370.89370.83760.8376
Loop-μ\muP (αN=1\alpha_N = 1, 1/N1/N)Untied (6464 distinct stacks)0.00000.00000.18240.1824

Under both scalings, the one-step increment changes remain strongly aligned in the looped model (0.840.84 under Loop-μ\muP and 0.720.72 unscaled), compared with 0.180.18–0.220.22 in the untied control (which is mildly positive because backpropagated gradients in untied residual networks still share the readout skip signal; Appendix B.2). Under the 1/N1/N multiplier, the skip connection stays dominant throughout the loop, making the cross-loop alignment even stronger (0.840.84 vs. 0.720.72) while keeping the total residual-stream update Θ(1)\Theta(1) as NN grows (0.192,0.190,0.1890.192, 0.190, 0.189 at N=16,32,64N = 16, 32, 64 in Figure 3, right).

(2) Wording for untied layers. We agree that "independent" is imprecise because the input to layer ℓ\ell depends on earlier layers. We replaced it in the Introduction and Figure 1 caption with "uncorrelated" (in the infinite-width limit), matching "nearly uncorrelated" in the Abstract and "nearly orthogonal" in Figure 3 (left).

AuthorsOpenReview ↗

Rebuttal 3/n

Q3: Verified vs. unverified assumptions in Appendix I and scope of uniqueness

(1) Analytical vs. empirical verification. In Appendix I, we separate what is proved analytically, what must be assumed analytically, and what we verify empirically in the full Transformer:

  • Proved analytically: Lemma 22 proves the activation smoothness and positive-covariance conditions in Assumption 1(i)–(ii) for standalone SiLU and GELU, and Lemma 24 establishes deterministic finite-network SGD upper bounds using only bounded activation derivatives.
  • Assumed analytically (and why): General infinite-width closure across NN, non-degeneracy lower bounds (Assumption 1(iii)–(vi)), and alignment along Adam's update direction (Remark 2) are stated as assumptions rather than theorems. In linear SGD, we rule out cross-loop cancellation using exact operator algebra (Lemmas 7 and 11). Nonlinear activations and Adam's entrywise normalization break this linearity, making general analytical lower bounds intractable (for the same reason, Depth-μ\muP [1] also proves its theorems only for linear networks).
  • Verified empirically in the full Transformer (including attention, SwiGLU, RMSNorm, and AdamW): Rather than testing simplified nonlinear MLPs in isolation, we directly test the empirical predictions of these conditions—together with attention and RMSNorm—in the full Transformer used in our experiments. Specifically, Sections 4.1, 4.3, and Appendix C.4 verify cross-loop increment alignment (Figures 3 left and 5), strong alignment of AdamW-induced increment changes (0.840.84 cosine similarity; Q1 and Table 3), and non-vanishing Θ(1)\Theta(1) residual updates at αN=1\alpha_N = 1 (Figures 3 right and 9).

(2) Scope of uniqueness. Uniqueness is proved rigorously for linear networks under SGD and holds conditionally for nonlinear MLPs and Adam under Appendix I. Although attention and RMSNorm are outside our formal proofs, Pre-LN normalizes each branch input rather than the residual stream itself, so shared branches still add Θ(1)\Theta(1) positively correlated increments at every loop. Consequently, our Transformer measurements in Figures 2–4 rule out αN<1\alpha_N < 1 (forward norm grows with NN) and αN>1\alpha_N > 1 (one-step updates vanish at fixed learning rate), matching the predicted unique point αN=1,γN=0\alpha_N = 1, \gamma_N = 0.

Reviewer_5gnwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how to scale looped Transformers, which repeatedly apply the same stack of layers, as the number of loops increases. The authors show that loops behave differently from additional layers because shared weights cause residual contributions to be positively correlated and accumulate linearly with the number of loops. Through an analysis of residual networks in the infinite width limit, they show that scaling each residual branch by the inverse of the loop count, while keeping the learning rate unchanged, is the only parameterization that ensures stable forward propagation and meaningful training updates. They formally establish this result for linear networks trained with SGD and extend the analysis heuristically to nonlinear networks and Adam. Building on Depth-µP, they propose Loop-µP, which combines this loop scaling with the existing depth scaling rule. Experiments on FineWeb-Edu with 12, 24, and 48 layers and up to eight loops show that Loop-µP preserves the optimal base learning rate across different loop counts and achieves lower validation loss than alternative residual scaling rules.

Critical strengths

The paper identifies why loops behave differently from extra layers. Shared weights make residual increments add in the same direction, so they grow linearly in N rather than as a square root.

Loop‑µP only requires multiplying each residual branch by an extra 1/N, and everything else from an existing depth recipe stays the same. This lets practitioners tune hyperparameters once on a model with N = 1 and reuse them for larger loop counts.

Critical weaknesses

The presentation of the paper lacks clarity and requires substantial improvement. The writing relies too heavily on mathematical symbols and notation, making it difficult for readers to follow the key ideas. I strongly recommend fundamentally revising the presentation by incorporating more intuitive visualizations, simplifying the notation, and providing clear, step-by-step explanations.

The strict proofs assume linear networks, SGD, and MLP blocks without attention or normalization, while the experiments use nonlinear activations, AdamW, attention, and RMSNorm. The extension to nonlinear activations and Adam in Appendix I is labeled as heuristic and relies on assumptions close to the conclusions it wants to justify. As a result, the main practical claim rests mostly on experiments rather than on proof.

The paper discusses several methods for stabilizing looped Transformers, such as modified normalization, constrained state transitions, DeepLoop, and FPRM, but compares only against other loop factors. Without these baselines, it is unclear how Loop‑µP performs relative to the alternatives practitioners would actually consider.

Questions for authors

I don’t quite understand why we need to assume L is infinitely larger than N. What impact this brings to the proof, and what it may have an effect in the actual experiments?

The learning rate sweeps use a single seed and a coarse grid, while loss differences between methods are around 0.01 to 0.04. Can the authors report results over multiple seeds, or a finer grid near the optimum, to show that the fixed optimal learning rate and the loss improvements are robust?

Do the loss improvements under Loop‑µP translate into better downstream or reasoning benchmark performance?

6 comments
AuthorsOpenReview ↗

Rebuttal 1/n

We thank Reviewer 5gnw for the thorough review. We are glad the reviewer found our explanation of why loops differ from extra depth useful, and noted that Loop-μ\muP only adds a 1/N1/N multiplier so that hyperparameters tuned at N=1N = 1 can be reused at larger NN. We reply to each point below.

W1: Heavy reliance on notation and presentation clarity

We revised both the main text and the appendices to define every index where it first appears and cut unnecessary symbols (detailed in General Response G1, and highlighted in blue in the updated PDF). Specifically: (1) the Introduction (p. 2) defines rr, ℓ\ell, and xrx^r at first use, Section 3 drops four intermediate scale shorthands (βN,sN,ρN,χ0\beta_N, s_N, \rho_N, \chi_0) and renames the loss to Lt\mathcal{L}_t to avoid colliding with ℓ\ell, and Table 2 (p. 16) collects all main-text notation in one place; (2) Section 3.2 (p. 5) now opens with a plain-English walkthrough of the three scaling constraints before introducing the equations, tying each step to Figures 1(c) and 7(a); and (3) Appendices G–I expand previously compressed proof steps and list right after Assumption 1 which conditions each claim relies on.

W2: Strict proofs cover only linear networks and SGD

The reviewer is right that our strict proofs cover only linear residual MLPs trained with SGD. This follows standard practice in the depth-scaling literature: Depth-μ\muP [1] proves its theorems in the same linear SGD setting, and CompleteP [2] similarly relies on scaling desiderata and experiments for Transformers.

For nonlinear activations and Adam, Appendix I states the additional conditions that the argument requires. Two of them are lower bounds on how much the network responds to an update (Assumption 1(iv) and (vi)), which we prove only for linear networks (Lemma 11). Establishing them for nonlinear networks from properties of the activation alone remains an open problem beyond the scope of this work.

To state this scope more clearly in the manuscript, we renamed Appendix I from "Heuristics for the Proofs in the General Case" to "Conditional Justifications for Nonlinear Networks and Adam". In Remark 2, we now state explicitly that the SGD conditions do not carry over to Adam automatically, and must instead hold along Adam's actual update direction.

Importantly, our empirical support does not rest on final loss alone: in the full Transformer (SwiGLU, AdamW, attention, RMSNorm), Sections 4.1 and 4.3 directly measure the intermediate quantities that the theory predicts. The residual increments of different loops are positively correlated and accumulate linearly (Figures 2, 3 left, and 5), and the one-step change of the residual stream follows the predicted ηN1−αN\eta N^{1-\alpha_N} scaling (Figures 3 right and 9). Our reply to Reviewer wqGf (Q1) adds a direct measurement of this correlation after an AdamW update (0.840.84 mean cosine similarity across 6464 looped passes vs. 0.180.18 in the untied control; Appendix C.4, Table 3).

AuthorsOpenReview ↗

Rebuttal 2/n

W3: Comparison with other stabilization methods

To provide a direct comparison, we ran new experiments against the applicable stabilization methods.

What each method targets. Loop-μ\muP targets both training stability and zero-shot learning-rate transfer from N=1N = 1 to larger NN, while preserving the standard Pre-LN block. The methods the reviewer mentions stabilize looped training by modifying the normalization placement or initialization, and they do not study learning-rate transfer across NN. Specifically, DeepLoop [3] (arXiv v1) uses a Post-LN block x+=Norm(αx+F(x))x^+ = \mathrm{Norm}(\alpha x + F(x)) with α=2LN\alpha = \sqrt{2LN}, and scales the initialization of selected branch weights by β=(8LN)−1/2\beta = (8LN)^{-1/2}, both set from the unrolled depth LNLN. The residual norm of Geiping et al. [4] adds a normalization after the branch, x+=Normout(x+F(Normin(x)))x^+ = \mathrm{Norm}_{\mathrm{out}}(x + F(\mathrm{Norm}_{\mathrm{in}}(x))). We also updated Section 2 to clarify that these works focus on stability rather than learning-rate transfer across NN.

We compare against both DeepLoop and the Geiping-style norm below. We do not compare against Parcae [5] or FPRM [6] because their stabilization relies on input injection, which our architecture does not use. Parcae loops only a middle block between a prelude and a coda, constraining the spectral norm of the matrices that inject the prelude output into each loop. FPRM loops the full stack but re-injects the input at every iteration, deriving its boundedness guarantee from the contraction induced by this input mixing. Without input injection, neither method applies directly.

Results. Following the setup of Section 4.2 (L=12L = 12, N=8N = 8, 10B tokens), we tune each baseline directly at N=8N = 8 over five learning rates, η0∈14,12,1,2,4×1.25×10−3\eta_0 \in \\{\frac{1}{4}, \frac{1}{2}, 1, 2, 4\\} \times 1.25 \times 10^{-3}. Loop-μ\muP is not retuned at N=8N = 8 and directly reuses η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3} from N=1N = 1.

Method \ η0\eta_0 (×10−3\times 10^{-3})0.31250.31250.6250.6251.251.252.52.55.05.0Best loss
DeepLoop (Post-LN), tuned at N=8N = 82.92982.92982.91492.91492.91772.91772.92602.92602.94052.94052.91492.9149
Geiping-style norm, tuned at N=8N = 82.93442.93442.92972.92972.94052.94052.94972.94972.96132.96132.92972.9297
Loop-μ\muP, η0\eta_0 from N=1N = 1——2.90782.9078——2.9078\mathbf{2.9078}

For reference, the unscaled Pre-LN baseline reaches 2.94822.9482 at its best learning rate (0.75×10−30.75 \times 10^{-3}, Figure 4).

We tune the learning rates for DeepLoop and the Geiping-style norm at N=8N = 8, finding that their optimal learning rate is lower, at 0.625×10−30.625 \times 10^{-3}. By contrast, Loop-μ\muP keeps the standard Pre-LN block unchanged and reuses η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3} directly from N=1N = 1 with zero tuning at N=8N = 8, reaching a loss 0.0220.022 lower than the tuned Geiping-style norm (2.82.8 seed stds) and matching or slightly improving on tuned DeepLoop (0.0070.007 lower, within 11 std).

AuthorsOpenReview ↗

Rebuttal 3/n

Q1: Ordered limit (L→∞L \to \infty before N→∞N \to \infty) and its impact on proofs and experiments

We take L→∞L \to \infty before N→∞N \to \infty to make the continuum limit tractable step by step, and Loop-μ\muP itself does not require L≫NL \gg N in practice.

Why the proof uses this order. Each loop runs a stack of LL distinct layers once, and the outer loop repeats that whole stack NN times. Taking L→∞L \to \infty at fixed NN lets us first characterize what a single pass through the LL-layer stack does (Lemma 15), and then taking N→∞N \to \infty characterizes what happens when that pass is repeated NN times.

Why Loop-μ\muP does not need L≫NL \gg N. The order of limits only matters when the LL-scaling and NN-scaling pull in opposite directions—for instance, if the LL-factor shrinks each pass toward zero while the NN-factor blows it up, whichever limit goes to infinity first wins (Figure 8). At the Loop-μ\muP point (αN,αL;γN,γL)=(1,1/2;0,1/2)(\alpha_N, \alpha_L; \gamma_N, \gamma_L) = (1, 1/2; 0, 1/2), neither side blows up or vanishes: both the LL-factor (L1/2−αL=1L^{1/2-\alpha_L} = 1) and the NN-factor (N1−αN=1N^{1-\alpha_N} = 1) in the forward scale, as well as the corresponding factors (L1−αL−γL=1L^{1-\alpha_L-\gamma_L} = 1 and N1−αN−γN=1N^{1-\alpha_N-\gamma_N} = 1) in the one-step output response, are balanced on their own. In fact, when we hold depth finite and send N→∞N \to \infty alone (the N≫LN \gg L regime, Appendix A.3, Claims 2′2'–3′3'), we recover the exact same loop rule (αN,γN)=(1,0)(\alpha_N, \gamma_N) = (1, 0). Analyzing simultaneous joint limits requires stronger proof techniques, which we note as future work in Appendix D (Limitations).

What the experiments show. In Figure 4 (L∈12,24,48L \in \\{12, 24, 48\\}, N∈1,2,4,8N \in \\{1, 2, 4, 8\\}), learning-rate transfer already holds at L=12,N=8L = 12, N = 8, where NN and LL are comparable. Furthermore, in Figures 2, 3, and 9 (L=12L = 12), the predicted forward-norm invariance and one-step update scaling hold up to N=64N = 64, well into the N>LN > L regime.

Q2: Multi-seed robustness and learning-rate grid resolution

General Response G2 reports our four-seed experiments and grid resolution analysis. Across all N∈1,2,4,8N \in \\{1, 2, 4, 8\\} at η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3}, the four-seed mean stays within 0.00100.0010 of seed 0, and at every N>1N > 1, Loop-μ\muP improves over the best alternative loop factor by 2.32.3–6.56.5 seed standard deviations (see G2). At N=8N = 8, running four seeds across the neighboring grid points η0∈1.0,1.25,1.5×10−3\eta_0 \in \\{1.0, 1.25, 1.5\\} \times 10^{-3} (spaced by 1.21.2–1.25×1.25\times, matching the finest grid in Tensor Programs V; G2) confirms that η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3} continues to achieve the lowest four-seed mean loss (2.9085±0.00782.9085 \pm 0.0078 vs. 2.9153±0.00192.9153 \pm 0.0019 and 2.9143±0.00312.9143 \pm 0.0031).

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response 1/n

General Response

We thank all reviewers for their time and constructive feedback. Because several questions were raised by more than one reviewer, we address the shared topics here in G1–G3 and give concise answers with cross-references in each individual response. We have also added the new experimental results and tables from this rebuttal to Appendix C of the revised manuscript, with all revisions highlighted in blue in the updated PDF.

G1. Presentation and notation (Reviewers 5gnw and wqGf)

Both reviewers found the notation dense and introduced too abruptly. In response, we made four changes to the manuscript (highlighted in blue in the revised PDF).

(1) Streamlined notation and definitions at first use. The Introduction now defines the loop index r=1,…,Nr = 1, \dots, N, the index ℓ=1,…,L\ell = 1, \dots, L of the distinct layers in the stack, and the residual stream xrx^r after loop rr where they first appear. To reduce symbol density in Section 3, we removed four intermediate shorthand symbols from the main text (βN\beta_N, sNs_N, ρN\rho_N, and χ0\chi_0) and wrote the powers N1−αNN^{1-\alpha_N}, N−γNN^{-\gamma_N}, N1−αN−γNN^{1-\alpha_N-\gamma_N} and loss derivative L0′(f0(ξ0))\mathcal{L}_0'(f_0(\xi_0)) directly; changed the loss notation from ℓt\ell_t to Lt\mathcal{L}_t so that it no longer collides with the layer index ℓ\ell; and added a brief explanation at Equation (2) for why the nNαN−1n N^{\alpha_N - 1} gradient rescaling normalizes the sum of NN aligned loop gradients to Θ(1)\Theta(1) while N−γNN^{-\gamma_N} is directly the loop factor of the Adam learning rate in practice (and of raw SGD at αN=1\alpha_N = 1). We also unified the superscript notation across the main text and appendices and added a notation table at the start of the appendix (Table 2), which Section 3.1 references.

(2) A plain overview before the derivation. Section 3.2 now opens by stating the three constraints in words. We write the branch multiplier as cN=aN−αNc_N = a N^{-\alpha_N}, with loop factor N−αNN^{-\alpha_N} and learning-rate factor N−γNN^{-\gamma_N}.

  • At initialization, the NN loop increments are aligned, so the residual stream grows like NcNN c_N. Keeping it bounded requires αN≥1\alpha_N \ge 1.
  • After one training step, the change in the first loop scales like N−γNN^{-\gamma_N}. Keeping it from growing with NN requires γN≥0\gamma_N \ge 0.
  • The same update is reused in all NN loops, so the output changes by about NcNN−γNN c_N N^{-\gamma_N}. Keeping this change at Θ(1)\Theta(1) requires αN+γN=1\alpha_N + \gamma_N = 1.

Only (αN,γN)=(1,0)(\alpha_N, \gamma_N) = (1, 0) satisfies all three constraints. The overview references Figure 1(c) at the start and the full phase diagram (Figure 7(a)) at the end.

(3) More proof detail. In Appendices G–H we expanded the previously compressed proof steps, covering the infinite-width limit, why the states and internal preactivations remain nonvanishing, and the single-step response. We also split the counterexample in Appendix G.4 into three parts (setup, two gradient steps, and output response) so that each step can be checked independently.

(4) Assumptions stated upfront. Right after Assumption 1, we now list which assumptions each claim uses. We also moved the Adam discussion (Remark 2) from the end of the appendix to follow Assumption 1 directly, and wrote out the Adam update together with the alignment conditions it requires.

AuthorsOpenReview ↗

General Response 2/n

G2. Multiple seeds and learning-rate grid (Reviewers 5gnw and V29m)

(a) Multi-seed comparison across loop counts. We ran three additional seeds (four in total, including the paper's seed 0) for Loop-μ\muP at L=12L = 12 with η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3}. The table below reports both the four-seed mean ±\pm std and the seed 0 loss (used for seed-matched comparison against the other loop factors in Figure 4, which were also run with seed 0).

Loop count NNLoop-μ\muP (4 seeds, mean ±\pm std)Loop-μ\muP (seed 0)Best other loop factor (seed 0)Loss gap (seed 0)
113.0053±0.00213.0053 \pm 0.00213.00613.0061—— (identical at N=1N = 1)
222.9702±0.00192.9702 \pm 0.00192.96942.96942.97382.9738 (N−1/2N^{-1/2})−0.0044-0.0044 (2.32.3 std)
442.9330±0.00352.9330 \pm 0.00352.93402.93402.95662.9566 (N−3/2N^{-3/2})−0.0226-0.0226 (6.56.5 std)
882.9085±0.00782.9085 \pm 0.00782.90782.90782.93452.9345 (N−3/2N^{-3/2})−0.0267-0.0267 (3.43.4 std)

Across all NN, the four-seed mean stays within 0.00100.0010 of seed 0 (≤0.4\le 0.4 std). At every N>1N > 1, Loop-μ\muP improves over the best alternative loop factor by 2.32.3–6.56.5 seed standard deviations on seed 0 (1.91.9–6.76.7 std using the four-seed mean).

(b) Multi-seed check across neighboring learning rates and grid resolution. Near η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3}, our grid points {1.0,1.25,1.5}×10−3\lbrace 1.0, 1.25, 1.5\rbrace \times 10^{-3} are spaced by 1.21.2–1.25×1.25\times, matching the finest grid in Tensor Programs V [1] (21/4≈1.19×2^{1/4} \approx 1.19\times) and finer than the 2×\sqrt{2}\times grid it uses for BERT. To verify that η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3} remains optimal across seeds relative to its neighboring grid points, we also ran all four seeds at η0∈{1.0,1.5}×10−3\eta_0 \in \lbrace 1.0, 1.5\rbrace \times 10^{-3} for L=12,N=8L = 12, N = 8.

η0\eta_0 (×10−3\times 10^{-3})Loss at N=8N = 8 (4 seeds, mean ±\pm std)Loss at N=8N = 8 (seed 0)
1.01.02.9153±0.00192.9153 \pm 0.00192.91552.9155
1.251.252.9085±0.0078\mathbf{2.9085} \pm 0.00782.9078\mathbf{2.9078}
1.51.52.9143±0.00312.9143 \pm 0.00312.91542.9154

At N=8N = 8, η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3} continues to achieve the lowest four-seed mean loss, 0.00680.0068 and 0.00580.0058 below the two adjacent grid points. Moreover, even where the curve is locally flat near the minimum, every Loop-μ\muP run in the 2×2\times interval η0∈[0.75,1.5]×10−3\eta_0 \in [0.75, 1.5] \times 10^{-3} at N=8N = 8 reaches a lower validation loss (2.90782.9078–2.93222.9322; see G3(b)) than the best retuned loss of any alternative loop factor (2.93452.9345–2.94822.9482).

AuthorsOpenReview ↗

General Response 3/n

G3. Transfer at larger depth and width (Reviewers wqGf and V29m)

Directly verifying hyperparameter transfer requires full learning-rate sweeps. Figure 4 already uses about 170 runs of 10B tokens each, scaling up to N=8N = 8 and 384384 executed layers (L=48L = 48, N=8N = 8). The main paper therefore holds the width fixed at 768768 to isolate the loop axis. To verify that Loop-μ\muP continues to transfer as depth and width grow, we ran two larger-scale sweeps within our compute budget.

(a) A deeper stack. We trained a model with L=96L = 96 (about 780M parameters) and up to N=8N = 8 loops (768768 executed layers), sweeping a 3-point grid η0∈{1.0,1.25,1.5}×10−3\eta_0 \in \lbrace 1.0, 1.25, 1.5\rbrace{} \times 10^{-3} around the N=1N = 1 optimum. Across all loop counts, η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3} remains optimal.

LLNNη0=1.0\eta_0 = 1.01.251.251.51.5Best η0\eta_0Cost of reusing 1.251.25
9696112.80492.80492.8031\mathbf{2.8031}2.80342.80341.251.25— (source)
9696222.76812.76812.7648\mathbf{2.7648}2.76582.76581.251.2500
9696442.74022.74022.7322\mathbf{2.7322}2.73382.73381.251.2500
9696882.72852.72852.7178\mathbf{2.7178}2.71982.71981.251.2500

(b) A wider model. Our theory takes n→∞n \to \infty first, and the loop multiplier 1/N1/N does not depend on width, so it combines directly with the width rules of μ\muP [2] and CompleteP [3] (Table 1). To test joint transfer, we scaled the width from 768768 to 10241024 (mn=4/3m_n = 4/3) at L=12L = 12 while keeping the base learning-rate grid unchanged. Following Table 1, scaling the width by mnm_n multiplies the hidden initialization variance, hidden learning rate, unembedding forward multiplier, and ϵopt\epsilon_{\mathrm{opt}} by mn−1m_n^{-1}, and multiplies weight decay by mnm_n, at every N∈{1,2,4,8}N \in \lbrace 1, 2, 4, 8\rbrace{}.

WidthNNη0=0.75\eta_0 = 0.751.01.01.251.251.51.52.02.0Best η0\eta_0Cost of reusing 1.251.25
768768113.01283.01283.00753.00753.0061\mathbf{3.0061}3.00943.00943.01893.01891.251.25— (source)
768768222.99112.99112.97922.97922.9694\mathbf{2.9694}2.97202.97202.97102.97101.251.2500
768768442.95642.95642.94182.94182.9340\mathbf{2.9340}2.95502.95502.95382.95381.251.2500
768768882.93222.93222.91552.91552.9078\mathbf{2.9078}2.91542.91542.94382.94381.251.2500
10241024112.90942.90942.90892.90892.9041\mathbf{2.9041}2.91122.91122.91602.91601.251.2500
10241024222.89542.89542.88162.88162.87632.87632.8750\mathbf{2.8750}2.87702.87701.501.500.00130.0013
10241024442.86202.86202.85402.85402.8439\mathbf{2.8439}2.86252.86252.85972.85971.251.2500
10241024882.84472.84472.82732.82732.8248\mathbf{2.8248}2.82802.82802.85542.85541.251.2500

The table reports final validation loss, with η0\eta_0 in units of 10−310^{-3}. The last column is the loss at η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3} (tuned at width 768768 and N=1N = 1) minus the best loss on the grid.

At width 10241024, η0=1.25×10−3\eta_0 = 1.25 \times 10^{-3} remains optimal on the grid for N=1,4,8N = 1, 4, 8. At N=2N = 2, the grid optimum is 1.5×10−31.5 \times 10^{-3}, but reusing 1.25×10−31.25 \times 10^{-3} incurs a loss difference of only 0.00130.0013, which is smaller than the width-768 seed standard deviation at N=2N = 2 (0.00190.0019, G2).

References

[1] Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., & Gao, J. (2022). Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466.

[2] Yang, G., & Hu, E. J. (2021). Tensor programs IV: Feature learning in infinite-width neural networks. In Proceedings of the 38th International Conference on Machine Learning (Vol. 139, pp. 11727–11737). PMLR.

[3] Dey, N., Zhang, B. C., Noci, L., Li, M., Bordelon, B., Bergsma, S., Pehlevan, C., Hanin, B., & Hestness, J. (2025). Don't be lazy: CompleteP enables compute-efficient deep transformers. In Advances in Neural Information Processing Systems (Vol. 38).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.