- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies how to scale the residual branch and its learning rate with respect to the loop count in the looped transformer architecture. The main idea is since loops shares weights, residual increments have positive correlation and thus grow linearly in N, unlike in untied architecture. Based on this, proposes Loop-P which multiplies each residual branch by 1/N on top of Depth-P and keeps the learning rate indepentdent of N. Emperical evaluations on FineWeb-Edu shows that the learning rate tuned on the smallest model transfers up to N=8, 384 executed layers.
Critical strengths
-
The motivation is clear and intuitive. Figure 3 shows that residual increments of a looped architecture have positive correlation while untied architecture are near orthogonal, which clearly explains why loops cannot be treated equal as extra depth.
-
The theory and its supporting experiments are consistent. Section 4.1 checks both predictions of Section 3.2; a bounded forward pass and order-one change after an optimizer step, and only satisfies both conditions. (Figure 2 and 3)
-
The recipe is practical. Table 1 shows only the residual multiplier have to change from the untied recipe, and Figure 4 shows that a model tuned on N=1 can be looped without further LR sweep. It makes the design simple and effective.
Critical weaknesses
-
Only a single seed was used for Figure 4 (mentioned in Appendix E.3), and the learning rate grid is somewhat coarse. It seems important since the curves in Figure 4 are quite flat near the optimum, therefore hard to conclude that the best LR truly stays fixed across N. Also the experiments are relatively small scale (10B tokens, 48 layers) so it is unclear whether it transfers to a larger scale.
-
The residual multiplier depends on N. Running more loops at test-time is one of the main motivations looped architectures have, also mentioned in Section 2 (Geiping et al., 2025). However, this paper does not discuss how Loop-P performs when the test-time loop count differs from its training.
Questions for authors
-
What is the seed variance for Figure 4? Does the best LR still holds with multiple seeds and a finer grid near the optimum?
-
What happens if the test-time loop count differs from training? Should the multiplier be fixed to the training N, or should it be changed?
4 comments
Rebuttal 2/n
W2 & Q2: Varying loop count at test time ( vs. )
(1) Which multiplier to use. Our continuum-limit analysis motivates rescaling the branch multiplier to . Under Loop-P, loops discretize a fixed integration horizon with step size (Claim 3, Lemma 5). Rescaling to keeps this total horizon fixed while taking finer steps, whereas keeping stretches the horizon by and pushes the final residual norm outside the scale seen during training.
(2) Test at . Evaluating our fixed- Loop-P checkpoints with additional loops at test time confirms this prediction.
| Test-time multiplier | +1 loop | +2 loops | +3 loops | ||
|---|---|---|---|---|---|
| Fixed | |||||
| Rescaled | |||||
| Fixed | |||||
| Rescaled |
Keeping fixed degrades validation loss rapidly as loops are added (rising by at with 3 extra loops). Rescaling to keeps the loss stable (within across 1 to 3 extra loops at and within at ) and even yields slight improvements for 1–2 extra loops.
(3) Why the gains stop after one or two extra loops. Because our experiments focus on hyperparameter transfer across training runs, each model is trained at a fixed , and the final pass specializes to the end of the loop (Figure 5). To keep improving over more extra loops at test time, models need variable- training (such as randomized loop counts or intermediate supervision [2, 3, 4]) rather than training at a single fixed . Importantly, because Loop-P's learning rate does not depend on (), it fits variable- training naturally: when a different loop count is sampled on each step or batch, only the forward branch multiplier changes, with no need to rescale the optimizer learning rate from step to step. We leave it for future work.
Thank you again for the helpful feedback.
References
[1] Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., & Gao, J. (2022). Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466.
[2] Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B. R., Kailkhura, B., Bhatele, A., & Goldstein, T. (2025). Scaling up test-time compute with latent reasoning: A recurrent depth approach. In Advances in Neural Information Processing Systems (Vol. 38).
[3] Zhu, R.-J., Wang, Z., Hua, K., Zhang, T., Li, Z., Que, H., Wei, B., Wen, Z., Yin, F., Xing, H., Li, L., Shi, J., Ma, K., Li, S., Kergan, T., Smith, A., Qu, X., Hui, M., Wu, B., . . . Eshraghian, J. (2025). Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741.
[4] Prairie, H., Novack, Z., Berg-Kirkpatrick, T., & Fu, D. Y. (2026). Parcae: Scaling laws for stable looped language models. arXiv preprint arXiv:2604.12946.
Thank you for the detailed response and the additional experiments. The multi-seed results, the larger-scale sweeps, and the test-time analysis address my questions, and I agree that the grid spacing near the optimum is reasonable. I will maintain my rating of Weak Acceptance.
Thank you again for your constructive feedback!
Rebuttal 1/n
We thank Reviewer V29m for finding our motivation clear and intuitive, our derivations well aligned with the experiments, and our recipe simple and practical. We reply to each point below.
W1 & Q1: Multi-seed robustness, learning-rate grid resolution, and model scale
(1) Multiple seeds, grid resolution, and flat curves. General Response G2 reports our four-seed results and grid resolution check: the four-seed mean stays within 0.0010 of seed 0 across N∈{1,2,4,8}, Loop-μP beats the best alternative loop factor by 2.3–6.5 seed stds at every N>1, and at N=8 our 1.2–1.25× grid (matching the finest grid in Tensor Programs V [1]) keeps η0=1.25×10−3 optimal over four seeds relative to both neighbors (1.0 and 1.5×10−3). Consistently, across all 20 Loop-μP sweeps in the paper and rebuttal (Figure 4 and G3), 19 out of 20 attain their discrete grid minimum at η0=1.25×10−3 (and the single exception, width 1024 at N=2, is within 0.0013<1σ).
(2) Model scale. We agree that evaluating larger scales is important, as also noted in Appendix D. Within our compute budget, General Response G3 reports two new scaled-up experiments. In a deeper stack (L=96, about 780M parameters, up to N=8 loops and 768 executed layers), η0=1.25×10−3 remains optimal across all loop counts. In a wider model (width 1024 at L=12), the learning rate tuned at width 768 and N=1 remains optimal on the grid for N=1,4,8, and reusing it at N=2 incurs a loss difference of only 0.0013 (below the 0.0019 width-768 seed std).