- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies residual scaling in looped Transformers where the same physical blocks are visited repeatedly.
The paper shows that the original DeepNorm scaling rule can be insufficient when Transformer blocks are repeatedly reused across loop iterations.
It proposes DeepLoop, which uses a stronger scaling rule and evaluates it on looped language models and a hierarchical recurrent reasoning model.
Critical strengths
The paper identifies why the original DeepNorm scaling can be insufficient when the same Transformer blocks are reused across loop iterations, and formalizes this effect with a loop aware perturbation analysis.
The proposed scaling rule improves performance on looped language models and also gives a clear gain on HRM under a controlled comparison.
Critical weaknesses
A closely related prior work [1] already argues that weight sharing creates correlated residual updates in looped Transformers and derives a residual scaling rule. DeepLoop’s parameterization also yields an update to residual ratio proportional to , so the incremental novelty over this prior work is unclear.
The main language modeling baseline uses a standard Pre LN block with standard initialization, while DeepLoop uses a Post LN sandwich block together with the proposed residual scaling. Therefore, the reported gains cannot be attributed specifically to the new scaling rule.
[1] On the Residual Scaling of Looped Transformers: Stability and Transferability, Latent & Implicit Thinking Workshop at ICLR 2026.
Questions for authors
Can the authors compare DeepLoop with DeepNorm under the same Post LN sandwich architecture, while keeping all other training settings fixed?
2 comments
Thank you for the experiment. However, I still think the novelty of the work is limited considering [1] and the bar of ICLR. So, I maintain my rating.
Response to Reviewer ZukM: p=1/4 vs p=1/2 in the same block, and relation to [1]
Thank you for the review.
Q (and W2): p=1/2 vs p=1/4 in the same sandwich block; can the gains be attributed to the scaling? You're right that Table 1 compares two full configurations (pre-LN vs sandwich + scaling), so on its own it can't tell you what the exponent contributes. In the submission the only exponent-only evidence was Appendix F, and that sweep started at 0.30. We ran the comparison you asked for (N1/N2 in the common comment). With p=1/4 the looped model trains on 0 of 5 seeds: val loss is 7.68 at step 2,000, which is the unigram floor. With p=1/2 it trains on 5 of 5 (3.72). To check whether this is about weight sharing or just about depth, we also ran both exponents with 36 untied blocks at the same depth and init. Untied, p=1/4 trains on 5 of 5 and is a bit better than p=1/2 (3.56 vs 3.66).
So in this block the DeepNorm exponent works without weight sharing and fails with it, which is what our analysis says should happen. What we still can't do is split the 0.016-0.028 nat gain over pre-LN in Table 1 into a part from the block and a part from the scaling. There is no trained sandwich model with p=1/4 at R≥3 to compare against, and we haven't run the sandwich block with no scaling at all. We will add N1/N2 to the paper, describe Table 1 as a full-configuration comparison, and say this explicitly.
W1: relation to [1]. Thanks for the pointer. We didn't know about [1] and should have cited it. We will add it and discuss it. Our work was done independently: an earlier version of this paper was already under review at another conference in May 2026, before [1] appeared on arXiv (June 2026).
You're right that both papers end up with a branch-to-skip ratio that scales as 1/R at fixed physical depth. We think the two results support each other more than they overlap. [1] works with pre-LN and multiplies each branch by N−1L−1/2 at run time (N loops, L layers). Its argument is an infinite-width one, proved for linear networks under SGD, and the goal is learning-rate transfer across loop counts. We work with the post-norm DeepNorm family, where the branch enters through 1/α after normalization (Lemma A.1) and the two knobs are the skip scale α and the init gain β. Our bound is finite-width and first-order, and it has the alignment coefficient κR in it, so it gives a range of thresholds p≥(1+γ)/4 and falls back to DeepNorm at γ=0. We also treat the truncated-gradient hierarchical case. The rules are not identical either. In our notation [1] uses a ratio ∝1/(RK), while DeepLoop uses 1/(4KR). If we take our aligned condition MR(β/α)2=O(1) with equality, we get exactly 1/(RK). DeepLoop is more conservative in K because it keeps DeepNorm's constants.
[1] also measures that residual increments from different passes are positively correlated in looped models and close to orthogonal in untied ones. That fits what we now measure directly on gradients and read-outs (N3): κ^=R at init for the tied model, and orthogonal gradients for the untied one. We will put this comparison into Related Work.