- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Delayed Recurrent Depth Expansion (DDE), a training strategy for looped Transformers that reduces training compute by first training a shallow dense Transformer and introducing recurrent depth later in training. To mitigate the performance degradation caused by converting a pretrained dense model into a recurrent one, the authors adopt an existing step-relaxed recurrence mechanism that scales down each recurrent update. Experiments on 155M and 314M models show that DDE reduces cumulative Transformer-block FLOPs by approximately 30–47% at matched validation loss compared to training recurrent models from scratch. The authors also demonstrate that step relaxation reduces the immediate validation-loss increase upon conversion and that the conversion mechanism can be applied to larger pretrained language models.
Critical strengths
-
Timely and interesting research problem. Improving the training efficiency of looped Transformers is a timely and important problem. I also find the idea of converting a standard dense Transformer into a looped model particularly interesting. The paper provides some useful empirical findings on the feasibility and compute efficiency of this approach.
-
Reasonably clear presentation. The paper is generally well-written and easy to follow, with a clear structure and presentation of the experimental results. The writing quality is better than that of many other submissions I have reviewed this cycle.
Critical weaknesses
-
The DDE method can't really be considered a new method. As explicitly acknowledged by the authors, step-relaxed recurrence has already been extensively studied in Chen et al. (2026a). Moreover, LoopCoder (Yang et al., 2026), also cited in the paper, adopts a very similar training setting (dense Transformer -> recurrent conversion -> continued pretraining). I don't see a significant difference between this paper and prior work. This doesn't necessarily diminish the value of its empirical findings, but I wouldn't consider it a novel methodological contribution.
-
Questionable usefulness of "validation-loss shock." I find "validation-loss shock" a rather strange metric and don't think it provides much useful information. The results in Section 5.4 are largely expected by construction: naive recurrence repeatedly applies the full transformation, while step-relaxed recurrence retains more of the original hidden state at each step. Naturally, the latter is designed to stay closer to the original dense model and should exhibit a smaller loss increase immediately after conversion.
-
Limited benefits over naive recurrence. The results do not convincingly show that step-relaxed recurrence is better than naive recurrence for continued training. In Section 5.4.2, the final validation loss differs by only 0.0004 between the two methods, suggesting that step relaxation provides little practical benefit to the final training outcome.
-
Unjustified expansion point. It is unclear why the authors choose 80B tokens (out of 100B) as the expansion point without further justification. The only relevant analysis I could find is Table 3, which measures "validation-loss shock," a metric I already find uninformative. For example, if conversion were performed at initialization (0B tokens), the loss shock could be negligible, but that would not imply it is a good expansion point.
Questions for authors
I gave a clear rejection rating only because I believe many of the paper's claims and conclusions are not sufficiently supported. That said, I still think this paper is substantially better than most of the AI slop I reviewed this cycle, and I honestly wish I could give it a score of 1.5. I would therefore be very willing to raise my score to 2 or higher in the rebuttal.
8 comments
W1. Positioning Relative to TFLT and LoopCoder and Methodological Contribution
We clarify how DDE differs from Chen et al. (2026a) and LoopCoder (Yang et al., 2026), and we will update the closing sentence of Section 2 to state our contribution more accurately.
-
Comparison with TFLT (Chen et al., 2026a). TFLT studies training-free forward inference on frozen weights, asking how to evaluate a pretrained block repeatedly without updating parameters. It does not investigate backward gradient flow, optimizer state survival after topology changes, or pretraining compute savings.
-
Comparison with LoopCoder (Yang et al., 2026). LoopCoder converts an off-the-shelf dense checkpoint for domain-specific continued pretraining in code generation using unscaled recurrence alongside depth-specific gated mechanisms, rather than studying how to minimize cumulative pretraining FLOPs of a fixed-parameter looped architecture at matched validation loss or how conversion perturbations scale with loop depth .
-
Formalizing post-conversion optimization dynamics. We provide a local analysis, which we will incorporate into Section 4.2, showing why step relaxation supports continued training after dense-to-recurrent conversion.
Let denote the hidden state entering the middle stack , and define so that the dense model computes . After conversion to recurrent steps with , naive looping applies , whereas step-relaxed DDE applies for .
In the backward pass, let and . Naive recurrence propagates gradients through the full-Jacobian product
whereas step-relaxed DDE uses , yielding
Each step scales the residual Jacobian contribution by rather than multiplying the full middle-stack Jacobian times. This reduces the per-step departure from the identity and provides a mechanism for mitigating depth-dependent gradient amplification after conversion. This analysis is consistent with the improved stability observed during continued training.
W2. Role and Interpretation of Validation-Loss Shock
We agree with the reviewer that validation-loss shock should not be treated as a standalone performance metric or used to select the expansion point (for instance, expanding at 0B tokens trivially produces minimal shock without saving compute). We will reframe Section 5.4.1 strictly as a diagnostic sanity check of the immediate functional discontinuity before resumed optimization.
At the same time, our local forward analysis (to be added to Section 4.2) shows a specific structural property beyond qualitative interpolation. Relative to the dense output , naive recurrence incurs error
whereas step-relaxed DDE gives
Step relaxation cancels the explicit mismatch. Letting , a first-order Taylor approximation yields
where the leading coefficient remains strictly below as increases. This explains why the initial loss jump stays bounded below even at (Table 4) rather than growing rapidly with , keeping the converted checkpoint within a recoverable optimization basin.
We thank the reviewer for the detailed feedback and for noting the clarity of presentation and the relevance of the problem. Below we address each of the four weaknesses with additional experiments and analysis.