acceptodds
Under review as a conference paper at ICLR 2027

DELAYED DEPTH EXPANSION FOR LOOPED TRANSFORMERS

Abstract

Looped Transformers recently have gained attention for increasing computational depth by repeatedly applying weight-shared Transformers blocks. While improving parameter efficiency, it requires heavy recurrent computation for every token during training. We ask whether this recurrent depth must be paid for throughout the entire training pipeline. To this end, we introduce Delayed Recurrent Depth Expansion (DDE), which first trains a shallower dense Transformer and introduces parameter-shared recurrent depth only later in training. Directly converting a trained dense model to repeated computation causes substantial functional disruption, so we use step-relaxed recurrence to provide a controlled dense-to-recurrent transition. Across 155M and 314M parameter models, DDE substantially reduces cumulative Transformer-block FLOPs at matched validation loss compared to training the corresponding recurrent architecture from scratch. Step relaxation notably reduces the instantaneous loss increase at conversion, with the advantage growing at larger recurrence depths, while remaining competitive with conventional looping when trained from scratch. We further show that the same conversion mechanism can be applied to pretrained language models across multiple model families and billion-parameter scales. These results demonstrate that the recurrent computation used at convergence need not be paid for throughout training, providing a simple route to more compute-efficient training of looped Transformers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM training, Looped Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_mWj7OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Delayed Recurrent Depth Expansion (DDE), a training strategy for looped Transformers that reduces training compute by first training a shallow dense Transformer and introducing recurrent depth later in training. To mitigate the performance degradation caused by converting a pretrained dense model into a recurrent one, the authors adopt an existing step-relaxed recurrence mechanism that scales down each recurrent update. Experiments on 155M and 314M models show that DDE reduces cumulative Transformer-block FLOPs by approximately 30–47% at matched validation loss compared to training recurrent models from scratch. The authors also demonstrate that step relaxation reduces the immediate validation-loss increase upon conversion and that the conversion mechanism can be applied to larger pretrained language models.

Critical strengths
  1. Timely and interesting research problem. Improving the training efficiency of looped Transformers is a timely and important problem. I also find the idea of converting a standard dense Transformer into a looped model particularly interesting. The paper provides some useful empirical findings on the feasibility and compute efficiency of this approach.

  2. Reasonably clear presentation. The paper is generally well-written and easy to follow, with a clear structure and presentation of the experimental results. The writing quality is better than that of many other submissions I have reviewed this cycle.

Critical weaknesses
  1. The DDE method can't really be considered a new method. As explicitly acknowledged by the authors, step-relaxed recurrence has already been extensively studied in Chen et al. (2026a). Moreover, LoopCoder (Yang et al., 2026), also cited in the paper, adopts a very similar training setting (dense Transformer -> recurrent conversion -> continued pretraining). I don't see a significant difference between this paper and prior work. This doesn't necessarily diminish the value of its empirical findings, but I wouldn't consider it a novel methodological contribution.

  2. Questionable usefulness of "validation-loss shock." I find "validation-loss shock" a rather strange metric and don't think it provides much useful information. The results in Section 5.4 are largely expected by construction: naive recurrence repeatedly applies the full transformation, while step-relaxed recurrence retains more of the original hidden state at each step. Naturally, the latter is designed to stay closer to the original dense model and should exhibit a smaller loss increase immediately after conversion.

  3. Limited benefits over naive recurrence. The results do not convincingly show that step-relaxed recurrence is better than naive recurrence for continued training. In Section 5.4.2, the final validation loss differs by only 0.0004 between the two methods, suggesting that step relaxation provides little practical benefit to the final training outcome.

  4. Unjustified expansion point. It is unclear why the authors choose 80B tokens (out of 100B) as the expansion point without further justification. The only relevant analysis I could find is Table 3, which measures "validation-loss shock," a metric I already find uninformative. For example, if conversion were performed at initialization (0B tokens), the loss shock could be negligible, but that would not imply it is a good expansion point.

Questions for authors

I gave a clear rejection rating only because I believe many of the paper's claims and conclusions are not sufficiently supported. That said, I still think this paper is substantially better than most of the AI slop I reviewed this cycle, and I honestly wish I could give it a score of 1.5. I would therefore be very willing to raise my score to 2 or higher in the rebuttal.

8 comments

We thank the reviewer for the detailed feedback and for noting the clarity of presentation and the relevance of the problem. Below we address each of the four weaknesses with additional experiments and analysis.

W1. Positioning Relative to TFLT and LoopCoder and Methodological Contribution

We clarify how DDE differs from Chen et al. (2026a) and LoopCoder (Yang et al., 2026), and we will update the closing sentence of Section 2 to state our contribution more accurately.

  • Comparison with TFLT (Chen et al., 2026a). TFLT studies training-free forward inference on frozen weights, asking how to evaluate a pretrained block repeatedly without updating parameters. It does not investigate backward gradient flow, optimizer state survival after topology changes, or pretraining compute savings.

  • Comparison with LoopCoder (Yang et al., 2026). LoopCoder converts an off-the-shelf dense checkpoint for domain-specific continued pretraining in code generation using unscaled recurrence alongside depth-specific gated mechanisms, rather than studying how to minimize cumulative pretraining FLOPs of a fixed-parameter looped architecture at matched validation loss or how conversion perturbations scale with loop depth KK.

  • Formalizing post-conversion optimization dynamics. We provide a local analysis, which we will incorporate into Section 4.2, showing why step relaxation supports continued training after dense-to-recurrent conversion.

Let h0=P(Emb(x))h_0 = P(\mathrm{Emb}(x)) denote the hidden state entering the middle stack MM, and define GM(h)=M(h)−hG_M(h) = M(h) - h so that the dense model computes M(h0)=h0+GM(h0)M(h_0) = h_0 + G_M(h_0). After conversion to KK recurrent steps with u0=h0u_0 = h_0, naive looping applies uk+1=uk+GM(uk)u_{k+1} = u_k + G_M(u_k), whereas step-relaxed DDE applies hk+1=hk+1KGM(hk)h_{k+1} = h_k + \frac{1}{K}G_M(h_k) for k=0,…,K−1k = 0, \ldots, K-1.

In the backward pass, let JG(h)=∂GM(h)∂hJ_G(h) = \frac{\partial G_M(h)}{\partial h} and JM(h)=I+JG(h)J_M(h) = I + J_G(h). Naive recurrence propagates gradients through the full-Jacobian product

∂uK∂u0=∏k=K−10JM(uk)=∏k=K−10(I+JG(uk))\frac{\partial u_K}{\partial u_0} = \prod_{k=K-1}^{0} J_M(u_k) = \prod_{k=K-1}^{0} \left(I + J_G(u_k)\right)

whereas step-relaxed DDE uses ∂hk+1∂hk=K−1KI+1KJM(hk)=I+1KJG(hk)\frac{\partial h_{k+1}}{\partial h_k} = \frac{K-1}{K}I + \frac{1}{K}J_M(h_k) = I + \frac{1}{K}J_G(h_k), yielding

∂hK∂h0=∏k=K−10(I+1KJG(hk)).\frac{\partial h_K}{\partial h_0} = \prod_{k=K-1}^{0} \left(I + \frac{1}{K}J_G(h_k)\right).

Each step scales the residual Jacobian contribution by 1/K1/K rather than multiplying the full middle-stack Jacobian KK times. This reduces the per-step departure from the identity and provides a mechanism for mitigating depth-dependent gradient amplification after conversion. This analysis is consistent with the improved stability observed during continued training.

W2. Role and Interpretation of Validation-Loss Shock

We agree with the reviewer that validation-loss shock should not be treated as a standalone performance metric or used to select the expansion point (for instance, expanding at 0B tokens trivially produces minimal shock without saving compute). We will reframe Section 5.4.1 strictly as a diagnostic sanity check of the immediate functional discontinuity before resumed optimization.

At the same time, our local forward analysis (to be added to Section 4.2) shows a specific structural property beyond qualitative interpolation. Relative to the dense output M(h0)M(h_0), naive recurrence incurs error

uK−M(h0)=(K−1)GM(h0)+∑k=1K−1[GM(uk)−GM(h0)]u_K - M(h_0) = (K-1)G_M(h_0) + \sum_{k=1}^{K-1} \left[G_M(u_k) - G_M(h_0)\right]

whereas step-relaxed DDE gives

hK−M(h0)=1K∑k=0K−1[GM(hk)−GM(h0)].h_K - M(h_0) = \frac{1}{K}\sum_{k=0}^{K-1} \left[G_M(h_k) - G_M(h_0)\right].

Step relaxation cancels the explicit (K−1)GM(h0)(K-1)G_M(h_0) mismatch. Letting JG=∂GM(h)∂h∣h=h0J_G = \left.\frac{\partial G_M(h)}{\partial h}\right|_{h=h_0}, a first-order Taylor approximation GM(hk)≈GM(h0)+JG(hk−h0)G_M(h_k) \approx G_M(h_0) + J_G(h_k - h_0) yields

hK−M(h0)≈1K∑k=0K−1(I+1KJG)kGM(h0)−GM(h0)=(K−12KJG+O(JG2))GM(h0)h_K - M(h_0) \approx \frac{1}{K}\sum_{k=0}^{K-1}\left(I + \frac{1}{K}J_G\right)^k G_M(h_0) - G_M(h_0) = \left(\frac{K-1}{2K}J_G + \mathcal{O}(J_G^2)\right) G_M(h_0)

where the leading coefficient K−12K\frac{K-1}{2K} remains strictly below 12\frac{1}{2} as KK increases. This explains why the initial loss jump stays bounded below +0.22+0.22 even at K=32K=32 (Table 4) rather than growing rapidly with KK, keeping the converted checkpoint within a recoverable optimization basin.

Reviewer_1PfFOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a cheaper way to train looped transformers. They identify that merely repeating a trained transformer block causes a huge spike in validation loss. For a given training budget (for example, 100B tokens) they propose a 2-stage training where a dense block is trained on 80% of the data and it is looped over remaining 20% of the data. In this way, they achieve the same final architecture for much fewer training FLOPs and avoid the validation loss spike (which they refer to as "conversion shock").

Critical strengths

Using a training schedule to significantly save FLOPs is a fresh idea.

Critical weaknesses
  1. Section 5.3 defines L* as DDE’s own final validation loss, and Table 5 shows that from-scratch training ends lower (2.8881 vs. 2.9122 at 155M; 2.6939 vs. 2.7237 at 314M). The reported 30–47% FLOPs savings therefore buy a worse model, and these savings carry the paper’s contribution, since Section 2 states that neither the conversion nor the relaxed update is new.

  2. The paper compares DDE only against from-scratch looping and naive conversion, which raises loss by up to +9.3 (Table 4), and reports no results for the 12-layer dense model listed in Table 7 or for progressive-growth methods such as Sparse Growing Transformer. Without these baselines, the paper does not show that delaying recurrence beats other uses of the same compute.

Questions for authors

See above

7 comments

We thank Reviewer 1PfF for the time spent reviewing our manuscript and for noting that using a training schedule to save FLOPs in looped Transformers is a fresh idea. Below we clarify the evaluation protocol and project scope to address both concerns directly.

W1-1. Misinterpretation of Iso-Token Results vs. Iso-Loss FLOPs Savings

This criticism conflates fixed-token endpoints with iso-loss compute comparisons. Table 5 compares final validation losses after a fixed 100B token budget, without matching the cumulative compute. In contrast in Section 5.3 and Table 2 report cumulative FLOPs at matched validation loss, including the initial dense training stage. The reported 29.5%-47.1% saving comes from these iso-loss comparisons.

Evaluating expansion across G∈{40B,60B,80B}G \in \{40\mathrm{B}, 60\mathrm{B}, 80\mathrm{B}\} through 100B tokens confirms that DDE consistently reduces training compute at matched loss and even surpasses the naive scratch final loss while using fewer FLOPs and fewer tokens.

ScaleKKGG (B)Final loss ↓\downarrowDDE final FLOPs (EFLOPs)Saving vs. naiveSaving vs. relaxed
155M4402.893154.3721.1%^†14.2%
155M4602.898947.5724.7%22.4%
155M4802.912240.7831.1%29.9%
155M8402.874781.5522.7%21.3%
155M8602.882265.6935.4%34.3%
155M8802.899349.8447.7%47.1%
155M16402.8610135.9245.2%^†28.5%
155M16602.8709101.9456.4%^†43.7%
155M16802.889367.9664.5%59.8%
314M4402.7007143.4616.5%13.5%
314M4602.7076125.5323.5%22.0%
314M4802.7237107.5930.5%29.5%
314M8402.6801215.1922.5%20.5%
314M8602.6893173.3535.1%34.0%
314M8802.7095131.5047.6%46.6%

For †^{\dagger} rows, DDE finishes lower than naive scratch, so the target is the naive scratch final loss and CDDEC_{\mathrm{DDE}} is measured when DDE first reaches it (53.5953.59, 111.13111.13, and 88.5388.53 EFLOPs). At 155M and K=16K=16, expanding at G=40BG=40\mathrm{B} beats the naive scratch endpoint after just 87.84B tokens with 45.2% fewer FLOPs, and reaches 2.86102.8610 at 100B.

W1-2. Post-Conversion Optimization Dynamics and Contribution

Section 2 credits Chen et al. (2026a) for training-free forward inference on frozen weights. Our contribution addresses continued training after dense-to-recurrent conversion, which we will formalize via a local analysis in the revised Section 4.2. Let h0=P(Emb(x))h_0 = P(\mathrm{Emb}(x)), GM(h)=M(h)−hG_M(h) = M(h) - h, and M(h0)=h0+GM(h0)M(h_0) = h_0 + G_M(h_0). After conversion to KK steps with u0=h0u_0 = h_0, naive looping applies uk+1=uk+GM(uk)u_{k+1} = u_k + G_M(u_k) whereas step-relaxed DDE applies hk+1=hk+1KGM(hk)h_{k+1} = h_k + \frac{1}{K}G_M(h_k).

  • Forward perturbation. Naive recurrence yields error uK−M(h0)=(K−1)GM(h0)+∑k=1K−1[GM(uk)−GM(h0)]u_K - M(h_0) = (K-1)G_M(h_0) + \sum_{k=1}^{K-1}[G_M(u_k) - G_M(h_0)], whereas step-relaxed DDE gives hK−M(h0)=1K∑k=0K−1[GM(hk)−GM(h0)]h_K - M(h_0) = \frac{1}{K}\sum_{k=0}^{K-1}[G_M(h_k) - G_M(h_0)], canceling the explicit (K−1)GM(h0)(K-1)G_M(h_0) mismatch. With JG=∂GM(h)∂h∣h=h0J_G = \left.\frac{\partial G_M(h)}{\partial h}\right|_{h=h_0}, a first-order Taylor approximation GM(hk)≈GM(h0)+JG(hk−h0)G_M(h_k) \approx G_M(h_0) + J_G(h_k - h_0) gives

    hK−M(h0)≈1K∑k=0K−1[(I+1KJG)k−I]GM(h0)=(K−12KJG+O(JG2))GM(h0)h_K - M(h_0) \approx \frac{1}{K}\sum_{k=0}^{K-1}\left[\left(I + \frac{1}{K}J_G\right)^k - I\right] G_M(h_0) = \left(\frac{K-1}{2K}J_G + \mathcal{O}(J_G^2)\right) G_M(h_0)

    whose leading coefficient K−12K\frac{K-1}{2K} stays below 12\frac{1}{2} as KK grows.

  • Backward Jacobian conditioning. With JG(h)=∂GM(h)∂hJ_G(h) = \frac{\partial G_M(h)}{\partial h} and JM(h)=I+JG(h)J_M(h) = I + J_G(h), naive recurrence propagates gradients through ∂uK∂u0=∏k=K−10JM(uk)=∏k=K−10(I+JG(uk))\frac{\partial u_K}{\partial u_0} = \prod_{k=K-1}^{0} J_M(u_k) = \prod_{k=K-1}^{0} (I + J_G(u_k)). Step-relaxed DDE instead uses ∂hk+1∂hk=K−1KI+1KJM(hk)=I+1KJG(hk)\frac{\partial h_{k+1}}{\partial h_k} = \frac{K-1}{K}I + \frac{1}{K}J_M(h_k) = I + \frac{1}{K}J_G(h_k), yielding

    ∂hK∂h0=∏k=K−10(I+1KJG(hk))\frac{\partial h_K}{\partial h_0} = \prod_{k=K-1}^{0} \left(I + \frac{1}{K}J_G(h_k)\right)

    which scales each per-step residual Jacobian contribution by 1/K1/K rather than applying the full middle-stack Jacobian at every iteration. Empirically, in 155M delayed expansion at K=16K=16 (G=80BG=80\mathrm{B}), naive recurrence degrades to 3.1201 final loss whereas step-relaxed DDE reaches 2.8893 (a 0.2308-nat gain).

Comments on the paper

7 comments

We thank the reviewer for the constructive feedback and for noting that our matched-loss compute comparison accounts for the initial dense training stage and demonstrates smoother transitions at conversion. Below we address each point with additional experiments and analysis.

Single-Step DDE vs. Gradual Recurrence Schedules

Our approach does not require recurrence to be introduced at a single point; the step-relaxed update applies directly to gradual schedules as well. To test this in our parameter-matched carry-loop setting, we ran a 155M experiment from the 60B dense checkpoint using a gradual schedule (K=2K=2 for 60-80B60\text{-}80\text{B}, K=3K=3 for 80-90B80\text{-}90\text{B}, and K=4K=4 for 90-100B90\text{-}100\text{B}) and compared it against single-step expansion to K=4K=4 at 60B and 80B (all evaluated at K=4K=4 at 100B tokens):

Expansion schedule (155M, K=4K=4)Total block FLOPs (EFLOPs)Step-relaxed ↓\downarrowNaive ↓\downarrow
Single-step at 60B47.572.898872.89920
Gradual (60B→80B→90B→100B60\text{B} \to 80\text{B} \to 90\text{B} \to 100\text{B})41.912.906412.90677
Single-step at 80B40.782.912172.91261

Step-relaxed recurrence achieves slightly lower validation loss than Naive recurrence under each schedule. Moreover, final loss tracks cumulative block FLOPs smoothly across schedules (47.57→41.91→40.7847.57 \to 41.91 \to 40.78 EFLOPs), showing that single-step DDE remains a competitive and simpler alternative that avoids selecting multiple curriculum boundaries.

Practical Importance of Step Relaxation at Larger Loop Counts

We agree that at K=4K=4, the final validation-loss difference after 20B continuation tokens is small. To enable exact compute accounting with full backpropagation through all unrolled steps, our controlled pretraining runs evaluated moderate loop counts (K=4,8,16K=4, 8, 16). However, many recurrent-depth and mid-loop architectures operate at much higher loop counts such as K=32,64,K=32, 64, or more (e.g., Geiping et al., 2025). As KK scales toward these regimes, step relaxation transitions from smoothing the initial shock to being necessary for stable training and inference:

  • Degradation of naive looping at K=16K=16: In our 155M delayed-expansion runs (G=80B→100BG=80\text{B} \to 100\text{B}), Naive recurrence suffers severe post-conversion spikes (Figure 2) and degrades to a final validation loss of 3.1201 at K=16K=16 (worse than its K=4K=4 result of 2.91262.9126). In contrast, step-relaxed DDE improves monotonically with depth (2.91222.9122 at K=4→2.8993K=4 \to 2.8993 at K=8→2.8893K=8 \to 2.8893 at K=16K=16), outperforming Naive conversion by 0.2308 nats while saving 64.5% FLOPs over naive scratch (59.8% over step-relaxed scratch).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.