acceptodds
Under review as a conference paper at ICLR 2027

One Model, Many Depths: A Recipe for Elastic-Depth Latent Recurrent Language Models

Abstract

A latent recurrent, or looped, language model applies a shared block of layers repeatedly, so one checkpoint can trade inference compute for accuracy by changing the loop count. At one loop, however, such models typically fall well short of a non-recurrent model of equal inference depth. This one-loop tax makes a separate non-recurrent model the better choice for latency-sensitive serving. We present Elastic Latent Recurrence (ELR), a recipe for training looped language models without this tax. Run at one loop, an ELR model costs the same to serve as a non-recurrent model of the same size and, when the two are trained on the same data, reaches similar downstream accuracy. Each additional loop of the same checkpoint then buys more accuracy for more inference compute. We show that two choices determine whether the one-loop accuracy survives mixed-depth training. The first is a gated update that sets a learned step size along each proposed state revision. The second is mixing loop counts within each optimizer update rather than across updates, which we implement by forcing a rotating subset of data-parallel ranks to run one loop. On a B dense hybrid model trained for 1T tokens and a hybrid mixture-of-experts model with 30B total and 3B active parameters trained for 300B tokens, each alongside a matched baseline, increasing inference depth from one to four loops improves the ten-task average by 7.25 and 4.75 points, respectively. Both models acquire elastic depth late in pretraining. Training at one loop for the first three quarters of pretraining and at mixed depth for the last quarter costs about 1.16 the baseline training FLOPs, and the resulting checkpoint improves at every added loop while giving the best one-loop quality of the schedules we tested. In our hybrid MoE experiments, reusing first-loop expert assignments on later loops removes repeated router evaluation and cuts mixed-depth training update time by 11.7% with no measurable change in downstream accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Transfomers, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xdpXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the one-loop tax for recurrent/looping models, which is the phenomenon that many loop models underperform their respective baselines when only one forward pass is allocated for the looped components. The introduced Elastic Latent Recurrence includes both the architecture design and the training recipe. The architecture consists of the input block, the recurrent block, which is repeated, and output block. At every loop of the recurrent block, the input to the initial loop is reinjected into the hidden state, and the hidden state at the end of each recurrent block pass is gated via an adaptive-delta rule. The ELR training consists of randomly sampling the loop count of each microbatch with some ranks forced to loop once, i.e., no repeats, to mitigate the one-loop tax. The authors reduce the overall training cost by only starting mixed-depth training partway through the full training run and fixing the MoE routing of subsequent loops to be the same as the first pass. The authors validate their architecture and training regime through three types of models, including hybrid and MoE designs. Their ablations demonstrate the benefit of the adaptive-delta state update and tradeoffs for delayed loop training and loop sampling.

Critical strengths

The scale of the experiments demonstrate the benefits of the ELR architecture and training procedure. The final reported models include both Hybrid and Hybrid-MoE architectures, which reflect the style of models used at the largest scale. Figure 2 demonstrates that through ELR the one loop forward pass performs comparably to the standard baseline and additional loops benefit performance, as expected.

Certain ablations help support design choices in the architecture and training regime. For instance, the mixed depth start and one-loop override ablations help elucidate the trade-offs for some of the more efficiency-focused components of ELR, and Figure 3a also helps show the importance of adaptive delta updates. The cross-loop route replay ablation also shows the compute-performance trade-off when determining whether to enable per-layer routing within a looped block. Overall, most ablations were insightful.

Critical weaknesses

I have some reservations about the gap between the ablation setup and the main models setup (Hybrid and Hybrid-MoE). One of my concerns is that the architecture setup between the two setups differ. For instance, ablations are trained with a maximum of 5 (k=5) loops while the full models are only trained to k=4. Another confounder is that the ablation model (GPT-0.5B) is a full global attention Transformer while the two “main” models are all hybrid setups using Gated DeltaProduct. Furthermore, there seems to be some discrepancy between the position encoding and activation between the three models, and on the optimization-side, the training horizon, warmup count, and optimizer seem to also differ. Furthermore, the ELR training setup, mainly maximum loop count, is also not constant across the three models and thus not constant across ablations and final experiments. It also seems like the baselines are not using adaptive-delta update? I think it would be helpful to benchmark this variant as well, since it is possible for the k=1 looped models to perform better due to better gradient/signal propagation (see Figure 3b which suggests that the one-loop tax might be cut down by the adaptive-delta update as the baseline is ~1.685 and the k=1 fixed is 1.660).

For some of the ablation experiments, I also have some questions/concerns about the setup. For instance, Table 9 seems to suggest that input re-injection actually harms performance by around 0.005 nats despite the footnote claiming the loss is within noise. Is there evidence supporting that 0.005 nats is within noise for a 0.5B model trained on 300B tokens? Furthermore, if input re-injection seems to be a wash, why is it used for other models?

Questions for authors

Do the ablations hold for the GDP-Hybrid and GDP-Hybrid MoE models? The ablations were mainly run on GPT-0.5B which lacks the input re-injection and also differs in non-trivial ways from the two main models.

Does the input reinjection formulation mean that the first loop’s input is actually 2h? What is this motivation, and how does this impact training dynamics and performance?

Why does the setting change across experiments and models? Is there a reason for these adjustments on optimizer, etc, as proper scaling would usually hold certain aspects constant.

For GDP-Hybrid-4B, GSM8K decreased by 5pp between baseline and k=1, but MATH500 increased by 5pp? What might explain these changes?

5 comments

Thank you for the review.

Difference between ablations and main experiments

Our goal was to show that our recipe transfers across architectures. The GPT ablations isolated the training choices in a standard Transformer and the full recipe was evaluated for GDP hybrid models. We agree the presentation did not sufficiently establish whether these training findings transfer to GDP.

We provide ablations with GDP-0.5B model trained with maximum training depth of four loops, matching the main experiments. All values are validation losses and the non-recurrent baseline is 1.6457.

Fixed versus mixed depth

GDP reproduces the elasticity finding: fixed-depth models specialize to their training depth, while mixed-depth training supports all four inference depths with one checkpoint.

Train Configk=1k=2k=3k=4
Mixed 1–41.65791.62021.61201.6112
Fixed 11.63761.81542.17592.6216
Fixed 22.52301.61561.70511.9261
Fixed 34.08742.03291.60131.6556
Fixed 43.94072.51541.91521.5952

One-loop overrides

With mixed-depth training from scratch, increasing overrides from 0% to 10% lowers one-loop loss from 1.6579 to 1.6520, while four-loop loss increases from 1.6112 to 1.6135. This supports the shallow-versus-deep trade-off observed on GPT, without establishing statistical significance or equivalence.

Delayed Training

We evaluate delayed looping with 10% overrides during the mixed phase:

Training schedulek=1k=2k=3k=4
Mixed from scratch1.65201.62101.61411.6135
One loop for first 75%, then mixed1.64701.62331.62051.6201

The delayed schedule retains 80% of the four-loop improvement over baseline (0.0256 versus 0.0322), with a one-loop gap of 0.0013.

Different Architectures

We also evaluate ELR with two other hybrid models: Mamba-2 and SWA-Hybrid (alternating sliding window and full attention in 3:1 ratio).

ArchitectureNon-recurrent baselinek=1k=2k=3k=4
Mamba-2 (0.5B)1.65091.65731.62851.62251.6221
SWA-Hybrid (0.5B)1.67981.68301.65501.64981.6493

Both these architectures retain val loss close to the non-recurrent baseline at k=1 while showing significant gains as we increase the number of loops during inference.

Maximum loop count and override rate

The GPT ablations used a smaller model trained up to five loops. We used four loops for the larger main models to limit training compute. The new GDP ablations above also use four loops, matching the main experiments.

Across both settings, we target approximately one-third of training ranks receiving one-loop supervision. With uniform sampling over 1,…,Kmax⁡1,\ldots,K_{\max} and a one-loop override fraction rr, the effective one-loop probability is p1=r+(1−r)/Kmax⁡p_1=r+(1-r)/K_{\max}. Thus, a 10% override at Kmax⁡=4K_{\max}=4 gives p1=32.5p_1=32.5%, while a 15% override at $K_{\max}=5$ gives $p_1=32%$. These settings preserve a consistent one-loop supervision rate across different maximum depths.


Baseline not using adaptive delta

  • Adaptive-delta is part of our ELR recipe, so we compare against a standard non-recurrent baseline without this update. Our goal is to avoid degradation at k=1k=1 while enabling further quality improvements with additional inference loops.
  • We observe that the k=1k=1 variant slightly outperforms the non-recurrent baseline. Figure 3b already includes this comparison: the fixed k=1k=1 model is equivalent to the non-recurrent baseline with adaptive-delta.

The first loop’s input is 2h?

Yes. With $s_0 = h^{\mathrm{in}}$, input reinjection gives 2hin2h^{\mathrm{in}} at loop 1. We use this rule to apply the same update at every loop. Since recurrent block is pre-norm, it reduces sensitivity to input scale. Input reinjection also supports future experiments with truncated backpropagation by preserving a direct gradient path to the input block when recurrent states are detached. Although our current experiments use full backpropagation, we include the reinjection ablation to show that this design choice preserves the main training trends, with very small validation-loss differences across all the loop counts.


GSM8K and MATH-500 Difference

We checked whether this is checkpoint noise by evaluating four adjacent checkpoints per arm under identical settings. The pattern replicates in later checkpoints, so it is systematic, but overall math performance is not degraded and the average over the two benchmarks is essentially unchanged. We do not yet know why the redistribution happens, though per sample audits rule out prompt format artifacts. What we can say is that math performance improves consistently as k increases, which is the mechanism the architecture targets.

Thank you for your response and additional experiments.

These new results have answered most of my questions.

However, it is still not clear to me how input re-injection improves the current model, and also doesn't answer whether 0.005 nats is significant enough for your setting. This is important, as the delayed-trained GDP ablations performs 0.009 nats worse than the fixed one-loop adaptive-delta model. Some rough examples of what architecture changes or general changes might result in a 0.005 nat difference vs a 0.01 nat difference would suffice.

In addition, could you elaborate on why the training setup differ across the different models? I don't think this part was answered in my original response.

On the 0.009 one-loop gap. This reflects the trade-off introduced by mixed-depth training. The delayed mixed-depth model has 0.009 higher loss at k=1 than the fixed-one-loop model, reflecting the trade-off between one-loop quality and supporting multiple inference depths. Increasing one-loop supervision shifts this balance toward k=1. At k=2 and k=4, the delayed model improves on the fixed-one-loop model's k=1 loss by 0.0143 and 0.0175 nats, respectively.

A paired sequence bootstrap over our 88M-token validation set gives 95% confidence intervals of approximately $\pm$0.0003 nats, so a 0.005-nat difference exceeds evaluation uncertainty. We therefore agree that “within noise” was too strong. We have not measured variability across training seeds at 300B tokens. For scale, other design choices change loss by approximately 0.002–0.007 nats, while increasing inference depth from k=2 to k=3 improves loss by approximately 0.008. We will replace “within noise” with this calibration and limitation.

The main models use input reinjection, while the GPT ablations use identity input. Reinjection increases loss by approximately 0.005 nats on GPT and 0.001 on SWA-Hybrid, with a difference below 0.002 on GDP. These comparisons do not establish a current performance benefit. We retained reinjection to support future truncated-backpropagation variants, and will clarify that this is a design motivation rather than a demonstrated advantage in the present experiments.


All training configurations are reported in Appendix A.3. Across models, we use:

  • same global batch size
  • approximately 3% warmup
  • a WSD schedule with learning-rate decay over the final 15% of training.
  • Only the MoE experiments use Muon.

Training horizons differ: the GPT ablations use 300B tokens, while GDP-Hybrid-4B uses 1T tokens to evaluate whether ELR remains effective at a larger token budget. We limited GDP-Hybrid-MoE-30B-A3B to 300B tokens because of training cost, and this budget was sufficient to evaluate cross-loop router caching.

These experiments show that the ELR recipe works across several architectures and optimizers. Within each model family, the non-recurrent baseline and ELR counterpart use matching training settings, ensuring a controlled comparison.

We used a maximum of five training loops for the smaller GPT ablations and four for the larger main models to limit training compute. We adjusted the one-loop override rate to preserve approximately the same one-loop supervision probability: p1=r+(1−r)/Kmax⁡p_1=r+(1-r)/K_{\max}, giving ~32% with a 15% override at five loops and 10% override at four loops.

Reviewer_2M4dOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Recurrent-depth models can outperform standard Transformers on downstream tasks by allocating additional computation through repeated applications of shared layers. However these models often underperform their fixed-depth counterparts when evaluated with a single iteration. This paper proposes a training method that aims to improve this trade-off. The method introduces a delta-rule formulation for recurrent state updates and extends the approach to MoE architectures. The reported results show that the resulting models remain competitive with recurrent baselines trained at a fixed depth, while supporting flexible loop counts at inference time.

Critical strengths

The paper clearly identifies the tension between strong single-pass performance and the ability to benefit from additional recurrent computation. The delta-rule formulation is well motivated, and the clearly presented state-update ablations provide evidence supporting the adaptive variant across the evaluated loop counts. This strengthens the case that the proposed mechanism addresses a meaningful limitation of recurrent-depth models. The extension to MoE architectures and the results across multiple model scales reflects practical generalization that goes beyond the the initial design studies.

Critical weaknesses
  • Limited evidence for the downstream relevance of the ablation conclusions: the key state-update and loop-sampling ablations are run on GPT-0.5B and evaluated through validation loss. Some margins are small, particularly the single-pass comparison against the non-recurrent baseline. Moreover, it is unclear whether the relative advantages of the selected configurations carry over to downstream tasks. These limitations reduce confidence in how reliably the ablations justify the final recipe and its claimed benefits.

  • Insufficient comparison with existing mechanisms for recurrent state updates: the fixed-depth comparisons establish the benefit of having different inference depths, while the residual and replacement ablations help motivate the proposed update mechanism. However, these comparisons do not establish its relative merits against closely related approaches such as Parcae (Prairie et al., 2026) and Hyperloop Transformers (Zeitoun et al., 2026).

Questions for authors
  1. Can you provide downstream evaluations of the existing checkpoints for the principal state-update and sampling alternatives, at one loop and at a larger loop count? This way we can assess if the validation-loss rankings support the practical benefits motivating the chosen recipe.

  2. Could you compare adaptive delta with some alternatives such as the recurrent update used in Parcae or the inter-loop hyper-connections of Hyperloop Transformers under the same training-depth distribution and comparable parameter and compute budgets?

7 comments

Thank you for the review and feedback.

Validation loss trends transferring to downstream benchmarks

We use GPT-0.5B models trained on 300B tokens for controlled ablations. At this model scale and training budget, downstream evaluations are very sensitive and unreliable, so we used validation loss.

To show that the validation loss trends translate to downstream benchmark results, we report below the validation loss of the GDP and the GDP-MoE runs shown in the main experiments section.

ModelNon-recurrent baselinek=1k=2k=3k=4
GDP-4B1.29401.29921.27021.25981.2596
GDP-MoE-30B-A3B1.23521.23631.21521.20881.2078

Adaptive Delta vs Parcae and Hyperloop

We evaluated Parcae and Hyperloop using the same GPT-0.5B model, 300B-token budget, and mixed-depth training recipe: uniform sampling over 1–5 loops with a 15% one-loop rank override.

  • Parcae and adaptive delta share their update parameters across loops, making them a natural comparison.
  • Hyperloop additionally introduces loop-specific gates and position embeddings, so we compare it with adaptive delta augmented with the AdaLN loop conditioning method.

Shared recurrent updates

Methodk=1k=2k=3k=4k=5
Adaptive delta (15% override)1.68301.63911.62991.62781.6273
Parcae (15% override)1.69041.64131.62981.62701.6264

Adaptive delta has a lower one-loop loss.

Mechanisms with explicit loop conditioning

Methodk=1k=2k=3k=4k=5
Adaptive delta + AdaLN (15% override)1.68141.63221.61791.61151.6084
Hyperloop (15% override)1.68041.62921.61761.61321.6100
Hyperloop (0% override)1.69751.63201.61721.61241.6084

With AdaLN conditioning, adaptive delta achieves similar validation loss as Hyperloop. Our contribution is the full ELR recipe, and we do not claim that adaptive delta is the best state-update rule. Hyperloop achieves comparable losses under the same recipe, and adding our 15% one-loop override improves its one-loop loss from 1.6975 to 1.6804 while preserving similar multi-loop performance. This supports the recipe’s value across different update mechanisms.

These comparisons use mixed-depth training from scratch. Delayed mixed-depth training introduces another consideration for Hyperloop. Its later-loop gates and position embeddings receive no training signal during the initial one-loop phase, whereas adaptive delta’s shared update parameters are trained throughout. Hyperloop may therefore require additional adaptation when more loops are introduced, although we have not tested this effect.

Thank you for the additional experiments. The Parcae and Hyperloop comparisons address my concern about missing comparisons with related recurrent update mechanisms. I appreciate the clarification that the contribution is the overall ELR recipe.

My first concern remains only partially addressed. The new large-model validation-loss table shows that increasing inference depth improves validation loss for the selected configurations, consistent with their aggregate downstream results. However, my question concerned a different comparison: whether the validation-loss rankings among the state-update and sampling alternatives are reflected in downstream performance at a fixed inference depth. The supplied results do not evaluate those alternatives downstream.

Could you provide a limited downstream comparison using existing GPT-0.5B checkpoints at one loop and one larger loop count? Two particularly informative comparisons would be adaptive delta versus ordinary delta, and rank overrides versus synchronized weighted sampling with the same marginal depth distribution. A small set of tasks that is informative at this model scale, with per-task scores and uncertainty estimates where feasible, would help. If the evaluations cannot meaningfully distinguish these alternatives, reporting that evidence would clarify the limitation and help appropriately qualify the claims about their downstream relevance.

For the new mechanism comparisons, could you also specify the AdaLN configuration and report the additional parameter and compute costs of the compared methods? The results show that Hyperloop has lower reported loss than the original adaptive-delta configuration at every tested depth, while adaptive delta with AdaLN is competitive.

I see the Hyperloop override result as supporting the combined sampling intervention, since it changes both the one-loop marginal probability and within-update mixing. The proposed delayed-training advantage over Hyperloop should likewise remain clearly identified as an untested hypothesis.

Downstream comparison of the ablation alternatives

We have evaluated the existing GPT-0.5B checkpoints trained on 300B tokens at one and five loops on four downstream tasks. We compare adaptive delta with ordinary delta, and rank overrides with synchronized weighted sampling using the same marginal depth distribution.

MethodLoopsARC-CCommonsenseQARACE-highWinoGrandeAverage
Adaptive delta + 15% overrides131.4824.4130.0555.0935.26
Ordinary delta + 15% overrides126.9620.7226.9953.6732.08
Adaptive delta + synchronized weighted sampling130.0323.9128.5254.2234.17
Adaptive delta + 15% overrides532.6828.2634.0755.8037.70
Ordinary delta + 15% overrides529.0119.9830.8156.2034.00
Adaptive delta + synchronized weighted sampling530.6324.8232.0653.5135.25

The observed downstream rankings largely agree with the validation-loss rankings:

  • State update: Adaptive delta improves the average score over ordinary delta by 3.17 points at one loop and 3.70 at five loops.
  • Loop sampling: The 15% rank override improves the average score over synchronized weighted sampling by 1.09 percentage points at one loop and 2.45 at five loops, with higher scores on all four tasks at both depths. Because this comparison holds the marginal depth distribution fixed, it more directly addresses the benefit of mixing depths across ranks within an update.

AdaLN configuration and overhead

Our AdaLN configuration uses conditioning width 256, normalized loop progress and budget, rank-256 conditioned channel mixing, input mixing, and clone initialization. It supplies normalization scale/shift and residual and loop-seam gates, with conditioning-site checkpointing and batched heads enabled.

MethodAdded parameters¹Forward runtime, k=1Forward runtime, k=5
Adaptive delta1.05M (0.21%)1.00×1.00×
Parcae2.10M (0.42%)1.00×0.99×
Hyperloop0.25M (0.05%)1.02×1.04×
Adaptive delta + AdaLN32.92M (6.60%)1.09×1.15×

¹Added parameters are relative to the common 498.50M backbone. Runtime is normalized to adaptive delta at each depth. Hyperloop’s matched training iteration time was also 0.90× that of adaptive delta with AdaLN.

We agree that Hyperloop has lower reported loss than unconditioned adaptive delta at every tested depth. Adaptive delta with AdaLN is competitive, but incurs greater parameter and runtime costs. We do not claim superiority over Hyperloop.

The Hyperloop override result supports the combined sampling intervention, since it changes both the one-loop marginal probability and within-update mixing. Our matched-marginal control isolates this distinction only for adaptive delta. The proposed delayed-training advantage remains untested and is not evidence of superiority.

Reviewer_bFgLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores how to make looping (recurrent) models adaptable, and perform the same with 1 loop (i.e. no loop) as models trained without any looping. They explore adding adaptive delta states rather than fully replacing a state block with the loop's output, and they also train with multiple loop counts in one microbatch (most GPUs train on some loop count K and a small portion train on K=1). They also only increase above one loop late in training (75% of the way through) as a pretraining cost minimization technique. For looping MoEs, they also calculate router assignments only on the first loop and reuse them on later loops. They train a 4B dense hybrid model to 1T tokens and a 30 total/3b active MoE to 300B tokens, and evaluate on standard tasks. They find that training with up to K=4 performs noticeably better than K=1, though it is diminishing; K=2/3 gets most of the benefit of 4.

Critical strengths

This paper makes a convincing case for their training scheme, and shows strong results at a real, useful scale on standard evaluation benchmarks. They also show, using standard, popular tools (vLLM) that their model when using K=1 has ~the same latency as a standard non-looping model, showing that they have improved the latency/single loop performance tradeoff. The gains they show on popular math evals are well outside of noise for both evals as well, so they seem to have made real improvements overall.

Critical weaknesses

Understandably, there are no multi-seed comparisons due to cost, though this would help strengthen the case if compute were abundant. The ablations also don't fully match the main training runs (e.g. one used a standard transformer vs the hybrid model used for the main runs).

At the 4B scale, many of the evals do drop at K=1, but the average is higher because of a few evals (math and winogrande) carrying them up. Most of the swings are probably within noise, but further analysis of this would be nice.

The mechanism itself is under-explored, and the experimental results would benefit from being expanded a bit. E.g. what happens if you evaluate outside the trained K range? A larger latency study would also be appreciated. Also, if I understand correctly, the kv cache memory usage grows with K? If so, an analysis of this would help with the systems story.

Also, unless I missed it, section 4.1 says the validation losses were reported, but I didn't see them in the paper.

Questions for authors

Do you think calculating the experts once before looping K times is worth the tradeoff? I assume the model would have a higher performance ceiling if it could use arbitrary experts when looping, but I'm not sure what the magnitude of the impact is.

1 comment

Thank you for the review and feedback.

Difference between ablations and main experiments

Our goal was to show that our recipe transfers across architectures.. The GPT ablations isolated the training choices in a standard Transformer and the full recipe was evaluated for GDP hybrid models. We agree the presentation did not sufficiently establish whether these training findings transfer to GDP.

We provide ablations with GDP-0.5B model trained with maximum training depth of four loops, matching the main experiments. All values are validation losses and the non-recurrent baseline is 1.6457.

Fixed versus mixed depth

GDP reproduces the elasticity finding: fixed-depth models specialize to their training depth, while mixed-depth training supports all four inference depths with one checkpoint.

Train Configk=1k=2k=3k=4
Mixed 1–41.65791.62021.61201.6112
Fixed 11.63761.81542.17592.6216
Fixed 22.52301.61561.70511.9261
Fixed 34.08742.03291.60131.6556
Fixed 43.94072.51541.91521.5952

One-loop overrides

With mixed-depth training from scratch, increasing overrides from 0% to 10% lowers one-loop loss from 1.6579 to 1.6520, while four-loop loss increases from 1.6112 to 1.6135. This supports the shallow-versus-deep trade-off observed on GPT, without establishing statistical significance or equivalence.

Delayed Training

We evaluate delayed looping with 10% overrides during the mixed phase:

Training schedulek=1k=2k=3k=4
Mixed from scratch1.65201.62101.61411.6135
One loop for first 75%, then mixed1.64701.62331.62051.6201

The delayed schedule retains 80% of the four-loop improvement over baseline (0.0256 versus 0.0322), with a one-loop gap of 0.0013.

Different Architectures

We also evaluate ELR with two other hybrid models: Mamba-2 and SWA-Hybrid (alternating sliding window and full attention in 3:1 ratio).

ArchitectureNon-recurrent baselinek=1k=2k=3k=4
Mamba-2 (0.5B)1.65091.65731.62851.62251.6221
SWA-Hybrid (0.5B)1.67981.68301.65501.64981.6493

Both these architectures retain val loss close to the non-recurrent baseline at k=1 while showing significant gains as we increase the number of loops during inference.


Evaluating outside trained K range

We evaluated the GPT-0.5B model with 15% loop override up to $K=10$, beyond its training range of $K=1$-$5$. Gains saturate around the maximum training depth: validation loss after $K=5$ remains nearly unchanged up to $K=10$. Performance remains stable beyond the training range, although additional loops provide no further improvement.

Loops12345678910
Val Loss1.6831.63911.62991.62781.62731.62731.62751.62781.6281.6283

KV cache growth

Each layer in the recurrent block maintains a separate KV cache for each loop, while the input and output block caches remain fixed. For blocks with LinL_{\mathrm{in}}, LrecL_{\mathrm{rec}}, and LoutL_{\mathrm{out}} layers, KV memory scales as Lin+KLrec+LoutL_{\mathrm{in}}+K L_{\mathrm{rec}}+L_{\mathrm{out}}. Model-weight memory remains unchanged, so additional loops trade increased compute and cache memory for improved quality.


Flexibility with MoE Router Caching

We agree that fresh routing offers more flexibility and potentially a higher performance ceiling. We reuse only expert selections and mixing weights. Experts still process the updated states every loop. Empirically, four-loop replay increases validation loss by about 0.006 on the same checkpoint, while separately trained models achieve similar benchmark averages (70.83 vs. 70.90). Replay reduces mixed-depth training step time by 11.7%, but inference forward time by only 1–3%. We therefore consider it a useful optional training-efficiency tradeoff, without claiming equal performance ceilings.


Validation losses for Section 4.1

Thank you for catching this omission. The validation losses for the main experiments are:

ModelNon-recurrent baselinek=1k=2k=3k=4
GDP-Hybrid-4B1.29401.29921.27021.25981.2596
GDP-Hybrid-30B-A3B1.23521.23631.21521.20881.2078

We will add these results to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.