- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the one-loop tax for recurrent/looping models, which is the phenomenon that many loop models underperform their respective baselines when only one forward pass is allocated for the looped components. The introduced Elastic Latent Recurrence includes both the architecture design and the training recipe. The architecture consists of the input block, the recurrent block, which is repeated, and output block. At every loop of the recurrent block, the input to the initial loop is reinjected into the hidden state, and the hidden state at the end of each recurrent block pass is gated via an adaptive-delta rule. The ELR training consists of randomly sampling the loop count of each microbatch with some ranks forced to loop once, i.e., no repeats, to mitigate the one-loop tax. The authors reduce the overall training cost by only starting mixed-depth training partway through the full training run and fixing the MoE routing of subsequent loops to be the same as the first pass. The authors validate their architecture and training regime through three types of models, including hybrid and MoE designs. Their ablations demonstrate the benefit of the adaptive-delta state update and tradeoffs for delayed loop training and loop sampling.
Critical strengths
The scale of the experiments demonstrate the benefits of the ELR architecture and training procedure. The final reported models include both Hybrid and Hybrid-MoE architectures, which reflect the style of models used at the largest scale. Figure 2 demonstrates that through ELR the one loop forward pass performs comparably to the standard baseline and additional loops benefit performance, as expected.
Certain ablations help support design choices in the architecture and training regime. For instance, the mixed depth start and one-loop override ablations help elucidate the trade-offs for some of the more efficiency-focused components of ELR, and Figure 3a also helps show the importance of adaptive delta updates. The cross-loop route replay ablation also shows the compute-performance trade-off when determining whether to enable per-layer routing within a looped block. Overall, most ablations were insightful.
Critical weaknesses
I have some reservations about the gap between the ablation setup and the main models setup (Hybrid and Hybrid-MoE). One of my concerns is that the architecture setup between the two setups differ. For instance, ablations are trained with a maximum of 5 (k=5) loops while the full models are only trained to k=4. Another confounder is that the ablation model (GPT-0.5B) is a full global attention Transformer while the two “main” models are all hybrid setups using Gated DeltaProduct. Furthermore, there seems to be some discrepancy between the position encoding and activation between the three models, and on the optimization-side, the training horizon, warmup count, and optimizer seem to also differ. Furthermore, the ELR training setup, mainly maximum loop count, is also not constant across the three models and thus not constant across ablations and final experiments. It also seems like the baselines are not using adaptive-delta update? I think it would be helpful to benchmark this variant as well, since it is possible for the k=1 looped models to perform better due to better gradient/signal propagation (see Figure 3b which suggests that the one-loop tax might be cut down by the adaptive-delta update as the baseline is ~1.685 and the k=1 fixed is 1.660).
For some of the ablation experiments, I also have some questions/concerns about the setup. For instance, Table 9 seems to suggest that input re-injection actually harms performance by around 0.005 nats despite the footnote claiming the loss is within noise. Is there evidence supporting that 0.005 nats is within noise for a 0.5B model trained on 300B tokens? Furthermore, if input re-injection seems to be a wash, why is it used for other models?
Questions for authors
Do the ablations hold for the GDP-Hybrid and GDP-Hybrid MoE models? The ablations were mainly run on GPT-0.5B which lacks the input re-injection and also differs in non-trivial ways from the two main models.
Does the input reinjection formulation mean that the first loop’s input is actually 2h? What is this motivation, and how does this impact training dynamics and performance?
Why does the setting change across experiments and models? Is there a reason for these adjustments on optimizer, etc, as proper scaling would usually hold certain aspects constant.
For GDP-Hybrid-4B, GSM8K decreased by 5pp between baseline and k=1, but MATH500 increased by 5pp? What might explain these changes?
5 comments
Thank you for your response and additional experiments.
These new results have answered most of my questions.
However, it is still not clear to me how input re-injection improves the current model, and also doesn't answer whether 0.005 nats is significant enough for your setting. This is important, as the delayed-trained GDP ablations performs 0.009 nats worse than the fixed one-loop adaptive-delta model. Some rough examples of what architecture changes or general changes might result in a 0.005 nat difference vs a 0.01 nat difference would suffice.
In addition, could you elaborate on why the training setup differ across the different models? I don't think this part was answered in my original response.
On the 0.009 one-loop gap. This reflects the trade-off introduced by mixed-depth training. The delayed mixed-depth model has 0.009 higher loss at k=1 than the fixed-one-loop model, reflecting the trade-off between one-loop quality and supporting multiple inference depths. Increasing one-loop supervision shifts this balance toward k=1. At k=2 and k=4, the delayed model improves on the fixed-one-loop model's k=1 loss by 0.0143 and 0.0175 nats, respectively.
A paired sequence bootstrap over our 88M-token validation set gives 95% confidence intervals of approximately $\pm$0.0003 nats, so a 0.005-nat difference exceeds evaluation uncertainty. We therefore agree that “within noise” was too strong. We have not measured variability across training seeds at 300B tokens. For scale, other design choices change loss by approximately 0.002–0.007 nats, while increasing inference depth from k=2 to k=3 improves loss by approximately 0.008. We will replace “within noise” with this calibration and limitation.
The main models use input reinjection, while the GPT ablations use identity input. Reinjection increases loss by approximately 0.005 nats on GPT and 0.001 on SWA-Hybrid, with a difference below 0.002 on GDP. These comparisons do not establish a current performance benefit. We retained reinjection to support future truncated-backpropagation variants, and will clarify that this is a design motivation rather than a demonstrated advantage in the present experiments.
All training configurations are reported in Appendix A.3. Across models, we use:
- same global batch size
- approximately 3% warmup
- a WSD schedule with learning-rate decay over the final 15% of training.
- Only the MoE experiments use Muon.
Training horizons differ: the GPT ablations use 300B tokens, while GDP-Hybrid-4B uses 1T tokens to evaluate whether ELR remains effective at a larger token budget. We limited GDP-Hybrid-MoE-30B-A3B to 300B tokens because of training cost, and this budget was sufficient to evaluate cross-loop router caching.
These experiments show that the ELR recipe works across several architectures and optimizers. Within each model family, the non-recurrent baseline and ELR counterpart use matching training settings, ensuring a controlled comparison.
We used a maximum of five training loops for the smaller GPT ablations and four for the larger main models to limit training compute. We adjusted the one-loop override rate to preserve approximately the same one-loop supervision probability: , giving ~32% with a 15% override at five loops and 10% override at four loops.
Thank you for the review.
Difference between ablations and main experiments
Our goal was to show that our recipe transfers across architectures. The GPT ablations isolated the training choices in a standard Transformer and the full recipe was evaluated for GDP hybrid models. We agree the presentation did not sufficiently establish whether these training findings transfer to GDP.
We provide ablations with GDP-0.5B model trained with maximum training depth of four loops, matching the main experiments. All values are validation losses and the non-recurrent baseline is 1.6457.
Fixed versus mixed depth
GDP reproduces the elasticity finding: fixed-depth models specialize to their training depth, while mixed-depth training supports all four inference depths with one checkpoint.
One-loop overrides
With mixed-depth training from scratch, increasing overrides from 0% to 10% lowers one-loop loss from 1.6579 to 1.6520, while four-loop loss increases from 1.6112 to 1.6135. This supports the shallow-versus-deep trade-off observed on GPT, without establishing statistical significance or equivalence.
Delayed Training
We evaluate delayed looping with 10% overrides during the mixed phase:
The delayed schedule retains 80% of the four-loop improvement over baseline (0.0256 versus 0.0322), with a one-loop gap of 0.0013.
Different Architectures
We also evaluate ELR with two other hybrid models: Mamba-2 and SWA-Hybrid (alternating sliding window and full attention in 3:1 ratio).
Both these architectures retain val loss close to the non-recurrent baseline at k=1 while showing significant gains as we increase the number of loops during inference.
Maximum loop count and override rate
The GPT ablations used a smaller model trained up to five loops. We used four loops for the larger main models to limit training compute. The new GDP ablations above also use four loops, matching the main experiments.
Across both settings, we target approximately one-third of training ranks receiving one-loop supervision. With uniform sampling over 1,…,Kmax and a one-loop override fraction r, the effective one-loop probability is p1=r+(1−r)/Kmax. Thus, a 10% override at Kmax=4 gives p1=32.5, while a 15% override at $K_{\max}=5$ gives $p_1=32%$. These settings preserve a consistent one-loop supervision rate across different maximum depths.
Baseline not using adaptive delta
The first loop’s input is 2h?
Yes. With $s_0 = h^{\mathrm{in}}$, input reinjection gives 2hin at loop 1. We use this rule to apply the same update at every loop. Since recurrent block is pre-norm, it reduces sensitivity to input scale. Input reinjection also supports future experiments with truncated backpropagation by preserving a direct gradient path to the input block when recurrent states are detached. Although our current experiments use full backpropagation, we include the reinjection ablation to show that this design choice preserves the main training trends, with very small validation-loss differences across all the loop counts.
GSM8K and MATH-500 Difference
We checked whether this is checkpoint noise by evaluating four adjacent checkpoints per arm under identical settings. The pattern replicates in later checkpoints, so it is systematic, but overall math performance is not degraded and the average over the two benchmarks is essentially unchanged. We do not yet know why the redistribution happens, though per sample audits rule out prompt format artifacts. What we can say is that math performance improves consistently as k increases, which is the mechanism the architecture targets.