acceptodds
Under review as a conference paper at ICLR 2027

What Makes Recurrence Effective in Looped Language Models?

Abstract

Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding parameters. However, it remains unclear when additional recurrence is beneficial and how architectural choices affect its effectiveness. Through controlled experiments, we systematically examine (1) when recurrence helps, (2) where it should be applied, and (3) how its conditioning affects performance. Our evaluation covers inference budgets below, within, and beyond the training horizon under knowledge and reasoning tasks. (1) We find that recurrence can improve reasoning beyond the training horizon while degrading knowledge performance, but harder reasoning instances do not consistently benefit more. (2) Performance also depends on how distinct layers and recurrent iterations are allocated, showing that effective depth alone is insufficient to predict behavior. Non-recurrent output layers improve robustness to under-unrolling, while the preferred placement of input and output layers varies with inference budget. (3) Finally, we find that conventional initial-state injection offers limited robustness to varying recurrence depth. We therefore propose history-state injection as an alternative, and show that channel-wise history-state injection combined with timestep conditioning offers a low-cost and more effective design, better preserving knowledge under extended unrolling while improving robustness across inference budgets. Overall, our results clarify when recurrent computation helps, where it fails, and offer practical guidelines for designing LoopLMs across variable inference budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, looped language models, reasoning, test-time scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fexHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically studies how loop count, non-recurrent layer placement, and conditioning affect looped language models. Additional recurrence can improve reasoning while hurting knowledge performance, and harder examples do not necessarily benefit more. The authors also propose combining history-state differences with timestep gating.

Critical strengths

S1: Systematic controlled experiments. The paper examines multiple architectural factors and stratifies tasks by reasoning depth, helping clarify when recurrence is useful.

S2: Informative negative results. More loops, more complex state injection, and more conditioning mechanisms do not guarantee better performance. These findings provide useful guidance for model design.

Critical weaknesses

W1: Limited robustness of the conclusions. All training runs use a single seed, and the combined method is mainly evaluated on Qwen 4×7. Its overall score is only marginally higher than the strongest simple history-only variant, limiting support for a broad design recommendation.

W2: imited validation across model scales. The experiments are restricted to small models, and the combined conditioning method is mainly evaluated on Qwen 4×7, with approximately 219M parameters. Its effectiveness at larger scales, such as 7B or above, remains unverified, limiting the generality of the design recommendations.

Questions for authors

Q1: Do the gains from combining history-state conditioning with timestep gating hold across additional seeds or architectures? How does the combination compare with the strongest history-only variant under the same settings?

Q2: Do you have results at larger model scales, or evidence showing how the gains change with model size? Such results would help assess whether the proposed conditioning method remains effective beyond the small-model settings studied here.

7 comments
AuthorsOpenReview ↗

Response to Reviewer fexH

We thank the reviewer for highlighting the controlled experiments and the value of the negative results. We agree that robustness across training seeds and the scale of the method-specific evidence matter for a design recommendation, and we address both below.

W1 / Q1. Robustness across seeds and architectures

Statistical significance at the original seed. We propagate the lm-evaluation-harness standard errors of the 11 tasks to the Overall score, which gives about 0.36 points per model. We then test the differences at D=84D=84 on Qwen 4×74\times7 (unpaired, which is conservative because all models are evaluated on the same items):

H+LG vs.BaseLoopLGH (matched component)Scalar history w=4w=4
Difference (Overall)+1.70+1.58+1.16+0.06
zz3.43.12.30.1

The gains over BaseLoop, LG and the matched H component exceed evaluation sampling error. The 0.06-point margin over scalar history w=4w=4, the strongest history-only variant, does not, and we do not claim a ranking between those two.

Additional training seed. Evaluation error does not capture variation across training runs. We therefore retrained H, H+LG and scalar history w=4w=4 with seed 123, keeping the data order and method definitions fixed. Overall accuracy on Qwen 4×74\times7 (training depth D=28D=28):

MethodSeedD=28D=28D=56D=56D=84D=84
BaseLoop4236.8036.5935.60
LG4236.6736.4335.72
Channel-wise history, w=2w=2 (H)42 / 12336.64 / 36.5436.77 / 36.5236.14 / 36.70
Scalar history, w=4w=442 / 12336.61 / 37.0836.80 / 36.8137.24 / 37.37
H+LG42 / 12336.41 / 36.4837.49 / 37.3237.30 / 35.98
  • H+LG improves over BaseLoop and LG at both extrapolated depths in both seeds. Over BaseLoop: +0.90 / +0.73 at D=56D=56 and +1.70 / +0.38 at D=84D=84.
  • H+LG improves over its matched H component at D=56D=56 in both seeds (+0.72 / +0.80). At D=84D=84 it stays ahead on the seed average (36.64 vs. 36.42), although the size of the margin varies between seeds.
  • Compared with the strongest history-only variant, H+LG is higher at D=56D=56 in both seeds (+0.69 / +0.51). Scalar history w=4w=4 is higher at D=84D=84 on the seed average (37.31 vs. 36.64) and is the most seed-stable variant (0.13-point spread). The two are therefore complementary choices: H+LG is stronger at 2×2\times the training depth, and scalar history w=4w=4 is more stable at 3×3\times.

Additional architectures. We trained H and H+LG on Llama BaseLoop 4×54\times5 and 5×45\times4 (training depth D=20D=20).

  • On Llama 4×54\times5, H improves Overall over BaseLoop by 0.75 and 1.39 points at D=40D=40 and 6060. H+LG is above BaseLoop at all three depths, with Knowledge gains of +5.86 and +6.82 under extrapolation, the same knowledge-retention pattern as on Qwen.
  • On Llama 5×45\times4, H again retains more knowledge than BaseLoop under extrapolation (+1.31, +3.65 at D=40D=40, 6060). Adding LG raises Reasoning by 1.85 and 1.69 points, and both H and H+LG exceed BaseLoop in Overall at D=60D=60.

The full table is in our response to Reviewer 3FVq, W3(a). All new runs are added to App. E.3 and E.5.

W2 / Q2. Larger-scale validation and dependence on model size

We are extending the H vs. H+LG comparison to a substantially larger model, the largest we can train within the discussion period:

  • Architecture: a BaseLoop 20×520\times5 model in the Llama3.1-3.5B configuration (hidden size 1536, 20 shared layers, 5 training loops, effective training depth D=100D=100).
  • Size: 1.01B unique parameters, nearly 5× the 219M of Qwen 4×74\times7. Its effective depth corresponds to a 3.5B-parameter non-recurrent model.
  • Comparison: H (channel-wise, w=2w=2) and H+LG share the same backbone and training recipe. We evaluate both at D=100D=100, 200200 and 300300 (the training depth, 2×2\times and 3×3\times), with the same benchmarks and aggregation as in the paper.

Training of these models is still running. We will post the results in this thread as soon as they finish, together with a comparison to the smaller-scale trend.

AuthorsOpenReview ↗

Response to Reviewer fexH [update]

Update: large-scale results in W2 / Q2

The larger-scale runs announced in our earlier response have finished: Llama BaseLoop 20×520\times5 with 1.01B unique parameters, training depth D=100D=100 (the depth of a 3.5B-parameter non-recurrent model), and 17.31B training tokens. Overall accuracy, with Knowledge / Reasoning in parentheses:

MethodD=100D=100 (train)D=200D=200 (2×2\times)D=300D=300 (3×3\times)
H38.99 (63.12 / 29.94)38.97 (62.28 / 30.23)38.22 (60.53 / 29.86)
H+LG38.78 (63.00 / 29.70)38.82 (62.47 / 29.95)38.31 (60.94 / 29.82)
  1. History-conditioned recurrence stays robust well beyond the training horizon at this scale.
    • At twice the training depth, Overall is unchanged (H: 38.97 vs. 38.99; H+LG: 38.82 vs. 38.78), and Reasoning rises slightly for both.
    • At three times the training depth, both models keep more than 95% of their training-depth Knowledge (H: 60.53 / 63.12; H+LG: 60.94 / 63.00).
    • For context, at the smaller scale the unconditioned Llama 4×54\times5 BaseLoop already loses 10.69 Knowledge points at 2×2\times depth (62.80 → 52.11).
  2. H+LG performs on par with H at every inference budget. Overall differences are at most 0.21 points, well within evaluation error, and H+LG has slightly higher Knowledge at 3×3\times depth (+0.41).

These results show that the knowledge-preserving extrapolation behavior of history-state conditioning, observed at 0.2–0.5B unique parameters, persists at 1B unique parameters. We will add this baseline, together with the results above, to App. E.5 of the revision.

Dear Reviewer fexH,

We would like to kindly follow up on our rebuttal and ask whether our responses have addressed your concerns. We would greatly appreciate any further feedback or questions you may have.

Thank you for your time and consideration.

Reviewer_hBAWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether additional recurrence benefits looped language models, especially when beyond the recurrence depth used during training. It studies the allocation of shared and non-shared layers, state injection, and timestep conditioning, and proposes combining history-state difference injection with timestep conditioning. Validation uses models trained from scratch with Llama- and Qwen-based configurations, downstream knowledge and reasoning benchmarks, and representation and computational-interaction probes. The results reveal task- and architecture-dependent extrapolation behavior, with the proposed combination improving performance in the evaluated Qwen configuration.

Critical strengths
  1. The paper evaluates recurrence below, at, and beyond the training horizon, revealing capability trade-offs that evaluation at the training depth would miss. The experiment results provide useful evidence about the limitations of recurrent test-time scaling.

  2. The experiments examine recurrent-core depth, boundary-layer allocation, injection parameterization, history-window size, and conditioning combinations. Matching effective training depth and training budgets in relevant comparisons makes these ablations informative about architectural choices.

Critical weaknesses
  1. The paper does not sufficiently establish which findings go beyond existing studies of recurrent depth, boundary-layer allocation, and timestep conditioning. More extensive ablations or longer unrolling horizons do not, by themselves, establish a substantial contribution. Which specific results provide previously unavailable insights or change conclusions from the closest prior work?

  2. The proposed history-state mechanism does not demonstrate consistent standalone benefits. The paper acknowledges limited gains and some reasoning degradation under lightweight parameterizations. The positive results therefore provide insufficient evidence for a broadly effective design principle.

  3. The presentation is poor, and the main text lacks a clear and complete research-paper structure. In particular, it contains neither a dedicated related-work section nor a conclusion. These omissions leave the positioning of the work and the synthesis of its findings insufficiently developed, making the manuscript feel incomplete and its contributions difficult to assess.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer hBAW [part 1/2]

We thank the reviewer for recognizing the value of our systematic study and controlled experiments.

W1. Specific contributions beyond prior studies

Our contribution is to identify how architectural and conditioning choices affect recurrence across inference budgets, all at matched physical depth, training depth and token budget, and to develop history-state difference injection within this analysis. The table lists, for each finding, what the closest prior work established and what our result adds or changes.

QuestionClosest prior workWhat our results add or change
When does extra recurrence help?Looped models favor reasoning over memorization at the trained loop count (Saunshi et al., 2025). Test-time depth shows diminishing returns in aggregate (Parcae; Prairie et al., 2026).Within one checkpoint, extrapolating beyond the training horizon raises reasoning while lowering knowledge, so an aggregate curve can hide gains and losses that cancel (Fig. 1a). Harder instances (longer CLUTRR chains) do not benefit more (Fig. 1c).
Where should non-recurrent layers go?Each design fixes one allocation: no boundary layers (BaseLoop; Saunshi et al., 2025), prelude/coda (Huginn; Geiping et al., 2025), ArBA^rB (RINS; Alabdulmohsin & Zhai, 2025).No allocation is best across budgets. A Coda mitigates under-unrolling, while Prelude-heavy allocations better sustain reasoning extrapolation. At the training horizon, the allocation matters little (Sec. 4).
How should recurrence be conditioned?Concatenation-based initial-state injection (Huginn); timestep conditioning (Xu & Sato, 2025; LoopFormer, Jeddi et al., 2026).Dense initial-state injection collapses on knowledge under extrapolation in our fixed-loop training, and lightweight forms give negligible net gains (Sec. 5.1). Scalar loop/branch gates compete with AdaLN in some configurations (Sec. 5.2).
A new conditioning signal–History-state difference injection. A parameter-matched control that reads h0h_0 instead of intermediate states collapses under extrapolation (29.88 vs. 37.24 Overall at D=84D=84), so the benefit comes from the intermediate states themselves (W2).
Do conditioning mechanisms compose?Typically studied one at a time.They do not compose additively. Initial-state conditioning adds nothing on top of H+LG, and computational-interaction probes give a supporting diagnostic. The H+LG complementarity itself depends on configuration (W2).

Three of these results change practical conclusions drawn from prior work:

  1. Aggregate metrics are not enough. A training-horizon evaluation or an aggregate test-time curve can hide opposing effects on knowledge and reasoning.
  2. Boundary-layer placement depends on the deployment budget. Whether to add boundary layers cannot be decided without knowing the intended inference budget.
  3. The widely used dense initial-state injection is not robust. Under fixed-loop training, it is the least robust option for depth extrapolation.

The new Related Work section (Sec. 7) is organized around this comparison.

W3. Main-text structure, related work, and conclusion

We agree that the main text should both position the work clearly and synthesize its findings. The original submission includes a detailed literature review in Appendix B. However, keeping it entirely in the appendix makes the relationship to the closest studies unnecessarily hard to assess, and the main text also lacked a dedicated conclusion. In the revision we have:

  1. added a Related Work section (Sec. 7) that discusses the closest prior studies in the main text. For each one it compares the research question, training protocol and findings, following the comparison in our response to W1. The extended review remains in Appendix B.
  2. added a Conclusion (Sec. 8) that answers the paper's three questions: when recurrence helps, where it should be applied, and how conditioning affects it, and states the limits of the evidence: the model scale, the fixed training loop count, the single seed for most runs, and the configuration dependence of combined conditioning.

References

  • Alabdulmohsin, I. & Zhai, X. Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systems. NeurIPS 2025.
  • Geiping, J. et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. NeurIPS 2025.
  • Jeddi, A., Ciccone, M. & Taati, B. LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation. ICLR 2026.
  • Prairie, H. et al. Parcae: Scaling Laws for Stable Looped Language Models. arXiv:2604.12946, 2026.
  • Saunshi, N. et al. Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR 2025.
  • Xu, K. & Sato, I. On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding. ICML 2025.
AuthorsOpenReview ↗

Response to Reviewer hBAW [part 2/2]

W2. Inconsistent standalone gains from history conditioning

We agree that the standalone evidence in the submission was limited, and that lightweight history injection on its own trades a small amount of reasoning for knowledge retention. To test the design principle directly, we ran three sets of new experiments: a source-replacement control, a second training seed, and a second model family. Overall accuracy on Qwen 4×74\times7 (training depth D=28D=28):

MethodSeedD=28D=28D=56D=56D=84D=84
BaseLoop4236.8036.5935.60
Scalar initial-state control, 4 lags read h0h_0 (new)4236.4030.9629.88
Scalar history, w=4w=442 / 12336.61 / 37.0836.80 / 36.8137.24 / 37.37
Channel-wise history, w=2w=2 (H)42 / 12336.64 / 36.5436.77 / 36.5236.14 / 36.70
H + Loop Gating (H+LG)42 / 12336.41 / 36.4837.49 / 37.3237.30 / 35.98

Three questions are answered.

(i) Is the benefit due to history, or to extra parameters? It is due to history. The control matches scalar history w=4w=4 in parameter count, lag sharing, initialization and lag schedule. The only difference is that every lag reads h0h_0. It performs about the same at the training depth but falls 7.36 points behind at D=84D=84, mainly on Knowledge (33.46 vs. 59.18).

(ii) Is it reproducible? Yes. Every history-conditioned variant beats BaseLoop at D=84D=84 in both seeds:

  • Scalar history w=4w=4 +1.64 and +1.77. Its two seeds differ by only 0.13 points.
  • Channel-wise H +0.54 and +1.10.
  • H+LG +1.70 and +0.38. H+LG also beats both of its components, H and LG, at D=56D=56 in both seeds (over H: +0.72 and +0.80). At D=84D=84 it stays ahead of H on the seed average (36.64 vs. 36.42), although the size of that margin varies between seeds.

(iii) Does it transfer to another family?

Yes.

On Llama 4×54\times5, H improves Overall over BaseLoop by 0.75 and 1.39 points at D=40D=40 and 6060, and over parameter-matched channel-wise initial-state injection by 1.71 and 2.84 points. H+LG is above BaseLoop at all three depths, and its Knowledge gain under extrapolation reproduces the Qwen pattern: +5.86 and +6.82 at D=40D=40 and 6060.

On Llama 5×45\times4, H again retains more knowledge than BaseLoop under extrapolation (+1.31, +3.65), adding LG raises Reasoning (+1.85, +1.69), and both exceed BaseLoop in Overall at D=60D=60. Full table, please refer to our response to Reviewer 3FVq, W3(a).

Dear Reviewer hBAW,

We would like to kindly follow up on our rebuttal and ask whether our responses have addressed your concerns. We would greatly appreciate any further feedback or questions you may have.

Thank you for your time and consideration.

Reviewer_3FVqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper systemacticly studies looped LM on the recurrence design, including when it helps, where it should apply and how the loop condition contribute to the performance. The experiment reveals that, recurrence helps more on reasoning, and how distinct layers and recurrent iterations matters the overall performance. The author further proposed a new history-state injection as better alternative recurrence conditioning design.

Critical strengths
  • This work systematically studies how recurrence depth and different recurrent configurations affect knowledge and reasoning performance, which provides good guidance for future LoopLM design and scaling experiment.
  • The experiments analysis conditioning design, and propose a new conditioning design which injects hidden-state information, and encodes the recurrent iteration, which shows better improvement.
Critical weaknesses
  • The benefit of history-state injection itself is not sufficiently supported. In Table 10, under same count of parameters, the setup of inital-state does not show clear advantage over history w=1 state. And it is better to run control study of scaled inital state experiment, i.e. scale up the parameter to match the history state with window larger than 1 to see the result.
  • The evaluation of reasoning benchmark does not clear isolate the reasoning performance with knowledge tasks. While benchmarks such as HellaSwag and CommonsenseQA require reasoning, it also involve commonsense, knowledge, which makes the distinguishment less.
  • Experiment setup is not fair.
    • History-state injection is limited to only Qwen3-0.6B BaseLoop 4×7 configuration.
    • The inference budget is not compable. For example, in fig 1d, BaseLoop 4×5 is effective 40 in inference, while NonLoop 20×1 is just 20.
Questions for authors

NA

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3FVq [part 1/3]

Thank you for recognizing the value of our systematic study and for identifying these important issues. We address each concern below.

W1. Benefit of history-state injection and parameter-matched controls

We first clarify that Tab. 10 already contains parameter-matched pairs: scalar initial-state injection and scalar history-state injection with w=1w=1 each add one trainable parameter. Channel-wise and dense initial-state injection match their history-state counterparts with w=2w=2, adding 2d2d and 2d22d^2 parameters, respectively. We agree, however, that the w=1w=1 results alone do not show a consistent advantage from accessing intermediate history. We also agree that matching parameter counts does not isolate the role of the information source.

Following your suggestion, we run a Source-replacement control: scaled up scalar initial-state injection to match scalar history-state injection with w=4w=4 on the same Qwen 4×74\times7 backbone. The control keeps the four scalar lag weights, relative-lag sharing, zero initialization and lag-activation schedule. The only change is that every active lag reads the initial state h0h_0 instead of its intermediate historical state:

h~ℓhist=hℓ+∑j=1mℓβj (hℓ−j−hℓ),h~ℓctrl=hℓ+∑j=1mℓβj (h0−hℓ).\tilde h_\ell^{\mathrm{hist}} = h_\ell + \sum_{j=1}^{m_\ell}\beta_j\,(h_{\ell-j}-h_\ell), \qquad \tilde h_\ell^{\mathrm{ctrl}} = h_\ell + \sum_{j=1}^{m_\ell}\beta_j\,(h_0-h_\ell).

Both variants use the same number of active lags, mℓ≤4m_\ell\le 4. The two models therefore have identical parameter counts, and the control's injection branch never sees an intermediate state.

Qwen 4×74\times7Inj. paramsOverall D=28D{=}28Overall D=56D{=}56Overall D=84D{=}84Know. D=84D{=}84Reas. D=84D{=}84
BaseLoop036.8036.5935.6052.2929.34
Scalar initial-state (Tab. 10)136.1636.3435.6952.1029.53
Scalar initial-state control, 4 lags read h0h_0 (new)436.4030.9629.8833.4628.53
Scalar history, w=4w=4 (Tab. 10, seed 42)436.6136.8037.2459.1829.01
Scalar history, w=4w=4 (new, seed 123)437.0836.8137.3757.7529.72

As shown in the table,

  • At the training depth, the control and history injection perform about the same (36.40 vs. 36.61). The extra lag weights alone do not change in-horizon fit.
  • Under extrapolation, the control falls to 30.96 and 29.88 at D=56D=56 and 8484. That is 5.84 and 7.36 points below history injection, almost entirely because Knowledge drops (33.46 vs. 59.18 at D=84D=84). Reasoning differs by only 0.48 points.
  • Adding initial-state capacity does not help. The 4-parameter control is also well below the original 1-parameter version at D=56D=56 and 8484, in line with the Dense initial-state collapse in Fig. 3.
  • The history-state result reproduces with a second training seed (37.37 vs. 37.24 at D=84D=84).

One plausible explanation is that the history differences hℓ−j−hℓh_{\ell-j}-h_\ell shrink as the trajectory settles, so this branch weakens on its own under extended unrolling. In contrast, h0−hℓh_0-h_\ell stays large and keeps pulling the state back toward the input representation. We have added this control to App. E.3 and revised Sec. 5.1 to separate the evidence for standalone history-state injection from the combination results.

AuthorsOpenReview ↗

Response to Reviewer 3FVq [part 2/3]

W2. Knowledge–reasoning grouping and within-task evaluation

Our grouping separates benchmarks by their predominant demand and it does not treat knowledge and reasoning as mutually exclusive. We agree that HellaSwag and CommonsenseQA involve both commonsense knowledge and inference. This is why the aggregate comparison is complemented by within-task analyses: Fig. 1(b,c) stratifies ProofWriter by proof depth and CLUTRR by relation-chain length.

We now report two further checks (added in App. E.4).

1. Grouping sensitivity. We remove HellaSwag and CommonsenseQA and recompute the reasoning aggregate over the remaining six components, with the same equal weighting (RexclR_{\mathrm{excl}}). The table shows the change from the training depth to twice the training depth:

BaseLoopLlama 2×102\times10Llama 4×54\times5Llama 5×45\times4Llama 10×210\times2Qwen 4×74\times7Qwen 7×47\times4
ΔRorig\Delta R_{\mathrm{orig}}+0.93+2.97+1.35+0.03+0.50−0.63
ΔRexcl\Delta R_{\mathrm{excl}}+1.67+4.39+2.34+0.41+0.67−0.73

Five of the six settings still improve, the same pattern as under the original grouping, and Qwen 7×47\times4 remains the exception. In every improving setting the gain is in fact larger without the two benchmarks. So the improvement is not driven by including them.

2. State-tracking complexity. We separately report BBH Tracking Shuffled Objects with three, five and seven objects. In this task the initial assignments and all swaps are given in the input. Uniform-random accuracies are 33.33%, 20.00% and 14.29%. For example, Llama 2×102\times10 improves on the three-object task from 31.20% at its training depth D=20D=20 to 42.80% at D=60D=60. Results vary across models and object counts, and several scores remain near chance. We therefore do not claim that recurrence helps every task or that gains grow monotonically with complexity.

Together, these analyses support the robustness of the aggregate trend while making its scope clearer.

W3(a). History conditioning evaluated on a single configuration

You are right that the original history-state and combined-conditioning experiments focus on Qwen 4×74\times7. We trained channel-wise history injection (w=2w=2, H) and its combination with loop gating (H+LG) on Llama BaseLoop 4×54\times5 and 5×45\times4. These test transfer to a second model family and to a different allocation of core depth and loop count. Both Llama configurations have training depth D=20D=20 and use the same token budget as the paper's Llama runs. Overall scores (Knowledge / Reasoning in parentheses at D=40D=40):

LlamaMethodD=20D=20D=40D=40D=60D=60
4×54\times5BaseLoop37.8737.11 (52.11 / 31.49)34.49
Channel-wise initial-state38.2136.15 (56.39 / 28.56)33.04
LG37.9937.43 (56.17 / 30.41)34.95
AdaLN38.5837.91 (59.68 / 29.75)36.06
H (new)37.7137.86 (59.16 / 29.88)35.88
H+LG (new)38.1237.20 (57.97 / 29.42)34.85
5×45\times4BaseLoop39.3237.26 (53.64 / 31.12)34.03
Channel-wise initial-state39.3437.35 (53.99 / 31.10)34.51
H (new)38.1636.24 (54.95 / 29.23)34.47
H+LG (new)38.4736.06 (49.34 / 31.08)34.38
  • Standalone history-state injection transfers to both Llama configurations. At D=40D=40 and 6060, H improves Overall over BaseLoop (+0.75, +1.39), over LG (+0.43, +0.93), and over channel-wise initial-state injection with the same parameterization (+1.71, +2.84). The last comparison agrees with the source-replacement result in W1. As on Qwen, the gain comes from knowledge retention (59.16 vs. 52.11 at D=40D=40), while Reasoning is below BaseLoop (29.88 vs. 31.49). On Llama 5×45\times4, H again retains more knowledge than BaseLoop under extrapolation (+1.31, +3.65 at D=40D=40, 6060) and than channel-wise initial-state injection (+0.96, +2.77), and is above BaseLoop in Overall at D=60D=60 (+0.44).
  • H+LG also transfers. On Llama 4×54\times5, H+LG is above BaseLoop at all three depths (+0.25, +0.09, +0.36). Its Knowledge gain under extrapolation (+5.86 and +6.82 at D=40D=40 and 6060) reproduces the knowledge-retention pattern seen on Qwen. On Llama 5×45\times4, adding LG to H raises Reasoning by 1.85 and 1.69 points at D=40D=40 and 6060, matching the reasoning contribution of timestep conditioning in Sec. 5.2, and H+LG exceeds BaseLoop at D=60D=60 (Overall +0.35, Reasoning +0.93). The size of the Overall gain depends on the allocation of core depth and loop count, as in Sec. 4.

We have added these results to App. E.5, and Sec. 5–6 now state which findings hold across the evaluated settings.

AuthorsOpenReview ↗

Response to Reviewer 3FVq [part 3/3]

W3(b). Inference budgets and the NonLoop reference

We agree that the comparison in Fig. 1(d) is not matched in inference compute, and that our original wording ("upper bound under equal compute") was imprecise. NonLoop 20×120\times1 is matched to the Llama BaseLoop models in training effective depth, token budget and training FLOPs, and in inference compute only at D=20D=20. Fig. 1(d) asks a different question: can an already-trained recurrent checkpoint improve by spending more test-time computation? The horizontal line is a single NonLoop evaluation at its native depth, not a set of evaluations at each inference budget.

Both views can be read from the figure:

  • Matched inference compute (D=20D=20). BaseLoop 4×54\times5 scores 28.52 on reasoning vs. 30.25 for NonLoop. At equal inference compute, the non-recurrent model, which has 5× more distinct layers, is better.
  • Additional test-time compute. Running the same BaseLoop 4×54\times5 checkpoint at D=40D=40 raises reasoning to 31.49, above the NonLoop reference, with no change to its learned parameters.

The claim we make is the second one: recurrence turns extra test-time compute into reasoning gains that a fixed-depth model of the same training cost cannot obtain without retraining. We do not claim that BaseLoop beats NonLoop at equal inference FLOPs.

Prior looped-model studies use non-recurrent baselines in the same way. Saunshi et al. (ICLR 2025) distinguish parameter-matched and FLOP-matched comparisons, and Geiping et al. (NeurIPS 2025) report a fixed-depth baseline alongside recurrent evaluations at different iteration counts. In our figure, NonLoop serves specifically as a fixed reference for the inference-scaling trajectory.

In the revision we have:

  1. rewritten the Fig. 1 caption to state that NonLoop is evaluated only at D=20D=20 and is matched in training cost, not in inference cost beyond D=20D=20;
  2. revised Sec. 2.2 to state which quantities each experiment controls: inference effective depth is matched in the architecture and conditioning comparisons, but deliberately varied in the inference-scaling analysis;
  3. replaced the "upper bound under equal compute" wording in Sec. 3.

References

[1] Reasoning with Latent Thoughts: On the Power of Looped Transformers. Saunshi et al. (ICLR 2025).

[2] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. Geiping et al. (NeurIPS 2025)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.