acceptodds
Under review as a conference paper at ICLR 2027

Recurrent Self-Improvement: Dynamic Cross-Loop On-Policy Distillation for Looped Language Models

10 views

Abstract

Looped Language Models (LoopLMs) offer a parameter efficient approach to scaling reasoning by reusing shared parameters across recurrent computation steps. Despite their promise, effective post-training of LoopLMs remains challenging. Existing approaches either provide reward based supervision that is sparse or costly to extend across loops, or rely on external teachers or privileged information, leading to limited teacher availability or teacher-student context mismatch. To address these limitations, we introduce LoopOPD, a cross-loop on-policy distillation framework that uses additional recurrent computation within a LoopLM as its own source of supervision. LoopOPD uses a frozen terminal loop policy as a compute privileged teacher for an intermediate loop student on student generated rollouts, providing dense supervision without an external teacher or privileged information. We further propose Dynamic LoopOPD (D-LoopOPD), which continually refreshes the terminal loop teacher as the shared model parameters are updated, enabling recurrent self-improvement. We characterize how distillation updates propagate across loop depths and derive sufficient conditions under which a single update yields simultaneous local improvement at both loop depths. Experiments on Ouro-Thinking models show that LoopOPD improves mathematical reasoning, while D-LoopOPD yields further gains through dynamic teacher updates. Despite being trained only on mathematical data, the resulting models also improve on general reasoning and code generation benchmarks, demonstrating that recurrent computation can serve as an effective source of supervision for LoopLMs. Our anonymous code repository is available at: https://anonymous.4open.science/r/LoopOPD-8C7F

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Language Models, On-Policy Distillation, Post-Training, Recurrent Computation, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9KfeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes using deeper-loop predictions from a looped language model to supervise a shallower loop, providing dense distillation signals without an external teacher or answer-privileged context. LoopOPD freezes the initial model's terminal-loop policy as the teacher. The student generates rollouts at an intermediate depth and minimizes reverse KL to the teacher on the same token prefixes. D-LoopOPD instead uses the current shared-parameter model's terminal policy as a stop-gradient teacher, so each student update also changes the teacher used subsequently.

Experiments train Ouro-1.4B/2.6B-Thinking on DAPO-Math-17k, generally using loop 2 as the student and loop 4 as the teacher, with main results evaluated at loop 4. For 2.6B, the mean Avg@16 across three AIME editions increases from 55.49 for the vanilla model and 56.60 for GRPO to 59.37 for LoopOPD and 62.36 for D-LoopOPD. The paper also evaluates other mathematics benchmarks, MMLU, code generation, cross-loop performance, and decoding budgets.

Critical strengths
  1. The method exploits recurrence to obtain a compatible internal teacher and achieves meaningful terminal-loop gains. The distillation objective does not require a larger model, answer hints, or verifiable rewards, and teacher and student share the vocabulary and problem context. In Table 2, 2.6B AIME 2024 Avg@16 improves from 62.08 to 71.67, with gains on AIME 2025 and 2026 as well. Appendix Table 5 shows improvements at loops 3 and 4 in addition to the directly supervised loop 2. The 1.4B model exhibits a similar overall trend. These findings suggest a useful direction for LoopLM post-training.

  2. The paper acknowledges both beneficial and harmful effects of parameter sharing and reports depth-dependent and budget-dependent behavior. The counterexample in Appendix A shows that a stronger terminal teacher does not guarantee a favorable terminal update direction. Figure 5 favors a loop-2 student over loops 1 and 3. Appendix C.3 also shows that RLTT substantially outperforms D-LoopOPD at short 2K–8K decoding budgets, while D-LoopOPD performs better at long budgets. These disclosures help define the method's operating regime.

Critical weaknesses
  1. The central comparisons do not adequately control training conditions, including the comparison intended to isolate dynamic teachers. Table 7 lists maximum rollout lengths of 2,048 for RLTT and 1,024 for OPSD, compared with 31,744 for the proposed methods. Optimizers, batches, learning rates, and warmup schedules also differ. The reported memory constraint on long-sequence RLTT supports an implementation-level feasibility advantage, but does not isolate superiority of the objective. Equal evaluation lengths do not remove training-length ceilings differing by approximately 15.5× and 31×. More directly, LoopOPD has no warmup whereas D-LoopOPD uses a 10-step warmup, so their comparison changes more than teacher refresh. Table 6 reports RLTT scores of 23.33/34.16/45.21 at 2K/4K/8K, versus 3.96/16.87/36.45 for D-LoopOPD, emphasizing the importance of budget. This is the main concern affecting the recommendation.

  2. The generality and mechanism of recurrent self-improvement remain insufficiently supported. The theorem assumes positive gradient alignment at both depths; it does not establish why that alignment should persist in real Ouro training. The appendix counterexample makes this limitation explicit. No empirical alignment measurements or EMA/periodic-refresh controls are reported, and higher entropy is correlational evidence. Independent training repetitions and uncertainty from repeated sampling are also missing. Each AIME edition contains only 30 questions, and a 0.61-point HumanEval gain corresponds to one problem. Such small changes alone do not establish robust broad transfer. The larger mathematics gains are promising, but training variation, model selection, and dynamic teacher effects need to be distinguished.

Questions for authors
  1. [Essential control] Does the dynamic teacher advantage persist when LoopOPD and D-LoopOPD use identical warmup, optimizer, learning rate, data order, update count, and rollout settings? Please report actual updates, processed tokens, and GPU hours for all methods, rather than hardware alone.

  2. [Budget comparability] Could RLTT, OPSD, and D-LoopOPD be compared at a common feasible training length, alongside the current resource-constrained long-context comparison? If long-sequence feasibility is a principal benefit, please quantify peak memory and throughput and distinguish objective-level requirements from implementation limitations. Shorter inference-budget evaluations do not replace this training control.

  3. [Mechanism] What fraction of actual updates has positive gradient alignment at both depths in Equation (16)? How do periodic refresh, EMA, or a temporally shuffled teacher-update schedule compare? These measurements would help distinguish the effect of dynamic terminal supervision from general cross-loop consistency regularization; an unconditional theorem is not required.

1 comment

Thank you for the detailed response. I have carefully considered the authors' clarifications and will maintain my score.

Reviewer_8bHJOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LoopOPD and D-LoopOPD for post-training looped language models. A terminal-loop teacher supervises an intermediate-loop student on student-generated trajectories, with the dynamic variant refreshing the teacher through shared parameter updates. Experiments on two Ouro model scales demonstrate improvements in mathematical reasoning at both intermediate and terminal loops.

Critical strengths
  • The approach is simple and well motivated, using additional recurrent computation as supervision without an external teacher or privileged information.

  • Improvements across three AIME benchmarks and two model scales provide encouraging evidence of effectiveness. Terminal-loop gains from intermediate-loop supervision are particularly interesting.

  • Cross-loop evaluation, student-depth ablations, and inference-budget analysis provide useful supporting evidence.

Critical weaknesses
  • Incomplete experimental controls. LoopOPD and D-LoopOPD use different warmup settings, while RLTT and OPSD use substantially shorter training rollouts, complicating attribution of the gains.

  • Missing sampling ablation. Without a teacher-generated-rollout baseline using the same objective, the contribution of student on-policy sampling remains unclear.

  • Deeper-loop stability. Shared parameter updates could improve the supervised loop while degrading subsequent loops. The conditional theoretical guarantees and final-checkpoint gains do not establish stability throughout training.

Questions for authors
  • Does the dynamic variant retain its advantage with identical warmup settings? Can baseline comparisons better account for training-length and compute differences?

  • How does teacher-generated sampling compare with student-generated sampling under the same cross-loop KL objective?

  • Could the authors provide per-loop performance curves throughout training? Have they observed student improvement accompanied by deeper-loop degradation?

Reviewer_Zjs6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LoopOPD, a cross-loop distillation method for looped language models. The basic idea is that later loops perform more computation, so they can supervise earlier loops within the same model.

D-LoopOPD extends this idea by updating the terminal-loop teacher throughout training.

I find the idea appealing. It uses a property that is already present in LoopLMs, and it avoids the need for a larger external teacher.

The paper also addresses several practical issues with existing approaches, such as sparse reward signals, memory overhead, and teacher-student mismatch.

Critical strengths

Later loops have more computation, so letting them teach earlier loops feels like a natural use of the LoopLM architecture.

Previous approaches often rely on sparse rewards, extra memory, or a stronger external teacher. LoopOPD avoids most of these requirements.

The AIME gains are clear, and the cross-loop results suggest that improving an intermediate loop can also benefit the final loop.

Critical weaknesses

The teacher and student share the same parameters. It is hard to tell whether the gains come from the moving teacher or the extra KL loss.

The teacher and student share the same parameters, and the teacher keeps changing during training. It is hard to tell whether the gains come from the moving teacher or simply from adding the KL loss.

This makes the evaluation somewhat narrow. We do not yet know whether it works on other LoopLM architectures or training setups.

The AIME results are strong, but the improvements on MMLU and code generation are much smaller. The broader transfer claim may need to be stated more carefully.

Questions for authors

The shared parameters make the source of the improvement difficult to isolate. A comparison with a fixed or detached teacher would help clarify how much D-LoopOPD benefits from the moving teacher itself.

How sensitive is the method to the choice of LoopLM architecture, the total number of loops, and the selected student depth?

It would be useful to know whether the reported gains remain stable across different random seeds and evaluation settings.

The transfer results are mixed. Could you provide more analysis of why mathematical training helps some reasoning and code benchmarks, but not others?

Do the conclusions remain the same when all methods are compared under the same training compute and wall-clock budget?

Comments on the paper

2 comments

I ran your official code. On AIME24 and AIME25, vanilla already achieved pass@16 scores of 83.33 and 73.33, respectively. Your paper does not report any variance, and reproducing the results with your code showed that you substantially underreported vanilla’s performance.

Looking at Tables 8 and 9 in the original Ouro paper, the vanilla performance reported in your paper is far too low. The baseline in this paper is clearly wrong.

Ultimately, in my reproduction, your method and vanilla achieved the same performance scores and showed the same variance. I am leaving this comment because I trusted your experimental results and ended up wasting a 10 day of B200 GPU*8 time.

To make the baseline discrepancies more concrete, the table below compares the results reported in Table 9 of the original baseline paper [1] with the corresponding Vanilla baseline results in your Table 2. Specifically, Table 9 reports the Ouro-Thinking-R4 models, while your Table 2 lists the Ouro-Thinking models with d=4d=4.

ModelBenchmarkTable 9 in [1]Your Table 2 (Vanilla)
Ouro-1.4BAIME2483.3 (pass@10\mathrm{pass}@10)73.33 (pass@16\mathrm{pass}@16)
Ouro-1.4BAIME2573.3 (pass@10\mathrm{pass}@10)70.00 (pass@16\mathrm{pass}@16)
Ouro-1.4BOlympiadBench71.6 (accuracy)59.40 (accuracy)
Ouro-1.4BBeyondAIME34.0 (accuracy)24.00 (accuracy)
Ouro-2.6BAIME2490.0 (pass@10\mathrm{pass}@10)83.33 (pass@16\mathrm{pass}@16)
Ouro-2.6BAIME2576.7 (pass@10\mathrm{pass}@10)73.33 (pass@16\mathrm{pass}@16)
Ouro-2.6BOlympiadBench76.4 (accuracy)64.30 (accuracy)
Ouro-2.6BBeyondAIME39.0 (accuracy)35.00 (accuracy)

Even at pass@16\mathrm{pass}@16, your AIME24 and AIME25 baseline scores are consistently lower than the pass@10\mathrm{pass}@10 scores reported in [1]. The accuracy scores on OlympiadBench and BeyondAIME are also lower for both models.

I have also reproduced the experiments in Table 2 using your official code. I repeated these experiments ten times, computed the average results, and retained all logs. I am willing to make both the logs and the averaged results publicly available upon request.

Could you clarify the evaluation settings used for these baseline results and explain the discrepancies with [1]?

[1] Scaling Latent Reasoning via Looped Language Models

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.