- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes using deeper-loop predictions from a looped language model to supervise a shallower loop, providing dense distillation signals without an external teacher or answer-privileged context. LoopOPD freezes the initial model's terminal-loop policy as the teacher. The student generates rollouts at an intermediate depth and minimizes reverse KL to the teacher on the same token prefixes. D-LoopOPD instead uses the current shared-parameter model's terminal policy as a stop-gradient teacher, so each student update also changes the teacher used subsequently.
Experiments train Ouro-1.4B/2.6B-Thinking on DAPO-Math-17k, generally using loop 2 as the student and loop 4 as the teacher, with main results evaluated at loop 4. For 2.6B, the mean Avg@16 across three AIME editions increases from 55.49 for the vanilla model and 56.60 for GRPO to 59.37 for LoopOPD and 62.36 for D-LoopOPD. The paper also evaluates other mathematics benchmarks, MMLU, code generation, cross-loop performance, and decoding budgets.
Critical strengths
-
The method exploits recurrence to obtain a compatible internal teacher and achieves meaningful terminal-loop gains. The distillation objective does not require a larger model, answer hints, or verifiable rewards, and teacher and student share the vocabulary and problem context. In Table 2, 2.6B AIME 2024 Avg@16 improves from 62.08 to 71.67, with gains on AIME 2025 and 2026 as well. Appendix Table 5 shows improvements at loops 3 and 4 in addition to the directly supervised loop 2. The 1.4B model exhibits a similar overall trend. These findings suggest a useful direction for LoopLM post-training.
-
The paper acknowledges both beneficial and harmful effects of parameter sharing and reports depth-dependent and budget-dependent behavior. The counterexample in Appendix A shows that a stronger terminal teacher does not guarantee a favorable terminal update direction. Figure 5 favors a loop-2 student over loops 1 and 3. Appendix C.3 also shows that RLTT substantially outperforms D-LoopOPD at short 2K–8K decoding budgets, while D-LoopOPD performs better at long budgets. These disclosures help define the method's operating regime.
Critical weaknesses
-
The central comparisons do not adequately control training conditions, including the comparison intended to isolate dynamic teachers. Table 7 lists maximum rollout lengths of 2,048 for RLTT and 1,024 for OPSD, compared with 31,744 for the proposed methods. Optimizers, batches, learning rates, and warmup schedules also differ. The reported memory constraint on long-sequence RLTT supports an implementation-level feasibility advantage, but does not isolate superiority of the objective. Equal evaluation lengths do not remove training-length ceilings differing by approximately 15.5× and 31×. More directly, LoopOPD has no warmup whereas D-LoopOPD uses a 10-step warmup, so their comparison changes more than teacher refresh. Table 6 reports RLTT scores of 23.33/34.16/45.21 at 2K/4K/8K, versus 3.96/16.87/36.45 for D-LoopOPD, emphasizing the importance of budget. This is the main concern affecting the recommendation.
-
The generality and mechanism of recurrent self-improvement remain insufficiently supported. The theorem assumes positive gradient alignment at both depths; it does not establish why that alignment should persist in real Ouro training. The appendix counterexample makes this limitation explicit. No empirical alignment measurements or EMA/periodic-refresh controls are reported, and higher entropy is correlational evidence. Independent training repetitions and uncertainty from repeated sampling are also missing. Each AIME edition contains only 30 questions, and a 0.61-point HumanEval gain corresponds to one problem. Such small changes alone do not establish robust broad transfer. The larger mathematics gains are promising, but training variation, model selection, and dynamic teacher effects need to be distinguished.
Questions for authors
-
[Essential control] Does the dynamic teacher advantage persist when LoopOPD and D-LoopOPD use identical warmup, optimizer, learning rate, data order, update count, and rollout settings? Please report actual updates, processed tokens, and GPU hours for all methods, rather than hardware alone.
-
[Budget comparability] Could RLTT, OPSD, and D-LoopOPD be compared at a common feasible training length, alongside the current resource-constrained long-context comparison? If long-sequence feasibility is a principal benefit, please quantify peak memory and throughput and distinguish objective-level requirements from implementation limitations. Shorter inference-budget evaluations do not replace this training control.
-
[Mechanism] What fraction of actual updates has positive gradient alignment at both depths in Equation (16)? How do periodic refresh, EMA, or a temporally shuffled teacher-update schedule compare? These measurements would help distinguish the effect of dynamic terminal supervision from general cross-loop consistency regularization; an unconditional theorem is not required.
Thank you for the detailed response. I have carefully considered the authors' clarifications and will maintain my score.