- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
The authors propose ReMPO, an RL training method for LoopLMs that incorporates depth decisions into the policy. During training, the method uses an additional policy to weight token predictions from different recurrent depths. The authors provide fairly detailed empirical evidence supporting the effectiveness of ReMPO. However, my main concern comes from the broader context of LoopLMs and latent-space reasoning. LoopLMs have not been widely adopted, and the improvements shown here appear relatively incremental. It is therefore difficult to see this work as sufficiently compelling to convince practitioners to switch to LoopLMs. Given that most researchers and practitioners still use standard LLMs, the contribution of RL training specifically for LoopLMs is correspondingly weakened. For this reason, I currently lean toward weak reject, while remaining open to changing my recommendation to accept.
Critical strengths
The empirical evidence is comprehensive, and the paper also provides several in-depth analyses. Given the existing methods and claims in the paper, I believe the empirical evidence is sufficient to support the authors' conclusions.
Critical weaknesses
-
The paper does not provide a sufficiently compelling reason for others to adopt LoopLMs, which weakens the novelty and broader impact of an RL method built specifically for them. In this sense, the work feels somewhat like addressing a problem created by the chosen modeling framework itself. Therefore, I request an additional comparison of ReMPO on LoopLM and GRPO on LLM with the same training budget.
-
The method is relatively straightforward and lacks a particularly striking new finding, such as a counterintuitive phenomenon or a deeper conceptual insight. However, this is not the main reason for my current inclination toward rejection.
Questions for authors
See Weaknesses.
3 comments
I thank the authors for their constructive and thoughtful response. The additional experiments provide a strong argument in favor of using ReMPO, and I have adjusted my score accordingly. However, some of my concerns remain unresolved, and my overall assessment is still around 2.75. My remaining concern is as follows.
My main concern is the substantial performance gap between LoopLM and conventional LLMs. The additional experiments provided by the authors establish a comparison by weakening the LLM baseline to bring it to a comparable performance level with LoopLM. However, Qwen2.5-3B may already be substantially behind more recent and capable LLMs (e.g., Qwen3.5). This raises an important question: Can LoopLM, even with the proposed training approach, eventually catch up with the performance of advanced LLMs?
From a practical perspective, if I need a relatively small model, I might prefer using a Qwen3.5-9B model rather than adopting LoopLM. Therefore, I remain uncertain about the practical advantage and competitiveness of LoopLM compared with simply using a stronger, more recent LLM.
We sincerely thank Reviewer qauM for reading our response carefully and raising the score!
We appreciate your practical perspective and agree that publicly available Ouro checkpoints (1.4B/2.6B, pretrained on 7.7T open-source tokens) still trail newer, larger conventional models such as Qwen3.5-9B in raw pretraining scale and capacity. In our added comparison, we selected Qwen2.5-3B specifically because Qwen2.5 is the native backbone of the Search-R1 benchmark at a comparable parameter scale, allowing us to evaluate the RL algorithm itself without confounding a 3.5x-6.4x larger parameter count and newer proprietary pretraining corpora.
More broadly, as stronger pretraining recipes continue to scale up compact looped backbones (e.g., recent 20T+ token looped models), having a reliable, outcome-only RL method becomes essential so that latent recurrent compute is not bottlenecked at post-training. We will include a concise discussion of this pretraining-versus-RL distinction in our revised manuscript. Thank you again for the constructive discussion!
Matched-Budget Qwen2.5-3B GRPO Comparison & Multi-Hop Gains
We thank Reviewer qauM for noting that our empirical evidence and analyses are comprehensive and support the paper's conclusions. Below we address your primary concern on the practical value of LoopLMs under a matched training budget, followed by ReMPO's conceptual contribution.
1. Matched-Budget Comparison: LoopLM + ReMPO vs. Standard LLM + GRPO
Direct answer: In the identical Search-R1 environment, Ouro-1.4B + ReMPO outperforms the 2.1x larger Qwen2.5-3B-Base + GRPO (0.351 All-7 EM; 128.5 H100 GPU-h over 500 steps) under both compute-matched and step-matched definitions of training budget:
Setup and results: We trained Qwen2.5-3B-Base with GRPO under the exact same Search-R1 setup as our Ouro runs (500 steps, 512 questions and G=5 rollouts/step on NQ + HotpotQA, same retriever, two-search cap, outcome EM reward, and validation rule), measuring 128.5 H100 GPU-h. Since per-step timestamps were not archived for the Ouro-1.4B ReMPO run (222.6 GPU-h / 500 steps), we report pro-rata GPU-hours at steps 200 (≈89.0h) and 300 (≈133.6h), a conservative upper bound as early steps run fewer searches (Figure 2(b)). Tested on all 51,713 questions (Single-Hop: NQ, TriviaQA, PopQA; Multi-Hop: 2Wiki, MuSiQue, Bamboogle):
(Dataset EM for Qwen / ReMPO s200 / s300 / s400: NQ 0.480/0.470/0.478/0.485, HotpotQA 0.344/0.416/0.444/0.452, TriviaQA 0.622/0.623/0.636/0.631, PopQA 0.461/0.442/0.464/0.464, 2Wiki 0.299/0.387/0.411/0.427, MuSiQue 0.084/0.142/0.155/0.177, Bamboogle 0.168/0.392/0.392/0.416.)
Why adopt LoopLM + ReMPO:
2. Conceptual Insight Behind a Simple Design
We appreciate your noting this is secondary to the adoption question. Rather than adding complex modules, ReMPO isolates a clean distinction: supervising intermediate predictions on terminal trajectories (credit assignment, as in RLTT) differs from letting intermediate predictions generate actions during interaction (behavior-policy design).
Under identical binary final-answer rewards with zero search shaping, changing only how recurrent readouts enter the policy yields distinct search behaviors (Figure 2, Section 7): GRPO collapses to zero searches after step 175, Qwen2.5-3B + GRPO and Ouro + RLTT plateau near one search (≈1.00), whereas ReMPO learns stable two-search behavior (1.992 searches, retrieving missing answer aliases in 16% to 29% of two-search multi-hop trajectories). Because all readouts share recurrent parameters θ and pθ,4 stays anchored in qθ, these training gains transfer directly to deterministic R4 evaluation (0.436 with R4-greedy vs. 0.439 with q-greedy, Table 7).