acceptodds
Under review as a conference paper at ICLR 2027

ReMPO: Behavior-Consistent Latent-Depth Policy Optimization for Search Agents

10 views

Abstract

Looped language models (LoopLMs) enable multi-step latent reasoning by repeatedly applying shared computation, offering a promising way to increase reasoning depth without increasing model parameters. However, their recurrent computation also creates a mismatch with standard reinforcement learning, which is defined around the final policy output. Recent work has taken an important first step by extending outcome supervision to intermediate predictions, allowing internal computation to receive learning signals. Yet reinforcement learning (RL) is fundamentally driven by the behavior policy that generates the trajectory: assigning credit to intermediate states does not by itself make them part of the policy that produces actions. We therefore introduce Readout-Marginalized Policy Optimization (ReMPO), which constructs the behavior policy by marginalizing token predictions across recurrent readouts. ReMPO uses the same readout-marginalized policy for both trajectory generation and policy optimization, allowing final-answer rewards to jointly train token prediction and readout selection. A terminal anchor and regularization stabilize this learned policy. Using only final-answer rewards, we train Ouro-1.4B and Ouro-2.6B as search agents for question answering. ReMPO outperforms the state-of-the-art method on all seven datasets at both model sizes; at 1.4B, it improves the average exact match by 11.30 percentage points and held-out multi-hop performance by 17.76 points. It also yields more effective two-search trajectories, with second retrievals recovering answer-bearing evidence missed by the first.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, looped language models, search agents, policy optimization, latent-variable policies

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qauMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The authors propose ReMPO, an RL training method for LoopLMs that incorporates depth decisions into the policy. During training, the method uses an additional policy to weight token predictions from different recurrent depths. The authors provide fairly detailed empirical evidence supporting the effectiveness of ReMPO. However, my main concern comes from the broader context of LoopLMs and latent-space reasoning. LoopLMs have not been widely adopted, and the improvements shown here appear relatively incremental. It is therefore difficult to see this work as sufficiently compelling to convince practitioners to switch to LoopLMs. Given that most researchers and practitioners still use standard LLMs, the contribution of RL training specifically for LoopLMs is correspondingly weakened. For this reason, I currently lean toward weak reject, while remaining open to changing my recommendation to accept.

Critical strengths

The empirical evidence is comprehensive, and the paper also provides several in-depth analyses. Given the existing methods and claims in the paper, I believe the empirical evidence is sufficient to support the authors' conclusions.

Critical weaknesses
  • The paper does not provide a sufficiently compelling reason for others to adopt LoopLMs, which weakens the novelty and broader impact of an RL method built specifically for them. In this sense, the work feels somewhat like addressing a problem created by the chosen modeling framework itself. Therefore, I request an additional comparison of ReMPO on LoopLM and GRPO on LLM with the same training budget.

  • The method is relatively straightforward and lacks a particularly striking new finding, such as a counterintuitive phenomenon or a deeper conceptual insight. However, this is not the main reason for my current inclination toward rejection.

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Matched-Budget Qwen2.5-3B GRPO Comparison & Multi-Hop Gains

We thank Reviewer qauM for noting that our empirical evidence and analyses are comprehensive and support the paper's conclusions. Below we address your primary concern on the practical value of LoopLMs under a matched training budget, followed by ReMPO's conceptual contribution.

1. Matched-Budget Comparison: LoopLM + ReMPO vs. Standard LLM + GRPO

"...request an additional comparison of ReMPO on LoopLM and GRPO on LLM with the same training budget."

Direct answer: In the identical Search-R1 environment, Ouro-1.4B + ReMPO outperforms the 2.1x larger Qwen2.5-3B-Base + GRPO (0.351 All-7 EM; 128.5 H100 GPU-h over 500 steps) under both compute-matched and step-matched definitions of training budget:

  • 31% lower compute (≈89.0 GPU-h, step 200): 0.410 All-7 (+5.9 points) and 0.307 multi-hop (+12.4 points) with <50% parameters.
  • Matched compute (≈133.6 GPU-h, step 300): 0.426 All-7 (+7.5 points) and 0.319 multi-hop (+13.6 points).
  • Matched 500-step rollout budget (≈222.6 GPU-h, step 400): 0.436 All-7 (+8.5 points) and 0.340 multi-hop (+15.6 points) with 3.8x lower parser failures (0.95% vs. 3.60%).

Setup and results: We trained Qwen2.5-3B-Base with GRPO under the exact same Search-R1 setup as our Ouro runs (500 steps, 512 questions and G=5G=5 rollouts/step on NQ + HotpotQA, same retriever, two-search cap, outcome EM reward, and validation rule), measuring 128.5 H100 GPU-h. Since per-step timestamps were not archived for the Ouro-1.4B ReMPO run (222.6 GPU-h / 500 steps), we report pro-rata GPU-hours at steps 200 (≈89.0h) and 300 (≈133.6h), a conservative upper bound as early steps run fewer searches (Figure 2(b)). Tested on all 51,713 questions (Single-Hop: NQ, TriviaQA, PopQA; Multi-Hop: 2Wiki, MuSiQue, Bamboogle):

System (Identical Search-R1 Setup)ParamsH100 GPU-hSingle-HopMulti-HopAll-7SearchesFail (%)
Qwen2.5-3B-Base + GRPO (s500, 500s run)3.0B128.5 (500s)0.5210.1830.3511.003.60
Ouro-1.4B + GRPO (s100; collapses >174)1.4B≈140.1 (500s)0.4920.1610.323--0.15
Ouro-1.4B + RLTT (s400, 500s run)1.4B≈150.6 (500s)0.4850.1620.323--8.52
Ouro-1.4B + ReMPO (s200, < budget)1.4B≈89.0 (pro-rata)0.5120.3070.410--1.28
Ouro-1.4B + ReMPO (s300, ≈matched)1.4B≈133.6 (pro-rata)0.5260.3190.426--1.19
Ouro-1.4B + ReMPO (s400, 500s run)1.4B≈222.6 (500s)0.5270.3400.4361.9920.95

(Dataset EM for Qwen / ReMPO s200 / s300 / s400: NQ 0.480/0.470/0.478/0.485, HotpotQA 0.344/0.416/0.444/0.452, TriviaQA 0.622/0.623/0.636/0.631, PopQA 0.461/0.442/0.464/0.464, 2Wiki 0.299/0.387/0.411/0.427, MuSiQue 0.084/0.142/0.155/0.177, Bamboogle 0.168/0.392/0.392/0.416.)

Why adopt LoopLM + ReMPO:

  1. Not fixing a LoopLM-only flaw: Local Qwen2.5-3B + GRPO is a strong baseline (0.351 All-7, above Search-R1's reported 3B Base/Instruct 0.312/0.336 and matching 7B Base 0.350) and matches ReMPO on single-hop QA (0.521 vs. 0.526/0.527). Yet under outcome-only EM rewards, standard LLM + GRPO also stalls in a single-search optimum (1.00 search), reaching only 0.183 on held-out multi-hop tasks.
  2. Unlocking multi-hop reasoning with <50% parameters and less training compute: With standard GRPO/RLTT (0.323), Ouro-1.4B trails Qwen2.5-3B (0.351). ReMPO breaks this single-search bottleneck, beating the 3.0B model by +12.4 to +15.6 multi-hop points even at 31% lower GPU-hours (s200) and scaling to 0.464 at Ouro-2.6B (Table 1).

2. Conceptual Insight Behind a Simple Design

"The method is relatively straightforward and lacks a particularly striking new finding..."

We appreciate your noting this is secondary to the adoption question. Rather than adding complex modules, ReMPO isolates a clean distinction: supervising intermediate predictions on terminal trajectories (credit assignment, as in RLTT) differs from letting intermediate predictions generate actions during interaction (behavior-policy design).

Under identical binary final-answer rewards with zero search shaping, changing only how recurrent readouts enter the policy yields distinct search behaviors (Figure 2, Section 7): GRPO collapses to zero searches after step 175, Qwen2.5-3B + GRPO and Ouro + RLTT plateau near one search (≈1.00), whereas ReMPO learns stable two-search behavior (1.992 searches, retrieving missing answer aliases in 16% to 29% of two-search multi-hop trajectories). Because all readouts share recurrent parameters θ\theta and pθ,4p_{\theta,4} stays anchored in qθq_\theta, these training gains transfer directly to deterministic R4R_4 evaluation (0.436 with R4R_4-greedy vs. 0.439 with qq-greedy, Table 7).

I thank the authors for their constructive and thoughtful response. The additional experiments provide a strong argument in favor of using ReMPO, and I have adjusted my score accordingly. However, some of my concerns remain unresolved, and my overall assessment is still around 2.75. My remaining concern is as follows.

My main concern is the substantial performance gap between LoopLM and conventional LLMs. The additional experiments provided by the authors establish a comparison by weakening the LLM baseline to bring it to a comparable performance level with LoopLM. However, Qwen2.5-3B may already be substantially behind more recent and capable LLMs (e.g., Qwen3.5). This raises an important question: Can LoopLM, even with the proposed training approach, eventually catch up with the performance of advanced LLMs?

From a practical perspective, if I need a relatively small model, I might prefer using a Qwen3.5-9B model rather than adopting LoopLM. Therefore, I remain uncertain about the practical advantage and competitiveness of LoopLM compared with simply using a stronger, more recent LLM.

We sincerely thank Reviewer qauM for reading our response carefully and raising the score!

We appreciate your practical perspective and agree that publicly available Ouro checkpoints (1.4B/2.6B, pretrained on 7.7T open-source tokens) still trail newer, larger conventional models such as Qwen3.5-9B in raw pretraining scale and capacity. In our added comparison, we selected Qwen2.5-3B specifically because Qwen2.5 is the native backbone of the Search-R1 benchmark at a comparable parameter scale, allowing us to evaluate the RL algorithm itself without confounding a 3.5x-6.4x larger parameter count and newer proprietary pretraining corpora.

More broadly, as stronger pretraining recipes continue to scale up compact looped backbones (e.g., recent 20T+ token looped models), having a reliable, outcome-only RL method becomes essential so that latent recurrent compute is not bottlenecked at post-training. We will include a concise discussion of this pretraining-versus-RL distinction in our revised manuscript. Thank you again for the constructive discussion!

Reviewer_Pd9AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Readout-Marginalized Policy Optimization (ReMPO) for training looped language models as search agents. It combines predictions from multiple recurrent readouts into a unified policy used consistently for trajectory generation and policy optimization, with anchoring and regularization to stabilize training. Experiments on two Ouro model sizes across seven QA benchmarks report improvements over GRPO and RLTT, alongside more effective use of additional retrievals.

Critical strengths
  1. ReMPO provides a coherent formulation that incorporates intermediate recurrent readouts into both trajectory generation and policy optimization through the same marginal policy.
  2. The method achieves substantial improvements across seven QA benchmarks and two model sizes. The additional three-seed results on Ouro-1.4B support the robustness of the main gains.
  3. The analyses of search behavior, second-retrieval evidence, parser failures, and computational costs provide useful insights beyond final-answer accuracy.
Critical weaknesses
  1. ReMPO changes both how training trajectories are generated and how they are optimized. The current comparisons do not establish whether the marginal optimization objective adds benefits beyond changing the rollout policy.
  2. A2 conflates contextual gating with direct gate-policy learning, while A4 tests only one mean-matched constant coefficient, limiting the conclusions from these ablations.
  3. Protocol reliability remains unresolved: RLTT exhibits high parser-failure rates, and ReMPO’s failures increase substantially beyond two retrievals.
  4. The distinction between credit assignment and behavior-policy design is repeatedly explained across several sections, while the local theoretical analysis appears separately in Section 7.1 after the main results. This creates redundancy and makes the methodological argument harder to follow.
Questions for authors
  1. Could the authors compare different behavior-consistent optimization objectives under the same marginal rollout construction to isolate the contribution of marginal-policy optimization?
  2. Could the authors disentangle the factors changed in A2 and evaluate additional constant coefficients for A4?
  3. What causes RLTT’s high parser-failure rates and ReMPO’s increased failures at longer horizons? Would protocol stabilization change the comparison?
  4. Could the local analysis in Section 7.1 be integrated with the method, and repeated explanations of the credit-versus-behavior distinction be removed?
5 comments
AuthorsOpenReview ↗

Response (1/2): Marginal vs. Joint Objective and Gate/Anchor Ablations

We thank Reviewer Pd9A for the rigorous feedback on causal attribution, ablation design, protocol reliability, and paper structure. Below we present our new controlled training runs and trajectory diagnostics under the matched Ouro-1.4B Search-R1 setup (split into two comments for OpenReview's character limit).

1. Isolating Marginal Optimization Under Identical Marginal Rollouts (Q1)

"...compare different behavior-consistent optimization objectives under the same marginal rollout construction..."

Direct answer: Yes. Under the identical ancestral marginal rollout (zt∼αθold(⋅∣st)z_t \sim \alpha_{\theta_{\mathrm{old}}}(\cdot\mid s_t), at∼pθold,zt(⋅∣st)a_t \sim p_{\theta_{\mathrm{old}},z_t}(\cdot\mid s_t)) and regularizers, optimizing the marginal ratio rtmarg=qθ(at∣st)/qθold(at∣st)r_t^{\mathrm{marg}} = q_\theta(a_t\mid s_t)/q_{\theta_{\mathrm{old}}}(a_t\mid s_t) (0.436 All-7 EM, 0.95% fail) outperforms the behavior-consistent sampled joint ratio rtjoint=[αθ(zt∣st)pθ,zt(at∣st)]/[αθold(zt∣st)pθold,zt(at∣st)]r_t^{\mathrm{joint}} = [\alpha_\theta(z_t\mid s_t)p_{\theta,z_t}(a_t\mid s_t)] / [\alpha_{\theta_{\mathrm{old}}}(z_t\mid s_t)p_{\theta_{\mathrm{old}},z_t}(a_t\mid s_t)] (0.404 All-7 EM, 3.19% fail) across all seven benchmarks (+3.22 points All-7; +4.38 points multi-hop).

Method (Ouro-1.4B)RolloutPolicy ObjectiveIDMulti-HopAll-7Fail (%)
GRPO / RLTTTerminal R4R_4Terminal / Uniform Loops0.367 / 0.3790.161 / 0.1620.323 / 0.3230.15 / 8.52
A1: Static Resp.Marginal qoldq_{\mathrm{old}}Marginal fwd / αold\alpha_{\mathrm{old}} bwd0.4600.2810.40314.79
Sampled-Joint GRPO (new)Marginal qoldq_{\mathrm{old}}Clipped Joint rtjointr_t^{\mathrm{joint}}0.4410.2960.4043.19
ReMPO (Full)Marginal qoldq_{\mathrm{old}}Clipped Marginal rtmargr_t^{\mathrm{marg}}0.4680.3400.4360.95

(Sampled-Joint per-dataset EM: NQ 0.472, HotpotQA 0.409, TriviaQA 0.622, PopQA 0.434, 2Wiki 0.365, MuSiQue 0.147, Bamboogle 0.376.)

By Appendix A.1, rtmarg=∑zγold(z∣st,at)rtjoint(z,at∣st)r_t^{\mathrm{marg}} = \sum_z \gamma_{\mathrm{old}}(z\mid s_t,a_t)r_t^{\mathrm{joint}}(z,a_t\mid s_t). Marginal rollouts let Sampled-Joint GRPO learn two-search behavior (1.98 searches) and rise from 0.323 to 0.404 (+8.08 points). Marginalizing over depth before clipping adds +3.22 points All-7 (+4.38 multi-hop) and cuts parser failures by 3.3x (3.19% to 0.95%) by removing single-sample latent variance and updating all readouts compatible with ata_t.

2. Disentangling A2 and Evaluating Additional Constants for A4 (Q2)

"...disentangle the factors changed in A2 and evaluate additional constant coefficients for A4?"

Direct answer: Both new 500-step Ouro-1.4B control sets confirm clear gains from contextual gating and stage-dependent anchoring:

  1. Disentangling A2 (Learnable Global Gate): Replacing A2's fixed prior gfixedg_{\mathrm{fixed}} with a context-free learnable gate gη(z)=softmax⁡(η)g_\eta(z)=\operatorname{softmax}(\eta) initialized at log⁡gfixed\log g_{\mathrm{fixed}} (matching A2's frozen-reference prior with active gate KL):
Gate Variant (Ouro-1.4B)Contextual ggLearned ggIDMulti-HopAll-7Fail (%)
A2: Calibrated Fixed GateNoNo0.4020.2030.3541.04
Learnable Global Gate gηg_\eta (new)NoYes0.4360.2690.3941.24
ReMPO (Full Contextual)YesYes0.4680.3400.4360.95

Co-adapting global depth weights lifts All-7 EM from 0.354 to 0.394 (multi-hop: 0.203 to 0.269), while context conditioning gθ(z∣st)g_\theta(z\mid s_t) lifts it further to 0.436 (0.340 multi-hop) with lower failures (as in Appendix D.3, A2 and gηg_\eta use the fixed initial gate in the frozen reference).

  1. Endpoint Constants for A4: Alongside mean-matched ϵˉ=0.192\bar{\epsilon}=0.192 (A4), we trained constants matching Full's post- and pre-retrieval endpoints (0.15 and 0.25) throughout generation:
Anchor (ϵ0,ϵ1)(\epsilon_0, \epsilon_1)IDMulti-HopAll-7Fail (%)
Constant (0.15, 0.15) (new)0.4000.2120.3581.84
Constant (0.192, 0.192) (A4)0.4050.1910.3512.07
Constant (0.25, 0.25) (new)0.3850.1790.3371.64
Stage-Dependent (0.25, 0.15) (Full)0.4680.3400.4360.95

All three constants cluster at 0.337 to 0.358 All-7 (7.81 to 9.85 points below Full) and lose 12.8 to 16.1 points on multi-hop tasks. Both endpoints stall at 1.00 search (vs. 1.992 for Full): 0.15 everywhere restricts initial query exploration, while 0.25 everywhere perturbs post-retrieval synthesis (0.337). Pairing 0.25 pre-retrieval with 0.15 post-retrieval unlocks two-search behavior and 0.436 All-7 EM.

AuthorsOpenReview ↗

Response (2/2): Parser-Failure Audit and Method Streamlining

3. Parser-Failure Diagnosis and Protocol Stabilization (Q3)

"What causes RLTT's high parser-failure rates and ReMPO's increased failures at longer horizons? Would protocol stabilization change the comparison?"

Direct answer: Auditing all 51,713 evaluation trajectories per checkpoint reveals fundamentally different failure mechanisms in RLTT versus longer-horizon ReMPO, and evaluating all checkpoints under a unified lenient parser leaves ReMPO ahead of RLTT by +11.20 percentage points (0.436 vs. 0.324 All-7 EM).

  1. Distinct causes and horizon locations:

    • RLTT (Bsearch=2B_{\mathrm{search}}=2, 4,406 failures = 8.52%): 100% of failures occur at ≤1\le 1 search (2.56% at 0 searches; 97.44% after 1 search; 0 after 2 searches), with 0.00% carrying length-truncation flags. Instead, 96.78% emit an unclosed <answer> fragment followed by an empty retry, reflecting brittle single-turn tag closure.
    • ReMPO at Bsearch=2B_{\mathrm{search}}=2 vs. Bsearch=3,4B_{\mathrm{search}}=3, 4: At Bsearch=2B_{\mathrm{search}}=2 (493 failures = 0.95%), 0.00% are unclosed <answer> tags and 97.97% are simple generation-length truncations. At Bsearch=3,4B_{\mathrm{search}}=3, 4 (4,591 / 5,434 failures = 8.88% / 10.51%), early turns remain clean while 94.88% (B=3B=3) and 95.01% (B=4B=4) of failures occur only after ≥2\ge 2 completed retrievals (B=3B=3 also logs 3.53% separate turn-budget exhaustion). Unlike RLTT, only 13.5% (B=3B=3) and 35.9% (B=4B=4) leave unclosed <answer> fragments; most remaining failures terminate immediately after a retrieved </information> block with empty output or short tag/passage remnants and almost no length truncation (0.15% of B=4B=4 failures; 0.02% of all trajectories), indicating post-observation generation stalling in long multi-passage contexts rather than missing closing tags on completed answers.
  2. Method comparison under optimistic bounds and unified lenient parsing: First, GRPO already has a 0.15% failure rate yet scores 0.323 All-7 EM, and even a dataset-weighted optimistic upper bound (MacroEM+17∑dFd/Nd\mathrm{MacroEM} + \frac{1}{7}\sum_d F_d/N_d, counting all failed trajectories with EM=0 as 100% correct) caps RLTT at 0.428 (42.78%) and GRPO at 0.324 (32.40%), both still below strict ReMPO (0.436 / 43.59%). Second, because injecting untrained test-time prompts shifts the rollout distribution and degrades valid trajectories, we evaluated all checkpoints with a unified zero-regeneration lenient parser that deterministically salvages unclosed answer fragments:

Checkpoint (51,713 Questions)Strict All-7 EMLenient All-7 EMSalvaged Non-Empty / Newly CorrectUnparsed Rate (Strict →\to Lenient)
GRPO (Bsearch=2B_{\mathrm{search}}=2)0.323 (32.27%)0.323 (32.28%)73 / +100.15% →\to 0.01%
RLTT (Bsearch=2B_{\mathrm{search}}=2)0.323 (32.29%)0.324 (32.39%)4,159 / +778.52% →\to 0.48%
ReMPO (Bsearch=2B_{\mathrm{search}}=2)0.436 (43.59%)0.436 (43.59%)1 / +00.95% →\to 0.95%
ReMPO (Bsearch=3B_{\mathrm{search}}=3)0.405 (40.51%)0.405 (40.54%)186 / +178.88% →\to 8.52%
ReMPO (Bsearch=4B_{\mathrm{search}}=4)0.434 (43.38%)0.436 (43.56%)599 / +4510.51% →\to 9.35%

Lenient parsing recovers non-empty answers on 94.4% (4,159 / 4,406) of RLTT's failed trajectories (cutting its unparsed rate to 0.48%), yet only 77 of those 4,159 answers are correct, raising RLTT by just +0.10 points (32.29% →\to 32.39%). Meanwhile, Bsearch=4B_{\mathrm{search}}=4 rises slightly to 0.436 (matching Bsearch=2B_{\mathrm{search}}=2), while resolving the remaining ≥2\ge 2-search empty turns requires training-time long-horizon stabilization rather than post-hoc parsing. Thus, protocol issues do not alter the core comparison between ReMPO, RLTT, and GRPO.

4. Integrating Section 7.1 into Method and Removing Redundancy (Q4)

"Could the local analysis in Section 7.1 be integrated with the method, and repeated explanations of the credit-versus-behavior distinction be removed?"

Direct answer: Thank you for this suggestion; we agree it makes the methodological flow cleaner. We will move the local gate-value analysis (fixed-state assumptions, Eq. (13) ∂Js/∂ξz=ϵgz(Vz−Vg)\partial \mathcal{J}_s/\partial \xi_z = \epsilon g_z(V_z - V_g), and Appendix H pointer) from Section 7.1 into Section 4.5 (Local Analysis of Depth Adaptation), and trim repeated credit-vs.-behavior comparisons in Sections 3.2, 4.1, 4.2, and 6.1 so Section 7 focuses directly on empirical ablations and trajectory analyses.

AuthorsOpenReview ↗

Revised Manuscript Uploaded

Dear Reviewer Pd9A,

We wanted to briefly follow up to let you know that we have uploaded the revised manuscript (with all updates highlighted in blue) incorporating all new experiments, analyses, and structural revisions discussed above:

  • Sampled-Joint vs. Marginal objective (Q1), context-free learnable global gate gηg_\eta (Q2a), and endpoint constant anchors (Q2b): summarized in Section 7.1 (pages 7-8) and detailed in Appendix D.3, Table 9 (page 19).
  • Trajectory-level parser-failure audit and unified lenient-parser verification (Q3): summarized in Section 7.1 (page 8) and detailed in Appendix E (pages 20-21) and Table 12 (page 21).
  • Relocated local depth-adaptation analysis and streamlined exposition (Q4): relocated to Section 4.5 (page 5), trimmed across Sections 3.2, 4.1, 4.2, and 6.1, with training-vs.-evaluation readout consistency clarified in Section 5 (page 6).

Please let us know if any further details or analyses would be helpful during the discussion period—we would be glad to discuss!

Reviewer_F1XGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates RL for looped language models as search agents, focusing on the credit assignment across intermediate recurrent depths and incorporating those depths directly into the policy. Standard GRPO relies solely on the final output and to fix this, the authors propose ReMPO to achieve behavior-consistent training across the looping depths, which constructs the policy by marginalizing token predictions across all readouts. ReMPO treats recurrent depth as a latent variable along with a learned depth gate, and therefore turning looped LM into a single unified policy. This policy formulation distributes policy gradients across depths and is further stabilized with both gate KL and policy KL regularization. The authors evaluated on Ouro-1.4B and Ouro-2.6B on open-domain QA benchmarks with a setup similar to Search R1, where ReMPO consistently outperforms both GRPO and RLTT across all datasets.

Critical strengths
  1. The paper is well-motivated with credit assignment in looped LMs. Unlike GRPO or RLTT, ReMPO incorporates intermediate readouts along with the depth gate to address the bottleneck within early depths.

  2. ReMPO achieves consistent improvements over GRPO and RLTT in QA benchmarks. In addition, the authors demonstrated that these accuracy gains can be attributed to healthier training dynamics, more effective second-hop retrievals and the provided comprehensive component ablations.

Critical weaknesses
  1. ReMPO relies specifically on Ouro-like looped models that expose aligned vocabulary readouts at every recurrent step, making it inapplicable to block-wise looped Transformers or continuous latent-reasoning models.

  2. Because the terminal anchor forces the final loop to retain at least 75%, early readouts only have low weights during training. This suggests that early depths have a relatively minor direct contribution and mainly serve as a training-time exploration.

  3. Despite emphasizing behavior consistency, the main evaluation discards the learned gate and early readouts to decode deterministically from last loop. Although in appendix the authors showed that marginal q-greedy decoding performs similarly, this also further that the multi-depth gate and intermediate readouts offer negligible direct benefit in inference.

Questions for authors
  1. Why does the proposed method struggle on certain static math tasks (eg. table 18), and how does ReMPO perform there under matched terminal / gated​ decoding alongside a matched GRPO baseline?
4 comments
AuthorsOpenReview ↗

Response (1/2): Aligned Static-Math Evaluation and Decoding Analysis

We thank Reviewer F1XG for the thoughtful review and for highlighting our clear motivation, healthier training dynamics, effective second-hop retrievals, and component ablations. Across Part 1 and Part 2, we address all four points raised in your review: Question 1 on static math below, followed by Weaknesses 2, 3, and 1 in Part 2.

1. Static Mathematics Benchmarks: Matched Decoding and Baselines (Question 1)

"Why does the proposed method struggle on certain static math tasks (eg. table 18), and how does ReMPO perform there under matched terminal / gated decoding alongside a matched GRPO baseline?"

Direct answer: Aligning GSM8K's completion limit from 512 to 3072 tokens raises ReMPO's GSM8K accuracy from 0.859 to 0.911 (1,201/1,319) under marginal qq-greedy and 0.904 under terminal R4R_4-greedy. Across all eight benchmarks under matched 3072-token evaluation, ReMPO-qq (0.466 5-math; 0.606 8-task) and ReMPO-R4R_4 (0.464 5-math; 0.604 8-task) perform nearly identically, outperforming Raw Ouro (0.396 5-math; 0.526 8-task) and the matched GRPO baseline (0.417 5-math; 0.540 8-task) while slightly exceeding RLTT (0.460 5-math; 0.578 8-task) on static math and gaining >18 points on MBPP.

Why Table 18 appeared mixed and what aligned evaluation shows: Table 18 evaluates a mathematics-trained Ouro-1.4B-Thinking checkpoint (ϵ=0.25\epsilon=0.25, no retrieval stages) trained on the same MATH data as RLTT. Including Raw Ouro and GRPO on the same backbone alongside RLTT (whose GSM8K scores for GRPO and RLTT we also locally reproduced and verified) and evaluating the identical ReMPO checkpoint under both qq-greedy (gated mixture) and terminal R4R_4-greedy cleanly separates length-cap effects from decoding effects:

Ouro-1.4B-ThinkingDecodeMATH500GSM8KAIME24BeyondAIMEAIME265-MathARC-CMMLU-STMBPP8-Avg.
Raw Ouro (RLTT paper)R4R_40.7320.9070.1670.0400.1330.3960.9070.7840.5360.526
+ GRPO (RLTT paper)R4R_40.7740.9170.1670.0600.1670.4170.9070.8110.5160.540
+ RLTT (Table 18)R4R_40.8120.903$^\dagger$0.2670.1200.2000.4600.9190.8380.5670.578
+ ReMPO (Table 18)qq0.8220.859$^\dagger$0.3330.1300.1330.4560.9200.8450.7540.600
+ ReMPO (same ckpt)qq0.8220.9110.3330.1300.1330.4660.9200.8450.7540.606
+ ReMPO (same ckpt)R4R_40.8200.9040.3330.1300.1330.4640.9190.8440.7510.604

(†^\daggerEvaluated under RLTT's 512-token GSM8K completion cap; the two bottom rows use the 3072-token completion limit matching GRPO on the identical ReMPO checkpoint.)

Analysis of task pattern:

  1. GSM8K 512- vs. 3072-token cap and N=30N=30 problem count on AIME: In RLTT's evaluation setup (Section 5.2), GSM8K was capped at 512 completion tokens whereas GRPO used 3072 tokens (which also caused RLTT at 0.903 to fall below Raw Ouro at 0.907 and GRPO at 0.917, as we reproduced and verified). Restoring the 3072-token limit on the same ReMPO checkpoint recovers GSM8K from 0.859 to 0.911 (qq-greedy) and 0.904 (R4R_4-greedy). Meanwhile, AIME24 and AIME26 contain only N=30N=30 problems each, where 6.7 points equals 2 problems (+2 over RLTT on AIME24: 10/30 vs. 8/30; -2 on AIME26: 4/30 vs. 6/30).
  2. Aligned decoding (R4R_4 vs. qq) and comparison to GRPO: Under identical 3072-token settings, R4R_4-greedy (0.464 5-math; 0.604 8-task) and qq-greedy (0.466 5-math; 0.606 8-task) match within 0.0 to 0.7 points on every benchmark (consistent with Table 7 on search), while doubling GRPO on harder competition math (AIME24: 0.333 vs. 0.167; BeyondAIME: 0.130 vs. 0.060).
  3. Why ReMPO shines most in interactive search: In single-turn static math, intermediate token proposals do not trigger external retrievals that reshape subsequent observations, so multi-readout credit assignment (RLTT: 0.460) already captures most static-math gains, whereas ReMPO's behavior-policy formulation yields its +11.30-point advantage in multi-turn search.
AuthorsOpenReview ↗

Response (2/2): Terminal Anchoring, Evaluation Readout, and Scope

2. Role of Early Readouts Under the 75% Terminal Anchor (Weakness 2)

"Because the terminal anchor forces the final loop to retain at least 75%, early readouts only have low weights during training. This suggests that early depths have a relatively minor direct contribution and mainly serve as a training-time exploration."

Direct answer: We agree that the terminal readout retains most prior mass and that training-time exploration is a central role of intermediate readouts; however, bounded average prior mass does not imply minor token-level impact or attribution during learning.

Setting (ϵ0,ϵ1)=(0.25,0.15)(\epsilon_0, \epsilon_1) = (0.25, 0.15) caps the non-terminal prior mass at 25% pre-retrieval and 15% post-retrieval so rollouts remain protocol-valid; removing this anchor entirely in A3 drops All-7 EM from 0.436 to 0.410 and raises parser failures from 0.95% to 16.67%. Yet for any action token where an early readout assigns much higher likelihood than the terminal readout, a small mixture weight still substantially shifts both sampling probability and posterior responsibility γt(z)=αθ(z∣st)pθ,z(at∣st)/qθ(at∣st)\gamma_t(z) = \alpha_\theta(z\mid s_t)p_{\theta,z}(a_t\mid s_t) / q_\theta(a_t\mid s_t). For example, under q=0.9p4+0.1pearlyq = 0.9p_4 + 0.1p_{\mathrm{early}}, a query token with p4(a)=0.001p_4(a)=0.001 and pearly(a)=0.10p_{\mathrm{early}}(a)=0.10 becomes 10.9x more likely to be sampled and receives γearly≈0.917\gamma_{\mathrm{early}}\approx 0.917. Moreover, under the identical anchored rollout, replacing exact γt\gamma_t with the rollout prior αold\alpha_{\mathrm{old}} (A1) drops All-7 EM from 0.436 to 0.403 (14.79% fail), and optimizing a sampled-joint ratio instead of the marginal ratio similarly drops All-7 EM to 0.404 (3.19% fail; see also Figure 3(a) replay attribution of 15.1% pre-retrieval and 7.5% to 8.0% post-retrieval), confirming that token-conditioned marginal attribution matters beyond rollout exploration alone.

3. Training Behavior Consistency vs. Terminal-R4R_4 Evaluation (Weakness 3)

"Despite emphasizing behavior consistency, the main evaluation discards the learned gate and early readouts to decode deterministically from last loop. Although in appendix the authors showed that marginal q-greedy decoding performs similarly, this also further that the multi-depth gate and intermediate readouts offer negligible direct benefit in inference."

Direct answer: We agree that ReMPO's primary benefit arises during RL training rather than from test-time multi-depth ensembling, which is why Table 1 and Table 7 answer complementary questions.

Behavior consistency in ReMPO refers specifically to training: trajectories are sampled from qθoldq_{\theta_{\mathrm{old}}}, and the proximal ratio qθ/qθoldq_\theta/q_{\theta_{\mathrm{old}}}, reference KL, and γt\gamma_t all use that exact marginal construction. At test time, Table 1 evaluates all trained models under one shared deterministic R4R_4 rule (executing all four loops) so accuracy differences reflect learned model weights rather than differing decoders, whereas Table 7 (Appendix D.2) compares R4R_4-greedy (0.436) and qq-greedy (0.439) on the same ReMPO checkpoint. Because all readouts share the recurrent parameters θ\theta and pθ,4p_{\theta,4} remains anchored in qθq_\theta, multi-depth training directly updates the terminal policy itself, retaining the +11.30-point training gain under standard single-readout R4R_4 evaluation without claiming extra test-time gating gains or inference speedup (Section 5).

4. Architectural Scope Beyond Ouro (Weakness 1)

"ReMPO relies specifically on Ouro-like looped models that expose aligned vocabulary readouts at every recurrent step, making it inapplicable to block-wise looped Transformers or continuous latent-reasoning models."

Direct answer: ReMPO requires one specific structural interface: at a given token position, multiple recurrent steps expose next-token distributions over a shared vocabulary, combined via normalized depth weights.

  • Block-wise looped Transformers that pair a recurrent core with a shared output module (e.g., Huginn's prelude-core-coda design) can expose shared-vocabulary distributions pθ,z(⋅∣st)p_{\theta,z}(\cdot\mid s_t) across recurrent iterations in principle, though adapting gating/caching and verifying empirical gains beyond Ouro remain future work.
  • Continuous latent-reasoning models that pass hidden states across sequence steps without token-aligned readouts (e.g., Coconut) indeed require a different formulation and are outside ReMPO's direct scope.

We will state this shared-vocabulary readout requirement explicitly in Section 3.2 and Section 8 (Limitations), keeping our empirical claims clearly bounded to Ouro.

AuthorsOpenReview ↗

Revised Manuscript Uploaded (Summary of Updated Sections & Tables)

Dear Reviewer F1XG,

We wanted to briefly follow up to let you know that we have uploaded the revised manuscript (with all updates highlighted in blue) incorporating all experimental additions and clarifications discussed above:

  • Matched GSM8K (3072-token) and terminal-R4R_4 vs. marginal-qq static math evaluation (Q1): updated in Appendix I, Table 21 (page 29).
  • Training-time behavior consistency vs. fixed terminal-R4R_4 evaluation and streamlined exposition (W3): clarified in Section 5 (page 6), with local depth-adaptation analysis relocated to Section 4.5 (page 5) and repeated comparisons trimmed across Sections 3.2, 4.1, 4.2, and 6.1.
  • Architectural scope for shared-vocabulary LoopLMs vs. continuous-latent models (W1): explicitly stated in Section 3.2 (page 3) and Section 8 Limitations (pages 9-10).

Please let us know if you have any further questions during the discussion period—we would be very happy to discuss!

Area_Chair_mpaZOpenReview ↗

ping

ping

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.