acceptodds
Under review as a conference paper at ICLR 2027

RMPO: Replay-Measured Turn-Level Credit for LLM Agent Training

Abstract

Multi-turn agent training often relies on sparse episode rewards, making it hard to determine which actions contributed to the final outcome. Standard solutions either infer credit from trajectory statistics or rely on compute-heavy rollout sampling. In this paper, we show that credit can be measured more directly: by restoring the execution state in the environment, applying a localized change to the action, and evaluating the difference in outcome. We formulate this procedure as a common replay interface and propose *Replay-Measured Policy Optimization* (RMPO), a critic-free method that adds normalized replay measurements to the episode-level advantage. Depending on what the environment supports, RMPO implements three pragmatic credit operators: leave-one-out action deletion, prefix-progress scoring, and same-state action replacement. We evaluate RMPO in ALFWorld, WebShop, and Search-QA with Qwen3-4B and Qwen3-8B. It consistently outperforms GRPO and GiGPO. In ALFWorld, deletion measurements correlate strongly with symbolic-planner progress (), and RMPO exceeds four-continuation sampling at equal GPU-hours (90.5% versus 85.5%). The method requires a resettable replay environment. Code is available at https://anonymous.4open.science/r/rmpo-CD48.

Primary area:
reinforcement learning
Keywords:
credit assignment, llm agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uA82OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RMPO, which constructs three types of turn-level signals—action deletion, prefix progress, and action replacement—through environment replay, normalizes them, and adds them to the episode-level advantage for GRPO-style policy optimization. The authors evaluate RMPO on ALFWorld, WebShop, and Search-QA with Qwen3-4B/8B, and report improvements over GRPO, GiGPO, and other baselines. The engineering implementation is relatively complete, but the methodological contribution, theoretical properties, and experimental persuasiveness are insufficient to support acceptance.

Critical strengths

1、 The paper implements replay-based credit assignment across three multi-turn interaction environments, covering two model scales and multiple baselines.

2、 It provides analyses of replay coverage, execution cost, and partial ablations, with relatively detailed experimental protocols and implementation details in the appendix.

3、 Delayed rewards and turn-level credit assignment in multi-turn LLM agents are important problems, and the paper’s engineering exploration of this problem has some value.

Critical weaknesses

1、 Limited novelty.

Action deletion is essentially an executable version of leave-one-out / difference reward, prefix progress resembles dense progress shaping, and action replacement resembles counterfactual action evaluation. The authors themselves acknowledge that the deletion primitive is not new. Overall, the work feels more like an engineering combination of existing credit assignment ideas for LLM agent training, rather than a new learning objective or theoretical mechanism.

2、 The method relies on strong environmental assumptions and external evaluators, and its theoretical properties are biased.

RMPO requires resettable environments, reproducible states, fixed evaluators, executable suffixes, and candidate action sets, which significantly limits its applicability. The so-called “critic-free” approach actually replaces the value critic with environment evaluators, fixed readers, or handcrafted completions, whose biases and costs directly enter the training signal. The authors also acknowledge in the appendix that the normalized deletion correction is generally not potential-based, that RMPO is a biased shaping objective, and that it need not preserve the optimal policy of the original reward.

3、 Experimental fairness and statistical significance are insufficient.

Some gains are small, and the paper does not report paired significance tests or confidence intervals. The base results use a single seed and thus have limited reference value. WebShop replay accounts for about 33% of per-iteration time, but compute-matched comparisons are only provided for ALFWorld; WebShop and Search-QA lack comparisons under equal GPU-hours or equal evaluator-call budgets.

4、 Ablations and audits are insufficient to isolate the core contribution.

Different environments use different replay rules, making it difficult to separate the independent contributions of each signal type. In ALFWorld, the trajectory-mean control already recovers about 38% of the gain, indicating that trajectory-level length or aggregation effects contribute substantially. The high correlation between deletion and symbolic planner progress also partly stems from their shared use of hindsight information, and does not demonstrate that the signal has ex-ante decision value for the acting policy.

Questions for authors

1、Compared with difference reward, COMA, leave-one-out, and counterfactual credit assignment, what is the core algorithmic or theoretical contribution of this paper beyond “executable replay”?

2、Is RMPO still applicable in real-world environments that are not resettable, whose stochasticity cannot be controlled, or that lack a fixed evaluator?

3、Can the authors provide significance tests, confidence intervals, and compute-matched comparisons on WebShop and Search-QA for the main results?

4、Does the absence of deletion credit for failed trajectories lead to reduced exploration or overfitting to successful trajectories? How can the authors demonstrate that the gains come from the turn-level replay signal itself, rather than normalization or length effects?

1 comment

Dear Reviewer uA82,

Thank you again for your detailed review. As the discussion period ends on Nov 18 (AoE), we wanted to check whether our response addressed your main concerns:

  1. Significance: with paired seeds, all seven 8B comparisons in Table 1 have 95% CIs that exclude zero. Under equal wall-clock budgets, WebShop and Search-QA margins are smaller and include zero, and we will report both settings.

  2. Length vs. turn placement: a residual-only control that keeps turn placement but removes the per-trajectory total recovers about 87% of RMPO’s gain over GRPO; the trajectory-mean control recovers about 38% (T5).

  3. Decision-time value: an observation-masking audit (T1) shows where deletion misses information value, and on held-out exploration tasks RMPO still beats PGPO by +4.64 [3.12, 6.16] (T2).

  4. Scope: we will state the resettable-environment requirement in the abstract, replace “critic-free” with “without a learned value critic”, and move the biased-shaping statement into the main text.

We would appreciate it if you could let us know whether any concerns remain. We are happy to clarify before the deadline.

Reviewer_EnXLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

RMPO improves multi-turn LLM agentic training by replaying interactions to estimate how single actions can contribute to the task success. The proposed method uses action deletion, progress measurements, or comparisons with other alternative actions to refine GRPO’s training signal. The results on Qwen3 for ALFWorld, WebShop, and Search-QA show overall good and consistent improvements over baselines.

Critical strengths

Thank you so much for submitting your paper. Overall, the concept is clear to me and well explained. Below are my main strengths in no particular order.

  • Great idea and simplicity: I specifically like the simplicity of the idea and I think that it is broadly applicable to agentic LLM training. When an environment can be replayed (which is not always the case, but very often so), launching controlled alternatives can provide useful evidence about an action’s contribution. RMPO takes this information and uses it with an additive advantage correction without requiring a learned critic (which is very good!). You have many individual ideas here, particularly action deletion and progress shaping, which are not really new, but their joint integration for LLM agents is useful. I also appreciate that you carefully distinguish hindsight relevance from an unbiased estimate of action quality. The theory and practice (your results) seem to make a lot of sense here.
  • Strong experiments: you validate RMPO in a really extensive setup with six seeds. This is more than enough and makes your method trustable and broadly applicable. Also, the ablations you run in both the main paper and appendix are really useful. While I could not take the time to have a look at every one, they support your overall narrative and show that gains come from the joint execution of your approaches. This is important because some reviewers might attack progress scores here, which I do not.

There are also other very-welcomed points about the paper that I would like to mention:

  • Presentation: your figures are high-quality and explain the concept very well. I kept coming back to Figs. 1 and 2.
  • Math details: appreciated in case someone wants to dig deeper.
Critical weaknesses

Despite the aforementioned strengths, there are some weaknesses that need to be addressed.

  • Deletion can penalize exploration?: I am not sure if I understood correctly, but to me it looks like deleting an action while keeping the recorded actions fixed does not really say that the agent could have chosen those actions without the deleted observation. Please correct me here if I am wrong. This comes from looking at Fig. 7 where unsuccessful searches receive negative corrections even though checking those locations was OK under partial observability? My point here is that RMPO may discourage information gathering that is useful for decision-making but appears unnecessary in hindsight. You acknowledge this limitation as well but additional experiments specifically testing such information-dependent decisions or extended commentary would help here.
  • Comparisons might be a bit weak: your main comparisons use matched rollout budgets, but RMPO also performs replay and evaluation. For WebShop, replay accounts for 33% of total iteration time, while the 8B improvement over the strongest baseline is 1.8 score points. While you do a compute-matched experiment for ALFWorld, we cannot be sure that the same benefits carry elsewhere. If you can, I would like to see WebShop comparisons under matched total resources, allowing baselines to spend the additional budget on training, to determine whether RMPO offers a better trade-off.

I think both points are well addressable!

Questions for authors

No particular questions from my side. I really appreciate the authors thoroughness and as said, I think my criticism is pretty addressable, either in text/commentary or additional experimental insights.

3 comments

Dear Reviewer EnXL,

Thank you again for your encouraging review. As the discussion period ends on Nov 18 (AoE), we wanted to check whether our response addressed your two points:

  1. Deletion and exploration: an observation-masking audit (T1) confirms that deletion misses most pure information value. However, on 200 held-out exploration-heavy tasks (T2), RMPO still beats PGPO by +4.64 [3.12, 6.16]. We will state this limitation clearly in the main text and the Figure 7 caption.

  2. WebShop under matched resources: at equal wall-clock time, RMPO is competitive with PGPO (+0.93 [−0.10, 1.95]), and adding the replay correction on top of PGPO gives +1.60 [0.72, 2.48] (T3b).

If these address your concerns, we would be grateful if you would consider reflecting this in your score. Thank you for your time.

Dear authors,

thank you for your replies. They are improving the paper and eliminating concerns. For now, I keep my already positive score but will carefully monitor this submission in case something goes wrong!

Thanks again!

Thank you for engaging with our response and for your support. We will include the T1–T3 results and the clarified limitation in the revision.

Reviewer_RCNgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to assign credit to individual actions when multi-turn agents receive sparse episode rewards and proposes RMPO. RMPO uses environment replay to obtain score differences through action deletion, prefix progress, and same-state action replacement. It adds the normalized measurements to the episode-level advantage without training a separate value critic. Experiments across three environments and two model sizes show improvements over baselines.

Critical strengths

Clear credit construction grounded in executed comparisons. RMPO combines three environment-side comparisons through a common interface, providing turn-level signals while retaining episode-level feedback. Replay is used only during training, so inference requires no environment restoration or additional comparisons.

Critical weaknesses

Replay credit can differ from an action’s value at decision time. ALFWorld deletion measures necessity for a realized suffix and may undervalue information-gathering actions that helped the policy choose that suffix. The authors acknowledge that the normalized correction is biased shaping and need not preserve the original optimal policy, leaving its effect on exploration insufficiently tested.

Questions for authors
  1. Can the authors quantify which information-gathering or exploration actions deletion undervalues, and test whether this bias affects performance on new tasks requiring exploration?

  2. Since failed ALFWorld trajectories receive no deletion correction, how do gains and measurement coverage change when the initial policy has a lower success rate?

  3. Can the authors provide wall-clock- or resource-matched comparisons for WebShop and Search-QA, accounting for the reported CPU replay work and reader calls?

  4. IGPO[1] is closely related to RMPO’s prefix-progress signal, especially in Search-QA, as both use changes across interaction prefixes to provide turn-level supervision. The paper should discuss this connection and ideally compare with IGPO under the same Search-QA setup to clarify the additional benefits of replay-based comparisons.

[1] Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents, ICLR'26

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.