- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes RMPO, which constructs three types of turn-level signals—action deletion, prefix progress, and action replacement—through environment replay, normalizes them, and adds them to the episode-level advantage for GRPO-style policy optimization. The authors evaluate RMPO on ALFWorld, WebShop, and Search-QA with Qwen3-4B/8B, and report improvements over GRPO, GiGPO, and other baselines. The engineering implementation is relatively complete, but the methodological contribution, theoretical properties, and experimental persuasiveness are insufficient to support acceptance.
Critical strengths
1、 The paper implements replay-based credit assignment across three multi-turn interaction environments, covering two model scales and multiple baselines.
2、 It provides analyses of replay coverage, execution cost, and partial ablations, with relatively detailed experimental protocols and implementation details in the appendix.
3、 Delayed rewards and turn-level credit assignment in multi-turn LLM agents are important problems, and the paper’s engineering exploration of this problem has some value.
Critical weaknesses
1、 Limited novelty.
Action deletion is essentially an executable version of leave-one-out / difference reward, prefix progress resembles dense progress shaping, and action replacement resembles counterfactual action evaluation. The authors themselves acknowledge that the deletion primitive is not new. Overall, the work feels more like an engineering combination of existing credit assignment ideas for LLM agent training, rather than a new learning objective or theoretical mechanism.
2、 The method relies on strong environmental assumptions and external evaluators, and its theoretical properties are biased.
RMPO requires resettable environments, reproducible states, fixed evaluators, executable suffixes, and candidate action sets, which significantly limits its applicability. The so-called “critic-free” approach actually replaces the value critic with environment evaluators, fixed readers, or handcrafted completions, whose biases and costs directly enter the training signal. The authors also acknowledge in the appendix that the normalized deletion correction is generally not potential-based, that RMPO is a biased shaping objective, and that it need not preserve the optimal policy of the original reward.
3、 Experimental fairness and statistical significance are insufficient.
Some gains are small, and the paper does not report paired significance tests or confidence intervals. The base results use a single seed and thus have limited reference value. WebShop replay accounts for about 33% of per-iteration time, but compute-matched comparisons are only provided for ALFWorld; WebShop and Search-QA lack comparisons under equal GPU-hours or equal evaluator-call budgets.
4、 Ablations and audits are insufficient to isolate the core contribution.
Different environments use different replay rules, making it difficult to separate the independent contributions of each signal type. In ALFWorld, the trajectory-mean control already recovers about 38% of the gain, indicating that trajectory-level length or aggregation effects contribute substantially. The high correlation between deletion and symbolic planner progress also partly stems from their shared use of hindsight information, and does not demonstrate that the signal has ex-ante decision value for the acting policy.
Questions for authors
1、Compared with difference reward, COMA, leave-one-out, and counterfactual credit assignment, what is the core algorithmic or theoretical contribution of this paper beyond “executable replay”?
2、Is RMPO still applicable in real-world environments that are not resettable, whose stochasticity cannot be controlled, or that lack a fixed evaluator?
3、Can the authors provide significance tests, confidence intervals, and compute-matched comparisons on WebShop and Search-QA for the main results?
4、Does the absence of deletion credit for failed trajectories lead to reduced exploration or overfitting to successful trajectories? How can the authors demonstrate that the gains come from the turn-level replay signal itself, rather than normalization or length effects?
Dear Reviewer uA82,
Thank you again for your detailed review. As the discussion period ends on Nov 18 (AoE), we wanted to check whether our response addressed your main concerns:
Significance: with paired seeds, all seven 8B comparisons in Table 1 have 95% CIs that exclude zero. Under equal wall-clock budgets, WebShop and Search-QA margins are smaller and include zero, and we will report both settings.
Length vs. turn placement: a residual-only control that keeps turn placement but removes the per-trajectory total recovers about 87% of RMPO’s gain over GRPO; the trajectory-mean control recovers about 38% (T5).
Decision-time value: an observation-masking audit (T1) shows where deletion misses information value, and on held-out exploration tasks RMPO still beats PGPO by +4.64 [3.12, 6.16] (T2).
Scope: we will state the resettable-environment requirement in the abstract, replace “critic-free” with “without a learned value critic”, and move the biased-shaping statement into the main text.
We would appreciate it if you could let us know whether any concerns remain. We are happy to clarify before the deadline.