- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces AgentOPSD, a credit-assignment method for multi-turn LLM agents trained with GRPO. It reweights the advantage at each turn using a self-distillation signal. For each turn, it measures how much a training-only retrieved “skill” in the prompt increases the policy’s likelihood of the action taken. It then accumulates these per-turn gaps into a recursive “success belief” in log-odds space, initialized with the group success rate. Experiments on ALFWorld, WebShop, and Search-QA with Qwen2.5-3B/7B and Qwen3-1.7B show that AgentOPSD achieves the best or near-best performance on most aggregate metrics.
Critical strengths
- Turn-level credit in critic-free agentic RL is an important open issue. Combining it with privileged self-distillation signals is timely. Also, the methodology is simple and drop-in. It adds only a cheap advantage reweighting on top of GRPO. It needs no critic and no extra rollouts.
- The baseline suite is comprehensive, covering training-free baselines, GRPO with and without skills, and six self-distillation variants, all using the same set of skills. This makes the comparison of how the supervision signal is utilized reasonably fair.
Critical weaknesses
- The evaluation sets are small. ALFWorld percentages imply roughly 128 evaluation tasks, so one task is about 0.8 points. The 7B gaps to StepOPSD (88.4) and Skill-GRPO* (88.3) are about one task, and per-category numbers rest on 8–30 episodes each.
- The reweighting is too small to plausibly explain the reported gains. Under and , each turn's advantage is scaled by a factor in , at most ±10% around GRPO, but the method improves on GRPO by 8 points on ALFWorld-7B. Removing a single component (e.g., the prior) drops performance to 78.9, which is below GRPO itself. A ±10% modulation is unlikely to produce swings of this size, which points to run-to-run variance rather than the mechanism.
Questions for authors
Questions: See weaknesses.
Minor issues:
- The abstract contains a broken phrase ("requiring neither additional rollouts").
- At the beginning of Section 2.1, Eq. (1) should be placed after "At turn k, it samples...".