- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes RACE, a method that credits actions of long-horizon LLM agents according to changes in recoverability. Particularly, the recoverability is estimated from the outcomes of semantically similar, task-compatible contexts in other trajectories within one training batch. It uses frozen embeddings, nearest-neighbor retrieval, and regularization toward the batch outcome average. The clipped change across an action adjusts the base advantage, optionally targeting steps with weak base credit.
Results show performance gains against GiGPO and several other baselines (from public results) using Qwen3-4B and Qwen3-8B on ALFWorld, ScienceWorld, and WebShop, with improved zero-shot transfer to BabyAI, PDDL, and TextCraft on average.
Critical strengths
The method combines nearest-neighbor value estimation with action-level advantage correction for group-relative RL. In particular, when all sampled trajectories within a group fail and within-group comparisons provide little learning signal, similar experiences from other task instances may help distinguish the contributions of individual actions. This is a well-motivated and targeted direction for improvement.
Critical weaknesses
- The advance over existing credit-assignment methods is insufficiently established.
- RACE motivates its contribution through GiGPO’s reliance on exact-state recurrence, but GiGPO (section 5.1 in [1]) already includes similarity-based anchor grouping. A controlled comparison against this variant is necessary to establish the added value of recovery-aware estimation beyond relaxing exact-state matching.
- Only GiGPO is reproduced under matched conditions; other published baselines retain their original training & evaluation protocols, making the comparison unreliable.
- Additional representative step-level credit methods or related work (such as [2-4] which also avoids additional rollouts and annotated process rewards) should also be compared both in performance and cost.
- The central mechanism and claimed plug-in capability lack validation.
- The support rate of over 98% in table 3 measures only retrieval coverage, so multiple records may come from the same trajectory. The paper should also report the number of distinct neighboring trajectories and whether the retrieval weights are overly concentrated. More directly, evaluation in a resettable environment should compare RACE’s action rankings with subsequent success rates estimated through repeated continuation rollouts from alternative actions at the same state (Monte-Carlo rollouts) to validate the credit accuracy of the step-level signals.
- Experiments integrate RACE with several group-based RL algorithms should be added to substantiate the plug-in claim.
- ALFWorld, WebShop, and ScienceWorld provide structured tasks and constrained interaction interfaces; an open-ended interaction benchmark (e.g., a realistic browser or tool-use env) would more directly test the motivation of learning from semantically comparable contexts when exact recurrence is rare.
References:
- Feng, Lang, et al. "Group-in-group policy optimization for llm agent training." Advances in Neural Information Processing Systems 38 (2026): 46375-46408.
- Liu, Xiaoqian, et al. "Agentic reinforcement learning with implicit step rewards." International Conference on Learning Representations. 2026.
- He, Shuo, et al. "Hierarchy-of-groups policy optimization for long-horizon agentic tasks." International Conference on Learning Representations. 2026.
- Fang, Yangyi, et al. "Proximity-based multi-turn optimization: Practical credit assignment for llm agent training." ACL 2026.
Questions for authors
Could we first identify transitions whose pre- and post-action contexts both have sufficient eligible neighbors, then compute recoverability differences only for those transitions? Would this be equivalent to applying the support mask in the advantage update (Eq.8), and are there computational or implementation considerations that motivate the chosen formulation?