- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RAHL, which converts incorrect rollouts generated by an agent during training into successful trajectories by reconstructing the tasks and agent sequences. This allows the method to utilize trajectories that fail their original tasks and aims to address the lack of training signals in GRPO when all rollouts in a group receive zero rewards. Through experiments and analyses on ALFWorld, WebShop, and ScienceWorld, and comparisons with a GRPO-only baseline, the paper demonstrates the effectiveness of the proposed method.
Critical strengths
-
The paper addresses a practical challenge in agent reinforcement learning: groups of failed rollouts with identical fail rewards provide no group-relative reward advantage in GRPO. The motivation is clearly presented, and the examples effectively illustrate the method.
-
The proposed method, RAHL, can be used to design tasks tailored to the agent and may also be extended to areas such as curriculum learning.
Critical weaknesses
-
RAHL is a task synthesis method based on achieved outcomes. However, utilizing the newly synthesized tasks requires additional rollouts, and the method does not fully reuse all previously collected rollout trajectories, which still results in wasted computation. Previous work has already discussed the lack of training signals caused by groups of entirely failed rollouts [1][2][3][4]. The authors should clarify how RAHL differs from these methods and include them as baselines in the experiments.
-
The results do not convincingly demonstrate the effectiveness of RAHL. For example, in Table 2, adding RAHL to GSPO and GiGPO results in worse performance than their respective baselines, while Figure 4 does not demonstrate the benefits of RAHL. The authors should also use appropriate visual formatting, such as boldface in the tables, to make the comparisons clearer to readers.
[1] "ExGRPO: Learning to Reason from Experience", Zhan, Li, et.al. [2] "ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay", Lu, Zhong, et.al. [3] "Learning from Hindsight for VLA Reinforcement Learning", Xu, Jiang, et.al. [4] "CRPS: Curriculum Replay via Progressive Suffixes from Successful Trajectories for Long-Horizon LLM Agents", Zhang, Yang.
Questions for authors
- How do the authors interpret “Recursive Self-Improvement” in the title? The paper does not improve the model’s capabilities as a goal proposer, span classifier, or in other such roles, but only improves its capabilities as an actor. Meanwhile, Figures 3 and 4 do not provide sufficient evidence of sustained improvement over longer training horizons.