- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to train multi-turn agents with few tasks when repeated failures leave GRPO without useful reward signals. It proposes PATH-GRPO, where a teacher chooses a restart turn and then uses the reference solution to write a corrective hint. The agent retries from that turn, and the hint is removed before the model update to encourage learning without teacher assistance. Experiments show that PATH-GRPO outperforms GRPO, with targeted turn-level hints working better than global hints.
Critical strengths
The paper addresses an important problem: how to train multi-turn agents with few tasks and sparse outcome rewards. This matters because repeated failures can leave GRPO without useful reward signals, making learning especially difficult when successful trajectories are scarce.
Critical weaknesses
W1: The placement experiments also change both the hint location and its content, making it difficult to attribute their effects.
W2: The paper lacks enough discussions and comparisons with process-reward methods, such as StepSearch[1] and IGPO[2]. A suitable comparison would clarify whether teacher-guided retries offer advantages over existing ways of handling sparse rewards.
[1] StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization,EMNLP 2025.
[2] Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents,ICLR 2026
Questions for authors
Q1. The AppWorld SFT baseline uses synthetic tool responses and reference programs that the agent cannot directly execute. Comparing against verified, executable teacher demonstrations would help clarify whether its poor performance comes from limited data or mismatched training examples.
Q2. The paper does not report how often hint removal fails or hint text remains in the training context. It would be useful to report these rates and check whether the gains remain after filtering such cases.
Q3. Equal trajectory counts do not imply equal training cost, since PATH-GRPO requires teacher calls and environment replay. A comparison under the same total budget would clarify whether it offers advantages over giving GRPO more rollouts.
Is the pdf updated to the lastest version?