- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies cross-segment credit misattribution as a structural flaw of GRPO for tool-calling agents: a shared trajectory advantage lets summary reward noise corrupt tool gradients, misassigning credit between tool-use and answer-generation segments. To address this within one unified model without extra rollouts, it proposes SLCA-GRPO. Using gradient masking, Hierarchical Rewards (HierR) and Segment-Locked Credit Assignment (SLCA), the method decouples, separately normalizes and exclusively routes tool and summary advantages to their corresponding tokens, isolating tool gradients from summary rewards. A schema-guided simulator SGLS enables low-cost large-scale RL without live APIs. With per-update theoretical isolation guarantees, SLCA-GRPO outperforms GRPO, ToolPO and RLTR across three backbones and benchmarks including BFCL and τ2-Bench and diagnostic and ablation experiments validate the identified failure and the necessity of SLCA, SGLS and HierR.
Critical strengths
This work proposes a novel training method targeting the credit assignment problem and validates its effectiveness.
Critical weaknesses
- Innovation is limit. Existing process-reward methods already support assigning position-wise differentiated rewards, which represents a conventional paradigm.
- The design of process rewards itself leaves substantial room for further investigation. The paper assigns low rewards to erroneous tool calls, yet this may be overly simplistic. For instance, an incorrect tool call within an otherwise complete reasoning chain, where the model detects the mistake and subsequently corrects it, may still carry learning value and should not necessarily receive a low score. Moreover, some redundant tool calls can serve as careful environmental verification to prevent downstream mistakes. Therefore, the criteria for judging correct/incorrect tool calls require more explicit elaboration.
Questions for authors
- Only single-round results on BFCL are reported.
- The empirical gains from the proposed credit assignment mechanism appear limited.
- What are the concrete rules used to judge whether a tool call is correct or incorrect?