- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SP-GRPO, which is a search-based agentic RL algorithm enhanced with process advantages. Specifically, SP-GRPO collects the relative advantages of intermediate steps via additional sampling and uses these signals for a later RL framework. Extensive experiments validate the effectiveness.
Critical strengths
- This paper introduces process advantages into agentic RL in a simple way.
- The experiments in this paper are solid and fully support the claims.
Critical weaknesses
- The explanation of the method in this paper lacks sufficient visual clarity. In fact, I spent a significant amount of time trying to grasp the content in Section 3. I believe it would be beneficial to include more illustrative examples and figures to help readers understand the shared-prefix sampling process, as well as how the "advantage" is calculated. The single diagram currently provided (Figure 1) is not enough.
- There is already a substantial body of work on search-agent RL (such as https://arxiv.org/pdf/2510.24302, https://arxiv.org/pdf/2507.19849, https://arxiv.org/abs/2602.11767). The primary contribution of this paper lies in synthesizing existing process-based rewards, group-relative advantages, and branch sampling into a coherent framework. Its unique design centers on a combination of "comparing candidates that share the same prefix," "continuing with a single randomly sampled path," and "defining the corresponding scope for updates." Notably, the concepts of process supervision, information-gain rewards, and the PPO/GRPO objectives themselves are not original to this paper. This makes the novelty less significant.
Questions for authors
In my view, the essence of the success of the proposed method lies in trading more computation for more precise supervision signals. Could you further discuss the advantages of these methods under the constraint of a fixed total computational cost?
2 comments
Thanks for your response. I think the clarification and the new figure have largely improved the illustration of this paper. It is an empirically solid paper, and I still maintain my positive recommendation.
Response to Reviewer iyRu
We thank the reviewer for acknowledging that SP-GRPO introduces process advantages into agentic RL in a simple way, and that our experiments are solid and fully support the claims. We respond to each concern below.
We thank the reviewer for this constructive suggestion. In the revised paper, we have added a new method-overview figure (Figure 2) that illustrates the shared-prefix sampling process, the within-group advantage computation for both process and answer groups, and how the collected trajectories enter the loss at the token level. We hope these changes make the method substantially easier to follow, and we are happy to add further illustrations if needed.
We thank the reviewer for raising this point, which allows us to clarify the nature of our contribution. Our goal is indeed not to introduce yet another reward signal or objective variant, but to identify a structural failure mode that arises when existing components are combined in the usual way, and to show that resolving it requires redesigning the rollout itself. Prior methods estimate turn-level advantages from evidence collected at the trajectory level. We identify this granularity mismatch as the unifying failure mode of chain- and tree-based designs, and show it is not benign: aligning turns by index with an identical process reward loses 11.3 points (Step-GRPO), and exact post-hoc state matching starves the comparison groups (Figure 3 in the revised manuscript, GiGPO −5.1). To address this limitation, we propose, to our knowledge, the first framework that obtains a complete same-state comparison group at every turn while keeping rollout cost linear in the horizon, through dense same-prefix branching, instantaneous within-state comparison, and uniform single-path continuation. The components interact non-trivially: dense branching with greedy or IG-weighted continuation collapses to 17.2/46.0 (Table 3), because content-dependent continuation breaks the on-policy property (Proposition 1). The uniform rule is therefore a load-bearing, non-obvious part of the design, and it is what makes the formal guarantees (Proposition 1, Theorems 1–2) possible at all, which prior combinations lack. The practical value of resolving the mismatch is demonstrated empirically: SP-GRPO achieves the best average performance across seven single- and multi-hop benchmarks at both model scales, outperforming trajectory-level, step-level, and tree-based baselines. As for the works cited by the reviewer, they branch only at selected points, triggered by token entropy or a learned score, and score candidates by simulated or trajectory-level returns, often with content-dependent selection, so they do not form a complete same-state comparison group at every turn. We have added these works to the related work of the revised manuscript for a complete comparison.