- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SA-GRPO, which samples tokens for the policy-gradient objective according to their functional roles in an agent trajectory. Search decisions and novel queries receive high inclusion probabilities, while answers, reasoning, and repeated queries receive different probabilities under an expected token budget. Experiments compare full GRPO, position-based sampling, role-based sampling, and an entropy baseline on two models and two multi-hop QA datasets. Analyses include overall performance and subsets defined by the number of generated searches.
Critical strengths
- The method is simple and exploits readily available trajectory structure. Tool-use tags provide role information without requiring a separately trained importance estimator.
- The abstract and main tables report unfavorable comparisons, such as S-GRPO leading on HotpotQA. However, some later claims are stronger than the evidence, including “confirms the mechanism” in Appendix C.2.
Critical weaknesses
- The experiments do not distinguish role weighting from stochastic sparsification. Role-dependent inclusion changes the contribution of different tokens to optimization. Without appropriate importance correction, it should not be interpreted as an unbiased estimate of the original uniformly weighted objective. A deterministic role-weighted baseline is missing, so the benefit cannot be attributed specifically to selecting fewer tokens. The motivating claim that GRPO is unstable under LoRA also lacks direct diagnostics and sits uneasily with the paper’s own results: under the same QLoRA configuration, GRPO on Qwen scores 0.763 on HotpotQA and 0.579 on MuSiQue, above S-GRPO on both. In addition, SA-GRPO’s non-mandatory inclusion probabilities are higher than S-GRPO’s, and the budget scaling applies only when the expected count exceeds k. Realized token counts may therefore differ across arms, but they are not reported.
- The main multi-hop subgroup evidence is affected by selection, while overall superiority is inconsistent. "Exactly two searches" is determined by each model's behavior, so the compared subsets contain different questions and sample counts. Their score difference is not a matched comparison on a common multi-hop test set. Overall, SA-GRPO does not lead on HotpotQA and has the lowest score among the RL arms on Qwen (0.759). The strongest result, Mistral/MuSiQue, is evaluated at step 200 rather than after a full pass, without an explained selection rule. Its significance test is against the base model, with no direct test against S-GRPO reported, so “the only arm to improve” does not establish that SA-GRPO beats S-GRPO. The paper does not mention multiple training seeds, and the bootstrap confidence intervals described in the methods do not appear in the results.
Questions for authors
- How was step 200 chosen for Mistral/MuSiQue, and did it apply to all arms?
- Are the improvements over S-GRPO stable across training seeds, and what is their direct comparison with confidence intervals?
- Does stochastic role sampling outperform deterministic role-weighted GRPO on a common test subset defined in advance by task properties?