- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes -GRPO, an actor-critic variant of GRPO for RLVR that combines group-normalized terminal rewards with a per-token length penalty and a lightweight critic trained on -return targets. Token-level GAE advantages are then standardized across the token batch and used in a clipped policy objective with asymmetric upper clipping and reference-policy KL regularization. Experiments on GSM8K and MATH with Qwen2.5-1.5B and 7B report improved greedy accuracy and shorter responses relative to PPO, RLOO, GRPO, VAPO, and DAPO.
Critical strengths
The paper addresses an important limitation of outcome-level RLVR: whether finer-grained credit assignment can improve accuracy without encouraging unnecessarily long responses. The proposed method achieves the highest accuracy among the compared methods on both GSM8K and MATH at both model sizes, while generating shorter GSM8K responses. These results suggest that the method could be practically useful.
The proposed method combines a lightweight critic, GAE, token-level advantage standardization, asymmetric clipping, and an annealed schedule. The paper provides a clear training procedure, component ablations, and additional analyses of the schedule and response length. These experiments help explain the behavior of the overall method, although they do not fully isolate the contribution of the learned critic.
Critical weaknesses
-
The contribution of learned token-level credit assignment is not sufficiently isolated, and the incremental novelty remains unclear. Although the paper provides useful ablations, the contributions of individual components are not fully isolated. The full method adds at least four consequential modifications to GRPO: the learned critic and -returns, a per-token penalty, token-batch advantage normalization, and an asymmetric clip-higher objective. It lacks two important controls: -GRPO without the length penalty (), and a critic-free GRPO/DAPO variant with the same penalty and otherwise matched optimization settings. As a result, the specific contribution of the learned critic cannot be separated from reward shaping and other changes to the training recipe. Another concern is the limited novelty compared with closely related work. GRPO- [1] already explores critic-free -based credit assignment, while VAPO [2] combines a learned value model with GAE variants. Although -GRPO uses a different training recipe, its incremental contribution beyond these approaches is not yet sufficiently established.
-
The theoretical analysis contains errors that weaken the justification for the proposed method. Proposition 3 contains an term, which is undefined under the experimental setting . Its discussion of the case is also inconsistent with the displayed bound. In addition, the variance recursion omits the covariance term , so the claimed monotonic increase in variance with is not established without further assumptions. These issues weaken the bias-variance justification for the proposed -annealing schedule.
Questions for authors
-
Could the authors report (i) -GRPO with and (ii) a critic-free GRPO/DAPO control with the same length penalty and otherwise matched clipping, KL treatment, and token-level loss?
-
What is the specific contribution beyond the critic-free -based method GRPO- [1] and the value-model/GAE design already explored in VAPO [2]? A budget-matched comparison with GRPO- would help determine whether the proposed method provides additional benefits under comparable computational resources.
-
Can the authors provide a finite-horizon version of Proposition 3 valid for , clarify the case, and account for the missing covariance term in the variance analysis?
-
Can the authors provide wall-clock throughput, peak memory, retained rollout counts after group filtering, and total FLOPs or a comparable compute accounting? This would clarify the equal-compute claim.
[1] Parthasarathi, Prasanna et al. “GRPO-λ: Credit Assignment improves LLM Reasoning.”, ArXiv abs/2510.00194 (2025)
[2] Yue, Yu et al. “VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks.” ArXiv abs/2504.05118 (2025)
I will maintain my original score, as I have not received a response from the authors.