- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines the consistency between GRPO’s training updates and the intended optimization objective, focusing on KL regularization. It proposes FRPO, which first computes GRPO advantages from the original outcome rewards and then applies a token-level correction based on the cumulative KL cost from the current position to the end of the response. Experiments across multiple mathematical reasoning benchmarks demonstrate the method’s effectiveness.
Critical strengths
- The paper addresses a well-defined problem and provides a clear and convincing motivation.
- The proposed method is simple to implement and adds little computational overhead to KL-regularized GRPO. Its default implementation reuses existing token log-probabilities to compute a reverse cumulative sum of log-probability ratios, requiring neither a value model nor additional forward passes.
Critical weaknesses
-
Evaluation on additional GRPO-style algorithms, such as DAPO, is needed to assess the method’s generality.
-
All experiments use Qwen backbones. Given the reported contamination concerns on mathematical benchmarks for this model family [1], evaluation on other backbone families would strengthen the empirical evidence.
-
The authors show that existing methods are inconsistent with their chosen objective, , but do not sufficiently justify why this objective should be preferred. Gradient inconsistency alone does not establish that the original method is flawed: a local KL penalty can serve as a constraint evaluated on fixed sampled contexts, even if it does not recover the gradient of a full trajectory-level objective.
-
The paper does not directly explain why restoring the missing gradient terms should preserve exploration and improve reasoning accuracy. The connection between the theoretical correction and these benefits remains primarily empirical.
[1] Shao R, Li S S, Xin R, et al. Spurious rewards: Rethinking training signals in rlvr[J]. arXiv preprint arXiv:2506.10947, 2025.
Questions for authors
See Weaknesses
2 comments
Response to Reviewer QRhP
Dear Reviewer QRhP,
Thank you again for your thoughtful feedback. When you have a chance, we would appreciate hearing whether our rebuttal addresses your concerns. We added results on Gemma 4 E2B-it and a tuned DAPO-style clip-higher baseline, and clarified why we use trajectory-level KL and how it can help preserve reference-supported solution paths.
If the new results and clarifications address your concerns, we would be grateful if you would consider revisiting your rating. We are happy to clarify any points that remain open.
Thank you for your time, The authors
Response to Reviewer QRhP
We sincerely thank the reviewer for their careful reading and thoughtful questions.
1. Additional GRPO-style algorithms
We re-ran DAPO on Qwen3-1.7B-Base under exactly our configuration, using clip-higher and no KL. We report pass@16, mean@16, and mean response length for FRPO and DAPO with different values of ϵhi.
We tune the hyperparameters of the DAPO baseline, and the one with ϵhi=0.28 reaches a mean response length of 3769 tokens, exceeding the response length of FRPO. Despite its longer responses, DAPO reaches 88.33, compared with FRPO's 90.13, still with a gap of 1.8 points.
2. Additional backbone model families other than Qwen
We use Gemma 4 E2B-it as an additional backbone model to verify the effectiveness of FRPO, and the table below summarizes the results across different configurations, with each cell showing the pass@16 performance as a percentage. FRPO achieves the best overall pass@n performance compared to other baselines, which further validates the generalizability of the proposed method.
3. Regularized RL objective
We agree that a local KL penalty is a valid way to regularize the policy on fixed sampled prefixes. Our objective asks how much the policy drifts from the reference over complete responses that it generates. We retain GRPO’s reward utility ℓ(p) and use Ktraj to measure that response-level drift.
The distinction matters because the model chooses its own prefixes. An early token can incur a small immediate KL cost and still lead the model into a line of reasoning where it repeatedly diverges from the reference. By the KL chain rule, those later costs contribute to the divergence between the two distributions over complete responses. A fixed-context penalty treats the distribution of prefixes as given. Future-KL accounts for how an early choice changes which prefixes the model visits and assigns the resulting downstream KL cost back to that choice. This is why we prefer the trajectory-level objective when the goal is to regularize the model’s full generated responses.
4. Connection between FRPO and exploration ability
With a binary verifier, every correct completion contributes the same reward. At a fixed success probability p, the reward utility ℓ(p) is therefore indifferent to whether the policy spreads its successful responses across several solution paths or concentrates them on one. Completion-level KL distinguishes these cases. By the same chain rule, at fixed p it is minimized by matching the reference’s conditional distribution over correct completions, and likewise over incorrect completions. Thus, at the unrestricted population optimum with β>0, increasing the overall probability of success can preserve the relative probability of every reference-supported correct path. This gives a concrete reason to regularize the completion distribution. It protects existing ways of solving the problem even though the verifier gives no extra reward for keeping them available. Future-KL carries that regularization signal back to the earlier tokens that select a path.
In Appendix F.7, at the same β, suffix credit reached 92.0% pass@16 and 68.1% mean@16 on MATH500, versus 86.0% and 61.2% for whole-response broadcast credit, even though the broadcast run had higher token entropy. This comparison supports the practical importance of appropriate credit assignment and regularization, not simply boosting the entropy.