- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SR-GRPO, a two-stage reinforcement learning framework that jointly trains a single language model to solve reasoning problems and refine its own solutions using binary outcome verification. The approach introduces three mechanisms: (1) wrong-first sampling, which prioritizes incorrect initial solutions for refinement; (2) a ground-truth-free refinement stage trained through GRPO; and (3) self-distilling feedback, which incorporates verified corrections into the initial solver's training group. The approach does not require an external teacher, learned critic, or process-level annotations.
The method is evaluated on three reasoning-model backbones at 1.5B and 4B scales, trained on DAPO-Math-17k. SR-GRPO improves over the corresponding GRPO baselines across nine mathematical reasoning benchmarks and two out-of-distribution benchmarks, GPQA Diamond and MMLU-Pro. The papers also includes ablations on a larger-group GRPO baseline at 4B, test-time refinement evaluations, and LLM-as-judge comparisons of reasoning quality.
Critical strengths
S1: The empirical results are a strong point of the paper. SR-GRPO improves average math accuracy from 54.8% to 63.3% on DeepSeek-1.5B, 64.5% to 71.4% on Nemotron-1.5B, and 76.0% to 80.1% on Qwen3-4B (Table 1). The gains are particularly noticeable on harder benchmarks such as AIME and HMMT. The improvements on GPQA Diamond and MMLU-Pro (Table 2), despite training only on math, further strengthen the empirical contribution.
S2: Table 4 shows substantial performance drops when individual components are removed. The G=16 GRPO comparison in Table 6 is also valuable, as it suggests that additional sampling alone does not explain the gains. Importantly, these improvements require no additional inference-time computation.
Critical weaknesses
- W1: The delta-accuracy refinement reward from the paper seems to be mathematically redundant under GRPO.
In Eq. (6), the authors define the refinement reward as:
However, all refinements in a group are conditioned on the same initial solution (Eq. (4)). Define:
Since is constant across the refinement group, the group mean and standard deviation become:
Substituting into GRPO's group-relative advantage (Eq. (2)):
Therefore, the initial-solution correctness term cancels exactly. The resulting advantages are identical to those obtained using ordinary outcome correctness plus the format reward.
This means the proposed delta reward does not provide a distinct optimization signal or explicitly distinguish correction from corruption beyond what ordinary correctness already provides. Although the two-stage training procedure may still be effective, this challenges the paper's explanation of its reward mechanism.
Questions for authors
-
WQ1: Can the authors confirm whether replacing Eq. (6) with ordinary correctness plus the format reward produces identical refinement-stage gradients? If not, which implementation detail breaks this mathematical equivalence?
-
WQ2: In Eq. (9), verified refinement traces are sampled from but added to the solve-stage GRPO group. Under which context are their policy log-probabilities and importance ratios computed: or ? If computed under , how is the mismatch between the sampling and optimization distributions handled?
6 comments
Reply to Reviewer 8Wvs [Part 2]
Q2: Can the authors confirm whether replacing Eq. (6) with ordinary correctness plus the format reward produces identical refinement-stage gradients? If not, which implementation detail breaks this mathematical equivalence?
A2: Thank you for this direct question. We would like to clarify that, as discussed in A1, this replacement indeed gives identical advantages, and hence identical gradients. However, these advantages still distinguish good self-refinements of the same from bad ones in the same way, which provides an effective improvement signal for self-refinement, as verified by our ablation (see A1).
Q3: In Eq. (9), verified refinement traces are sampled from but added to the solve-stage GRPO group. Under which context are their policy log-probabilities and importance ratios computed: or ? If computed under , how is the mismatch between the sampling and optimization distributions handled?
A3: Thank you for this precise question. We would like to clarify that, when training the solver, they are computed under the question-only context : each verified refinement is appended to the solve group of the same query and trained exactly like the model's own solve rollouts.
That is, when training the solver, we treat solutions generated under an augmented prompt (i.e., self-refinement) as solutions to the original question without an off-policy correction (e.g., importance weighting for the change of context), an off-policy approximation widely used in prior work [1, 2, 3, 4]: STaR's rationalization trains on hint-conditioned rationales "as if the model had come up with the rationale without the hint" [1], and imitation learning from language feedback (ILF) [2], critique-in-the-loop self-improvement [3] and RL from text feedback (RLTF) [4] train in a similar way. Among them, RLTF [4] uses this approximation even though its augmented prompt contains feedback from an external model, and finds that "variance dominates bias": an importance-weighting correction increases variance and lowers performance. In contrast, our method relies on no external model, teacher or critique: the same model performs a single round of self-refinement. We therefore adopt this widely used approximation, which keeps our method simple and easy to use. It is also effective in our setting: removing the recycling of verified refinements into the solve group loses most of our gain over GRPO (Table 4). Moreover, a dedicated off-policy correction is not the main focus of this paper, and we leave it to future work.
Following your suggestion, Sec. 3.4 and App. C.7 of the revised paper now clearly state that the recycled refinements are off-policy, and that we use them to train the solver in the context of the original questions without an off-policy correction, following a similar practice in prior work [1, 2, 3, 4].
Thank you again for your valuable comments on the delta reward and on how recycled refinements are trained, which have greatly helped improve the clarity of our paper. Following your suggestions, the revised paper now clarifies that the delta reward, although it gives the same advantage as the outcome reward, still provides a meaningful learning signal for self-refinement, and that the recycled refinements are trained in the context of the original questions as an off-policy approximation, following similar practice in prior work. We hope our responses have fully addressed your concerns, and we sincerely thank you for your positive assessment of our work. Should any point require further clarification, we would welcome the opportunity to continue the discussion.
References
[1] Zelikman, et al. "STaR: Bootstrapping Reasoning With Reasoning." Advances in Neural Information Processing Systems 35, 2022.
[2] Scheurer, et al. "Training Language Models with Language Feedback at Scale." arXiv:2303.16755, 2023.
[3] Xi, et al. "MathCritique: Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision." IJCAI 2026.
[4] Song, et al. "Expanding the Capabilities of Reinforcement Learning via Text Feedback." ICML 2026.
Thank You for Your Careful Review and Insightful Suggestions
Dear Reviewer 8Wvs,
Thank you again for your valuable comments on the delta reward and on how recycled refinements are trained, which have greatly helped improve the clarity of our paper. Following your suggestions, the revised paper now clarifies that the delta reward, although it gives the same advantage as the outcome reward, still provides a meaningful learning signal for self-refinement, and that the recycled refinements are trained in the context of the original questions as an off-policy approximation, following similar practice in prior work. We hope these clarifications have fully addressed your concerns, and we sincerely thank you again for your positive assessment of our work. Thank you again for your time and effort in reviewing our paper.
Reply to Reviewer 8Wvs [Part 1]
We sincerely thank you for the careful and constructive review. We greatly appreciate your recognition that "The empirical results are a strong point of the paper," that "The G=16 GRPO comparison in Table 6 is also valuable, as it suggests that additional sampling alone does not explain the gains," and that "these improvements require no additional inference-time computation."
We understand your concerns focus primarily on the role of the delta reward under GRPO's group normalization and on the context under which the log-probabilities and importance ratios of recycled refinements are computed.
To address these thoroughly, we have (1) clarified that, even though the delta reward and the outcome reward give the same advantage, this advantage still distinguishes good self-refinements from bad ones in the same way, so self-refinement is trained effectively, as confirmed by our ablation (Sec. 3.2; Table 4, w/o Self-Refine Stage Training); and (2) clarified that, when training the solver, recycled refinements are used under the question-only context without an off-policy correction (e.g., importance weighting for the change of context), following a similar practice in prior work [1, 2, 3, 4] (Sec. 3.4, App. C.7). All revisions have been integrated into our revised manuscript. Below, we address each specific question systematically.
r2(j)=1[correct(t2(j),q)]−1[correct(t1∗,q)]+λ1[format(t2(j))] xjcr2(j)=1[correct(t2(j),q)]+λ1[format(t2(j))]=1[correct(t1∗,q)]=xj−c μr σr =G21j=1∑G2(xj−c)=μx−c=G21j=1∑G2((xj−c)−(μx−c))2=σx A2(j)=σrr2(j)−μr=σx(xj−c)−(μx−c)=σxxj−c−μx+c=σxxj−μxA1: Thank you for this careful derivation. We would like to clarify that, since all refinements of a query revise the same solution t1∗, the delta form and the outcome reward indeed give identical refinement-stage advantages, but these advantages still effectively distinguish good self-refinements from bad ones in the same way and thus provide an effective signal for training self-refinement. For example, when t1∗ is incorrect, a refinement that corrects it receives a higher advantage than one that leaves it incorrect; when t1∗ is correct, a refinement that keeps it correct receives a higher advantage than one that corrupts it. This is confirmed by our ablation: removing the refinement-stage training loses most of our gain over GRPO (Table 4):
We keep the delta form only because it is more intuitive to read, and we have added an explanation of this equivalence of the advantages under GRPO to Sec. 3.2, App. C.5 and the caption of Figure 1 of the revised paper, which makes the paper clearer; thank you for this suggestion.