acceptodds
Under review as a conference paper at ICLR 2027

SR-GRPO: Reinforcement Learning with Self-Refinement from Verifiable Rewards

Abstract

Reinforcement learning from verifiable rewards (RLVR) with a rule-based binary outcome verifier produces strong chain-of-thought reasoners, yet it rewards only the final answer. The policy never learns to inspect or correct its own intermediate steps, so any error in the reasoning chain propagates to the final answer. Addressing this self-refinement gap faces a dilemma: inference-time prompting methods are deployable but, without ground-truth feedback, often degrade accuracy; training-time approaches do learn to self-refine, but mostly by using a supervision signal stronger than RLVR's binary verifier—teacher correction traces, step-level annotations, or separately trained process reward models. We introduce SR-GRPO (Group Relative Policy Optimization with Self-Refinement), a two-stage RLVR framework in which a single shared policy acts as both solver (generating an initial solution) and refiner (correcting its own mistakes), with both stages trained jointly by GRPO. SR-GRPO uses only the binary verifier RLVR already requires—no external teacher, no separate critic, and no ground-truth answer at refinement. SR-GRPO has three mechanisms: wrong-first sampling keeps self-refinement focused on genuine solver errors; a ground-truth-free refinement prompt makes the refinement stage deployable at inference, since the model never sees the answer when it refines; and self-distilling feedback recycles every verified refinement into the solve stage as additional supervision, so the same policy serves as solver, critic, and teacher within a single training loop. Under a single fixed recipe across three backbone families and two model scales (1.5B and 4B), SR-GRPO matches or surpasses the strongest RL baselines on several math reasoning benchmarks: at 1.5B it matches QwQ-32B with far fewer parameters; at 4B, where vanilla GRPO saturates, SR-GRPO continues to improve, surpassing several open-source 20B+ reasoners. Beyond the math benchmarks, SR-GRPO also improves on out-of-distribution general-reasoning benchmarks (GPQA Diamond, MMLU-Pro) across all three backbone families—evidence that the gains are not math-specific reward hacking. Double-blind LLM judges prefer SR-GRPO's reasoning over strong RL baselines, and qualitative analysis confirms that SR-GRPO genuinely locates and revises erroneous steps during self-refinement. Overall, SR-GRPO turns self-refinement into a training-time self-distillation procedure that adds no inference-time overhead and generalizes across backbone families, model scales, and reasoning domains.

Primary area:
reinforcement learning
Keywords:
reinforcement learning from verifiable rewards, self-refinement, GRPO, LLM reasoning, mathematical reasoning, self-distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8WvsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SR-GRPO, a two-stage reinforcement learning framework that jointly trains a single language model to solve reasoning problems and refine its own solutions using binary outcome verification. The approach introduces three mechanisms: (1) wrong-first sampling, which prioritizes incorrect initial solutions for refinement; (2) a ground-truth-free refinement stage trained through GRPO; and (3) self-distilling feedback, which incorporates verified corrections into the initial solver's training group. The approach does not require an external teacher, learned critic, or process-level annotations.

The method is evaluated on three reasoning-model backbones at 1.5B and 4B scales, trained on DAPO-Math-17k. SR-GRPO improves over the corresponding GRPO baselines across nine mathematical reasoning benchmarks and two out-of-distribution benchmarks, GPQA Diamond and MMLU-Pro. The papers also includes ablations on a larger-group GRPO baseline at 4B, test-time refinement evaluations, and LLM-as-judge comparisons of reasoning quality.

Critical strengths

S1: The empirical results are a strong point of the paper. SR-GRPO improves average math accuracy from 54.8% to 63.3% on DeepSeek-1.5B, 64.5% to 71.4% on Nemotron-1.5B, and 76.0% to 80.1% on Qwen3-4B (Table 1). The gains are particularly noticeable on harder benchmarks such as AIME and HMMT. The improvements on GPQA Diamond and MMLU-Pro (Table 2), despite training only on math, further strengthen the empirical contribution.

S2: Table 4 shows substantial performance drops when individual components are removed. The G=16 GRPO comparison in Table 6 is also valuable, as it suggests that additional sampling alone does not explain the gains. Importantly, these improvements require no additional inference-time computation.

Critical weaknesses
  • W1: The delta-accuracy refinement reward from the paper seems to be mathematically redundant under GRPO.

In Eq. (6), the authors define the refinement reward as:

r2(j)=1[correct(t2(j),q)]−1[correct(t1∗,q)]+λ1[format(t2(j))]r_2^{(j)} = \mathbb{1}[\mathrm{correct}(t_2^{(j)},q)] - \mathbb{1}[\mathrm{correct}(t_1^*,q)] + \lambda\mathbb{1}[\mathrm{format}(t_2^{(j)})]

However, all refinements in a group are conditioned on the same initial solution t1∗t_1^* (Eq. (4)). Define:

xj=1[correct(t2(j),q)]+λ1[format(t2(j))]c=1[correct(t1∗,q)]r2(j)=xj−c\begin{aligned} x_j &= \mathbb{1}[\mathrm{correct}(t_2^{(j)},q)] + \lambda\mathbb{1}[\mathrm{format}(t_2^{(j)})] \\\\[6pt] c &= \mathbb{1}[\mathrm{correct}(t_1^*,q)] \\\\[6pt] r_2^{(j)} &=x_j-c \end{aligned}

Since cc is constant across the refinement group, the group mean and standard deviation become:

μr=1G2∑j=1G2(xj−c) =μx−cσr=1G2∑j=1G2((xj−c)−(μx−c))2 =σx\begin{aligned} \mu_r &=\frac{1}{G_2}\sum_{j=1}^{G_2}(x_j-c)\\\ &=\mu_x-c \\\\[10pt] \sigma_r &=\sqrt{\frac{1}{G_2}\sum_{j=1}^{G_2} \left((x_j-c)-(\mu_x-c)\right)^2}\\\ &=\sigma_x \end{aligned}

Substituting into GRPO's group-relative advantage (Eq. (2)):

A2(j)=r2(j)−μrσr=(xj−c)−(μx−c)σx=xj−c−μx+cσx=xj−μxσx\begin{aligned} A_2^{(j)} &=\frac{r_2^{(j)}-\mu_r}{\sigma_r} \\\\[8pt] &=\frac{(x_j-c)-(\mu_x-c)}{\sigma_x} \\\\[8pt] &=\frac{x_j-\cancel{c}-\mu_x+\cancel{c}}{\sigma_x} \\\\[8pt] &=\boxed{\frac{x_j-\mu_x}{\sigma_x}} \end{aligned}

Therefore, the initial-solution correctness term cancels exactly. The resulting advantages are identical to those obtained using ordinary outcome correctness plus the format reward.

This means the proposed delta reward does not provide a distinct optimization signal or explicitly distinguish correction from corruption beyond what ordinary correctness already provides. Although the two-stage training procedure may still be effective, this challenges the paper's explanation of its reward mechanism.

Questions for authors
  • WQ1: Can the authors confirm whether replacing Eq. (6) with ordinary correctness plus the format reward produces identical refinement-stage gradients? If not, which implementation detail breaks this mathematical equivalence?

  • WQ2: In Eq. (9), verified refinement traces are sampled from (πθ(t2∣q,t1))(\pi_\theta(t_2 \mid q,t_1^)) but added to the solve-stage GRPO group. Under which context are their policy log-probabilities and importance ratios computed: (q)(q) or ((q,t1))((q,t_1^))? If computed under (q)(q), how is the mismatch between the sampling and optimization distributions handled?

6 comments
AuthorsOpenReview ↗

Reply to Reviewer 8Wvs [Part 1]

We sincerely thank you for the careful and constructive review. We greatly appreciate your recognition that "The empirical results are a strong point of the paper," that "The G=16 GRPO comparison in Table 6 is also valuable, as it suggests that additional sampling alone does not explain the gains," and that "these improvements require no additional inference-time computation."

We understand your concerns focus primarily on the role of the delta reward under GRPO's group normalization and on the context under which the log-probabilities and importance ratios of recycled refinements are computed.

To address these thoroughly, we have (1) clarified that, even though the delta reward and the outcome reward give the same advantage, this advantage still distinguishes good self-refinements from bad ones in the same way, so self-refinement is trained effectively, as confirmed by our ablation (Sec. 3.2; Table 4, w/o Self-Refine Stage Training); and (2) clarified that, when training the solver, recycled refinements are used under the question-only context without an off-policy correction (e.g., importance weighting for the change of context), following a similar practice in prior work [1, 2, 3, 4] (Sec. 3.4, App. C.7). All revisions have been integrated into our revised manuscript. Below, we address each specific question systematically.

Q1: The delta-accuracy refinement reward from the paper seems to be mathematically redundant under GRPO.

In Eq. (6), the authors define the refinement reward as:

r2(j)=1[correct(t2(j),q)]−1[correct(t1∗,q)]+λ1[format(t2(j))]r_2^{(j)} = \mathbb{1}[\mathrm{correct}(t_2^{(j)},q)] - \mathbb{1}[\mathrm{correct}(t_1^*,q)] + \lambda\mathbb{1}[\mathrm{format}(t_2^{(j)})]

However, all refinements in a group are conditioned on the same initial solution t1∗t_1^* (Eq. (4)). Define:

xj=1[correct(t2(j),q)]+λ1[format(t2(j))]c=1[correct(t1∗,q)]r2(j)=xj−c\begin{aligned} x_j &= \mathbb{1}[\mathrm{correct}(t_2^{(j)},q)] + \lambda\mathbb{1}[\mathrm{format}(t_2^{(j)})] \\\\[6pt] c &= \mathbb{1}[\mathrm{correct}(t_1^*,q)] \\\\[6pt] r_2^{(j)} &=x_j-c \end{aligned}

Since cc is constant across the refinement group, the group mean and standard deviation become:

μr=1G2∑j=1G2(xj−c) =μx−cσr=1G2∑j=1G2((xj−c)−(μx−c))2 =σx\begin{aligned} \mu_r &=\frac{1}{G_2}\sum_{j=1}^{G_2}(x_j-c)\\\ &=\mu_x-c \\\\[10pt] \sigma_r &=\sqrt{\frac{1}{G_2}\sum_{j=1}^{G_2} \left((x_j-c)-(\mu_x-c)\right)^2}\\\ &=\sigma_x \end{aligned}

Substituting into GRPO's group-relative advantage (Eq. (2)):

A2(j)=r2(j)−μrσr=(xj−c)−(μx−c)σx=xj−c−μx+cσx=xj−μxσx\begin{aligned} A_2^{(j)} &=\frac{r_2^{(j)}-\mu_r}{\sigma_r} \\\\[8pt] &=\frac{(x_j-c)-(\mu_x-c)}{\sigma_x} \\\\[8pt] &=\frac{x_j-\cancel{c}-\mu_x+\cancel{c}}{\sigma_x} \\\\[8pt] &=\boxed{\frac{x_j-\mu_x}{\sigma_x}} \end{aligned}

Therefore, the initial-solution correctness term cancels exactly. The resulting advantages are identical to those obtained using ordinary outcome correctness plus the format reward.

This means the proposed delta reward does not provide a distinct optimization signal or explicitly distinguish correction from corruption beyond what ordinary correctness already provides. Although the two-stage training procedure may still be effective, this challenges the paper's explanation of its reward mechanism.

A1: Thank you for this careful derivation. We would like to clarify that, since all refinements of a query revise the same solution t1∗t_1^*, the delta form and the outcome reward indeed give identical refinement-stage advantages, but these advantages still effectively distinguish good self-refinements from bad ones in the same way and thus provide an effective signal for training self-refinement. For example, when t1∗t_1^* is incorrect, a refinement that corrects it receives a higher advantage than one that leaves it incorrect; when t1∗t_1^* is correct, a refinement that keeps it correct receives a higher advantage than one that corrupts it. This is confirmed by our ablation: removing the refinement-stage training loses most of our gain over GRPO (Table 4):

DeepSeek-R1-Distill-Qwen-1.5BMATH-500AIME24AIME25AMC23MinervaOlympiadBRUMO25CMIMC25HMMT25Avg.
GRPO (JustRL-DeepSeek-1.5B; Table 1)91.852.538.391.351.368.051.725.622.554.8
SR-GRPO w/o Self-Refine Stage Training (Table 4)91.850.840.894.455.469.347.523.821.755.1
SR-GRPO (ours)94.361.755.895.660.372.662.533.134.263.3

We keep the delta form only because it is more intuitive to read, and we have added an explanation of this equivalence of the advantages under GRPO to Sec. 3.2, App. C.5 and the caption of Figure 1 of the revised paper, which makes the paper clearer; thank you for this suggestion.

AuthorsOpenReview ↗

Reply to Reviewer 8Wvs [Part 2]

Q2: Can the authors confirm whether replacing Eq. (6) with ordinary correctness plus the format reward produces identical refinement-stage gradients? If not, which implementation detail breaks this mathematical equivalence?

A2: Thank you for this direct question. We would like to clarify that, as discussed in A1, this replacement indeed gives identical advantages, and hence identical gradients. However, these advantages still distinguish good self-refinements of the same t1∗t_1^* from bad ones in the same way, which provides an effective improvement signal for self-refinement, as verified by our ablation (see A1).

Q3: In Eq. (9), verified refinement traces are sampled from (πθ(t2∣q,t1))(\pi_\theta(t_2 \mid q,t_1^)) but added to the solve-stage GRPO group. Under which context are their policy log-probabilities and importance ratios computed: (q)(q) or ((q,t1))((q,t_1^))? If computed under (q)(q), how is the mismatch between the sampling and optimization distributions handled?

A3: Thank you for this precise question. We would like to clarify that, when training the solver, they are computed under the question-only context qq: each verified refinement is appended to the solve group of the same query and trained exactly like the model's own solve rollouts.

That is, when training the solver, we treat solutions generated under an augmented prompt (i.e., self-refinement) as solutions to the original question without an off-policy correction (e.g., importance weighting for the change of context), an off-policy approximation widely used in prior work [1, 2, 3, 4]: STaR's rationalization trains on hint-conditioned rationales "as if the model had come up with the rationale without the hint" [1], and imitation learning from language feedback (ILF) [2], critique-in-the-loop self-improvement [3] and RL from text feedback (RLTF) [4] train in a similar way. Among them, RLTF [4] uses this approximation even though its augmented prompt contains feedback from an external model, and finds that "variance dominates bias": an importance-weighting correction increases variance and lowers performance. In contrast, our method relies on no external model, teacher or critique: the same model performs a single round of self-refinement. We therefore adopt this widely used approximation, which keeps our method simple and easy to use. It is also effective in our setting: removing the recycling of verified refinements into the solve group loses most of our gain over GRPO (Table 4). Moreover, a dedicated off-policy correction is not the main focus of this paper, and we leave it to future work.

Following your suggestion, Sec. 3.4 and App. C.7 of the revised paper now clearly state that the recycled refinements are off-policy, and that we use them to train the solver in the context of the original questions without an off-policy correction, following a similar practice in prior work [1, 2, 3, 4].


Thank you again for your valuable comments on the delta reward and on how recycled refinements are trained, which have greatly helped improve the clarity of our paper. Following your suggestions, the revised paper now clarifies that the delta reward, although it gives the same advantage as the outcome reward, still provides a meaningful learning signal for self-refinement, and that the recycled refinements are trained in the context of the original questions as an off-policy approximation, following similar practice in prior work. We hope our responses have fully addressed your concerns, and we sincerely thank you for your positive assessment of our work. Should any point require further clarification, we would welcome the opportunity to continue the discussion.

References

[1] Zelikman, et al. "STaR: Bootstrapping Reasoning With Reasoning." Advances in Neural Information Processing Systems 35, 2022.

[2] Scheurer, et al. "Training Language Models with Language Feedback at Scale." arXiv:2303.16755, 2023.

[3] Xi, et al. "MathCritique: Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision." IJCAI 2026.

[4] Song, et al. "Expanding the Capabilities of Reinforcement Learning via Text Feedback." ICML 2026.

AuthorsOpenReview ↗

Thank You for Your Careful Review and Insightful Suggestions

Dear Reviewer 8Wvs,

Thank you again for your valuable comments on the delta reward and on how recycled refinements are trained, which have greatly helped improve the clarity of our paper. Following your suggestions, the revised paper now clarifies that the delta reward, although it gives the same advantage as the outcome reward, still provides a meaningful learning signal for self-refinement, and that the recycled refinements are trained in the context of the original questions as an off-policy approximation, following similar practice in prior work. We hope these clarifications have fully addressed your concerns, and we sincerely thank you again for your positive assessment of our work. Thank you again for your time and effort in reviewing our paper.

Reviewer_3JjyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SR-GRPO trains a shared policy both to solve a problem and to refine one of its initial responses, prioritizing an incorrect response when available. The solving and refinement stages receive separate GRPO updates, and successful corrections are recycled into the solving group. Training uses verifiable answers, although the refinement prompt itself does not reveal the correct answer. Experiments on three backbones report improved direct-solving performance in mathematics and additional gains on scientific and general reasoning benchmarks, supported by component ablations and a larger-sampling GRPO control.

Critical strengths
  1. Across the three evaluated backbones, reported mathematical averages improve by 4.1–8.5 percentage points over GRPO. The headline evaluation does not use inference-time refinement, so the gains reflect the trained solver's direct-solving performance. This provides evidence that refinement-based training can benefit subsequent direct solving.

  2. The component ablations show lower performance when wrong-first sampling, refinement training, or feedback is removed. The 4B result also exceeds a 16-sample GRPO control, reaching 80.1 versus 76.2, so response count alone does not explain the benefit. These controls support the contribution of the training components, although they do not establish equal total compute.

Critical weaknesses
  1. Under the stated group normalization, the initial-correctness subtraction has no effect on the refinement advantages. All refinements in a group share the same initial response, so subtracting its correctness contributes a constant that disappears when group rewards are centered and normalized. Refinement training can still learn from correct and incorrect outcomes, but an additional optimization effect from the delta subtraction is not established by this formulation.

  2. The description of the feedback update leaves the cross-context optimization objective unclear. Recycled responses are sampled while conditioned on an initial solution, whereas the solving policy is conditioned on the problem alone; collecting them from a fresh policy does not remove this difference. The actual conditioning, likelihood ratios, and loss masks determine how this distribution change is handled. Without these details, the on-policy interpretation and reproducibility of the update remain unclear.

Questions for authors

See Weaknesses.

9 comments
AuthorsOpenReview ↗

Reply to Reviewer 3Jjy [Part 1]

We sincerely thank you for the careful and constructive review. We greatly appreciate your recognition that "the headline evaluation does not use inference-time refinement, so the gains reflect the trained solver's direct-solving performance," and that "the component ablations show lower performance when wrong-first sampling, refinement training, or feedback is removed."

We understand your concerns focus primarily on the delta reward under group normalization, the cross-context objective of the feedback update, and compute-matched baselines.

To address these thoroughly, we have (1) clarified that, even though the delta reward and the outcome reward give the same advantage, this advantage still distinguishes good self-refinements from bad ones in the same way, so self-refinement is trained effectively, as confirmed by our ablation (Sec. 3.2; Table 4, w/o Self-Refine Stage Training); (2) clarified that, for solver training, recycled refinements are treated as responses to the original question without an off-policy correction (e.g., importance weighting for the change of context), similar to prior work [1, 2, 3, 4], with their likelihood ratios and loss masks computed as in the GRPO baseline [5, 6] (Sec. 3.4, App. C.7); and (3) clarified that the G=16 control provides a rollout-matched baseline, and added two finer-grained controls that keep the base model, training data, number of training steps and total number of rollouts in the solve and refinement stages the same as ours, and vary how the refinement-stage rollouts are generated (resampling vs. self-refinement) and how they are used to strengthen solver training (SFT vs. GRPO) (App. E.4, E.3), summarized in Sec. 4.7 (Figure 3). All additions have been integrated into our revised manuscript. Below, we address each specific question systematically.

Q1: Under the stated group normalization, the initial-correctness subtraction has no effect on the refinement advantages. All refinements in a group share the same initial response, so subtracting its correctness contributes a constant that disappears when group rewards are centered and normalized. Refinement training can still learn from correct and incorrect outcomes, but an additional optimization effect from the delta subtraction is not established by this formulation.

A1: Thank you for this careful observation. We would like to clarify that, since all refinements of a query revise the same solution t₁*, the delta form and the outcome reward indeed give identical refinement-stage advantages, but, as you note, "refinement training can still learn from correct and incorrect outcomes": these advantages still effectively distinguish good self-refinements from bad ones in the same way and thus provide an effective signal for training self-refinement. For example, when t₁* is incorrect, a refinement that corrects it receives a higher advantage than one that leaves it incorrect; when t₁* is correct, a refinement that keeps it correct receives a higher advantage than one that corrupts it. This is confirmed by our ablation: removing the refinement-stage training loses most of our gain over GRPO (Table 4):

DeepSeek-R1-Distill-Qwen-1.5BMATH-500AIME24AIME25AMC23MinervaOlympiadBRUMO25CMIMC25HMMT25Avg.
GRPO (JustRL-DeepSeek-1.5B; Table 1)91.852.538.391.351.368.051.725.622.554.8
SR-GRPO w/o Self-Refine Stage Training (Table 4)91.850.840.894.455.469.347.523.821.755.1
SR-GRPO (ours)94.361.755.895.660.372.662.533.134.263.3

We keep the delta form only because it is more intuitive to read, and we have added an explanation of this equivalence of the advantages under GRPO to Sec. 3.2, App. C.5 and the caption of Figure 1 of the revised paper, which makes the paper clearer; thank you for this suggestion.

AuthorsOpenReview ↗

Reply to Reviewer 3Jjy [Part 2]

Q2: The description of the feedback update leaves the cross-context optimization objective unclear. Recycled responses are sampled while conditioned on an initial solution, whereas the solving policy is conditioned on the problem alone; collecting them from a fresh policy does not remove this difference. The actual conditioning, likelihood ratios, and loss masks determine how this distribution change is handled. Without these details, the on-policy interpretation and reproducibility of the update remain unclear.

A2: Thank you for this important question. We would like to clarify that, when training the solver, a verified refinement is treated as a response to the original question: its context is the question alone, and it is trained exactly like the model's own solve rollouts, with its likelihood ratio and loss mask computed in the same way (i.e., the same as the GRPO baseline [5, 6] for a fair comparison).

That is, when training the solver, we treat solutions generated under an augmented prompt (i.e., self-refinement) as solutions to the original question without an off-policy correction (e.g., importance weighting for the change of context), an off-policy approximation widely used in prior work [1, 2, 3, 4]: STaR's rationalization trains on hint-conditioned rationales "as if the model had come up with the rationale without the hint" [1], and imitation learning from language feedback (ILF) [2], critique-in-the-loop self-improvement [3] and RL from text feedback (RLTF) [4] train in a similar way. Among them, RLTF [4] uses this approximation even though its augmented prompt contains feedback from an external model, and finds that "variance dominates bias": an importance-weighting correction increases variance and lowers performance. In contrast, our method relies on no external model, teacher or critique: the same model performs a single round of self-refinement. We therefore adopt this widely used approximation, which keeps our method simple and easy to use. It is also effective in our setting: removing the recycling of verified refinements into the solve group loses most of our gain over GRPO (Table 4). Moreover, a dedicated off-policy correction is not the main focus of this paper, and we leave it to future work.

Following your suggestion, the revised paper now clearly states that these refinements are off-policy, and that we use them to train the solver in the context of the original questions without an off-policy correction, following a similar practice in prior work [1, 2, 3, 4] (Sec. 3.4, App. C.7).

AuthorsOpenReview ↗

Reply to Reviewer 3Jjy [Part 3]

Q3: The 4B result also exceeds a 16-sample GRPO control, reaching 80.1 versus 76.2, so response count alone does not explain the benefit. These controls support the contribution of the training components, although they do not establish equal total compute.

A3: Thank you for recognizing our G=16 control and that "response count alone does not explain the benefit," and thanks for noting that "they do not establish equal total compute." We would like to clarify that the G=16 control provides a rollout-matched baseline, but how these rollouts are generated and used is hard to unify across methods, so exactly equal total compute is hard to establish. To further address your concerns, we have added two finer-grained controls that keep the base model, training data, number of training steps and total number of rollouts in the solve and refinement stages (i.e., 8 each) the same as ours, and vary how the refinement-stage rollouts are generated and how they are used. For how they are generated, one control replaces self-refinement with resampling from the question alone and recycles the verified correct resamples under the same rule as ours (App. E.4, Table 19). For how they are used, the other control trains on the verified refinements with SFT instead of feeding them back into the solver's GRPO update (App. E.3, Table 18). Both stay far below our method, which is better on every benchmark:

DeepSeek-R1-Distill-Qwen-1.5BMATH-500AIME24AIME25AMC23MinervaOlympiadBRUMO25CMIMC25HMMT25Avg.
GRPO (JustRL-DeepSeek-1.5B; Table 1)91.852.538.391.351.368.051.725.622.554.8
Verified resampling (new; App. E.4)92.549.240.095.054.268.148.326.924.255.4
Verified refinements via SFT (new; App. E.3)92.652.543.392.557.467.555.826.323.356.8
SR-GRPO (ours)94.361.755.895.660.372.662.533.134.263.3

Together with our ablations (Table 4), each of which removes one component of our method, these controls show that the gain comes from jointly training self-refinement with the solver and feeding its verified refinements back to the solver through GRPO, not from more rollouts or from recycling verified responses alone.


Thank you again for your valuable comments on the delta reward, the cross-context objective of the feedback update and compute-matched baselines, which have greatly helped improve the clarity and rigor of our paper. Following your suggestions, the revised paper now clarifies the role of the delta reward and the training context of the recycled refinements, and adds two rollout-matched controls on how the refinement-stage rollouts are generated (resampling vs. self-refinement) and used (SFT vs. GRPO). We hope our responses have fully addressed your concerns; if so, we would greatly appreciate it if you would consider updating your assessment. Should any point require further clarification, we would welcome the opportunity to continue the discussion.

References

[1] Zelikman, et al. "STaR: Bootstrapping Reasoning With Reasoning." Advances in Neural Information Processing Systems 35, 2022.

[2] Scheurer, et al. "Training Language Models with Language Feedback at Scale." arXiv:2303.16755, 2023.

[3] Xi, et al. "MathCritique: Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision." IJCAI 2026.

[4] Song, et al. "Expanding the Capabilities of Reinforcement Learning via Text Feedback." ICML 2026.

[5] Sheng, et al. "HybridFlow: A Flexible and Efficient RLHF Framework." EuroSys 2025.

[6] He, et al. "JustRL: Scaling a 1.5B LLM with a Simple RL Recipe." arXiv:2512.16649, 2025.

Reviewer_kkUJOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SR-GRPO, which trains a single policy to produce an initial solution and then refine one of its attempts. An incorrect attempt is selected whenever available. Solver and refiner losses use separate group normalizations, and successful corrections are also added to the solver training group. The refinement prompt does not reveal the reference answer. The reported improvements cover three backbones at 1.5B and 4B, nine math benchmarks, and two general-reasoning benchmarks. The main results use the first-stage answer, without an additional refinement pass at evaluation.

Critical strengths
  1. The gains carry over to the first answer. I like that refinement is used to improve the deployed solver, rather than making the headline result depend on a second attempt at test time. The method also uses the existing answer verifier and a shared policy, without adding a separate critic or requiring correction annotations. This makes the proposed training loop practically useful for small reasoning models.
  2. The empirical improvements are substantial across the tested backbones. In Table 1, the nine-benchmark average increases from 54.8 to 63.3 on the DeepSeek backbone, from 64.5 to 71.4 on Nemotron, and from 76.0 to 80.1 on Qwen3-4B. The gains on GPQA Diamond and MMLU-Pro in Table 2 are also encouraging because training uses math data only. The effect is therefore not confined to the weakest starting model or one benchmark.
  3. The paper tests the main parts of the proposed recipe. Table 4 removes wrong-first sampling, refinement-stage training, and feedback separately; each removal loses most of the full method's gain. The G=16 control in Table 6 and the trained-versus-untrained refinement comparison in Table 7 are useful additions. These experiments give more support to the design than a comparison against vanilla GRPO alone, even though some confounds remain.
Critical weaknesses
  1. The feedback update is not specified precisely enough. A verified refinement is sampled with the initial answer and refinement instruction in its context, whereas an ordinary solver rollout is sampled from the question alone. Sections 3.4 and C.7 then add the former to the solver group and treat it like an ordinary correct rollout. If the initial answer is removed, the sampling distribution no longer matches the solver policy used in the loss; filtering for correctness changes the distribution further. Calling the samples fresh does not resolve either issue. If the initial answer is retained, the update remains conditioned on the refinement context, and its interpretation as question-only solver training needs clarification. I need the exact contexts, loss masks, and old-policy log-probabilities to understand what is being optimized. There is a related ambiguity in Eq. (3): it clips a full-sequence likelihood ratio, while standard GRPO clips token-level ratios before aggregating the losses. These objectives are not generally equivalent. A sequence-level surrogate is not inherently invalid, but the paper should state which objective was actually implemented, including length normalization and zero-variance group handling.
  2. The delta in Eq. (6) disappears from the stated refinement update. All responses in a refinement group share the same initial trace. Its correctness is therefore a group-wide constant. Subtracting that constant from every reward leaves both the centered rewards and their standard deviation unchanged. The resulting advantages are exactly those obtained from final-answer correctness plus the format bonus alone. Wrong-first sampling can still matter, but the claimed additional effect of the delta reward does not follow from this objective. This affects the explanation in Figure 1 and Section 3.2, rather than showing that refinement training itself cannot work.
  3. The closest alternative uses of self-generated feedback are not tested. Appendix A.1 discusses SDPO, but its scalar-reward-only setting is a relevant experimental baseline: it can use successful rollouts as feedback without an external critic. OPSD also deserves discussion, with its stronger reference conditioning clearly distinguished. A comparison using the same verified corrections for ordinary solver SFT would help establish whether the particular GRPO feedback update is needed. The novelty claim about jointly training a solver and refiner should also be narrower: Stage II of SCoRe already jointly optimizes both attempts. Recycling corrected trajectories as question-only solver data is the more specific distinction here.
  4. The G=16 control matches rollout count, but not the feedback training procedure. Table 6 is useful: vanilla GRPO with G=16 scores 76.2, compared with 80.1 for SR-GRPO. Both generate 16 responses per question. However, Eq. (9) adds m verified refinements to the solver group while retaining all eight refinements in the refinement loss. SR-GRPO therefore has 16+m trajectory entries across its two loss groups, with 0 <= m <= 8, compared with 16 for the control. These are reused responses, not additional independent samples; separate stage averaging also means this is not equivalent to GRPO with G=24. The selective reuse changes the solver group and its relative advantages, so the comparison does not isolate the benefit of refinement from that of reusing verified positive examples. Please report the feedback group sizes, generated and training token counts, and GPU-hours. A control using independent second-stage resamples with the same verified-positive recycling rule and stage weighting would make this distinction much clearer.
  5. The reported errors do not measure variation between RL training runs. Appendix D.5 does report standard errors, but these are across four evaluation seeds. They measure decoding variability for the evaluated checkpoints, not sensitivity to the training seed. They also do not provide confidence intervals for the paired improvement over the baseline. This matters for the smaller gains and the ablation conclusions. I would like to see independent training runs for the main comparison, together with uncertainty on the performance difference. The statement that the current results rule out seed-level variance is stronger than this evidence supports.
Questions for authors
  1. For a verified refinement reused in the solver group, what exactly are the input tokens, response mask, and denominator in the importance ratio? Is the intended objective an auxiliary off-policy imitation loss or standard on-policy GRPO? Please also provide the token-level objective used in the implementation.
  2. Given that the initial-answer reward is constant within each refinement group, is there any implementation detail that prevents its cancellation in Eq. (2)? If not, will the interpretation of the delta reward be revised?
  3. Can you provide a matched-backbone comparison with SDPO in its scalar-reward-only setting, or a controlled comparison using the same verified refinements for solver SFT? These would directly test the benefit attributed to the feedback update.
13 comments
AuthorsOpenReview ↗

Reply to Reviewer kkUJ [Part 1]

We sincerely thank you for the careful and constructive review. We greatly appreciate your recognition that "refinement is used to improve the deployed solver, rather than making the headline result depend on a second attempt at test time," that the "empirical improvements are substantial across the tested backbones," and that the "G=16 control in Table 6 and the trained-versus-untrained refinement comparison in Table 7 are useful additions."

We understand your concerns focus primarily on the G=16 control, comparisons with self-generated-feedback baselines, the statistical significance of the gains across training runs, the exact feedback objective, and the delta reward.

To address these thoroughly, we have (1) run the verified-resampling control you suggested, with the same base model, training data and number of training steps as ours, which shows that the gain comes from trained self-refinement rather than from reusing verified positive examples (App. E.4); (2) trained SDPO [1] and run the SFT control under the same setting, both of which stay below our method on every benchmark (App. E.1, E.3); (3) added the experiments with multiple training seeds, showing that our gain is now significant across both training and evaluation seeds (App. E.2); (4) clarified the context, loss mask and importance ratio of recycled refinements, and, to avoid confusion, expanded the shorthand Eq. (3) to token-level clipping, as in the GRPO baseline [2, 3] (Sec. 3); (5) clarified that, even though the delta reward and the outcome reward give the same advantage, this advantage still distinguishes good self-refinements from bad ones in the same way, so self-refinement is trained effectively, as confirmed by our ablation (Sec. 3.2; Table 4, w/o Self-Refine Stage Training); and (6) clarified the distinction from SCoRe and our novelty claim (Sec. 1, Sec. 2). The new results are reported in App. E and summarized in Sec. 4.7 (Figure 3). All additions have been integrated into our revised manuscript. Below, we address each specific question systematically.

AuthorsOpenReview ↗

Reply to Reviewer kkUJ [Part 2]

Q1: The feedback update is not specified precisely enough. A verified refinement is sampled with the initial answer and refinement instruction in its context, whereas an ordinary solver rollout is sampled from the question alone. Sections 3.4 and C.7 then add the former to the solver group and treat it like an ordinary correct rollout. If the initial answer is removed, the sampling distribution no longer matches the solver policy used in the loss; filtering for correctness changes the distribution further. Calling the samples fresh does not resolve either issue. If the initial answer is retained, the update remains conditioned on the refinement context, and its interpretation as question-only solver training needs clarification. I need the exact contexts, loss masks, and old-policy log-probabilities to understand what is being optimized.

There is a related ambiguity in Eq. (3): it clips a full-sequence likelihood ratio, while standard GRPO clips token-level ratios before aggregating the losses. These objectives are not generally equivalent. A sequence-level surrogate is not inherently invalid, but the paper should state which objective was actually implemented, including length normalization and zero-variance group handling.

A1: Thank you for this careful question. We would like to clarify that, when training the solver, a verified refinement t2t_2 is treated as a response to the original question qq and is trained exactly like the model's own solve rollouts (i.e., without an off-policy correction such as importance weighting for the change of context, following a similar practice in prior work [4, 5, 6, 7]). Accordingly, its context is qq alone, and its loss mask and per-token clipped ratio are the same as for the other solve rollouts (i.e., the same as the GRPO baseline [2, 3] for a fair comparison). Length normalization and zero-variance handling are not the main focus of this work, so we also keep them the same as in the GRPO baseline [2, 3]. The sequence-level ratio in Eq. (3) of the submission was only a shorthand; we have expanded it to its token-level form and stated these details explicitly in the revised paper (Sec. 3.1), and thank you for pointing this out.

That is, when training the solver, we treat solutions generated under an augmented prompt (i.e., self-refinement) as solutions to the original question without an off-policy correction (e.g., importance weighting for the change of context), an off-policy approximation widely used in prior work [4, 5, 6, 7]: STaR's rationalization trains on hint-conditioned rationales "as if the model had come up with the rationale without the hint" [4], and imitation learning from language feedback (ILF) [5], critique-in-the-loop self-improvement [6] and RL from text feedback (RLTF) [7] train in a similar way. Among them, RLTF [7] uses this approximation even though its augmented prompt contains feedback from an external model, and finds that "variance dominates bias": an importance-weighting correction increases variance and lowers performance. In contrast, our method relies on no external model, teacher or critique: the same model performs a single round of self-refinement. We therefore adopt this widely used approximation, which keeps our method simple and easy to use. It is also effective in our setting: removing the recycling of verified refinements into the solve group loses most of our gain over GRPO (Table 4). Moreover, a dedicated off-policy correction is not the main focus of this paper, and we leave it to future work.

Following your suggestion, Sec. 3.4 and App. C.7 of the revised paper now clearly state that the recycled refinements are off-policy, and that we use them to train the solver in the context of the original questions without an off-policy correction, following a similar practice in prior work [4, 5, 6, 7].

AuthorsOpenReview ↗

Reply to Reviewer kkUJ [Part 3]

Q2: The delta in Eq. (6) disappears from the stated refinement update. All responses in a refinement group share the same initial trace. Its correctness is therefore a group-wide constant. Subtracting that constant from every reward leaves both the centered rewards and their standard deviation unchanged. The resulting advantages are exactly those obtained from final-answer correctness plus the format bonus alone. Wrong-first sampling can still matter, but the claimed additional effect of the delta reward does not follow from this objective. This affects the explanation in Figure 1 and Section 3.2, rather than showing that refinement training itself cannot work.

A2: Thank you for this careful observation. We would like to clarify that, since all refinements of a query revise the same solution t1∗t_1^*, the delta form and the outcome reward indeed give identical refinement-stage advantages, but these advantages still effectively distinguish good self-refinements from bad ones in the same way and thus provide an effective signal for training self-refinement. For example, when t1∗t_1^* is incorrect, a refinement that corrects it receives a higher advantage than one that leaves it incorrect; when t1∗t_1^* is correct, a refinement that keeps it correct receives a higher advantage than one that corrupts it. This is confirmed by our ablation: removing the refinement-stage training loses most of our gain over GRPO (Table 4):

DeepSeek-R1-Distill-Qwen-1.5BMATH-500AIME24AIME25AMC23MinervaOlympiadBRUMO25CMIMC25HMMT25Avg.
GRPO (JustRL-DeepSeek-1.5B; Table 1)91.852.538.391.351.368.051.725.622.554.8
SR-GRPO w/o Self-Refine Stage Training (Table 4)91.850.840.894.455.469.347.523.821.755.1
SR-GRPO (ours)94.361.755.895.660.372.662.533.134.263.3

We keep the delta form only because it is more intuitive to read, and we have added an explanation of this equivalence of the advantages under GRPO to Sec. 3.2, App. C.5 and the caption of Figure 1 of the revised paper, which makes the paper clearer; thank you for this suggestion.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response [Part 1]

Dear Area Chairs and Reviewers,

Thank you for your time and effort in reviewing our work. Below is our global response:

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are particularly encouraged by the recognition of our substantial empirical improvements across all tested backbones, including on out-of-distribution benchmarks; of the fact that these gains come from the trained solver's direct solving and require no additional inference-time computation; of our component ablations and G=16 control, which support the design; and of our training loop as practically useful for small reasoning models, using the existing answer verifier and a shared policy without a separate critic. We greatly appreciate the time all reviewers invested in providing constructive suggestions that have significantly strengthened our work.

After careful consideration of the reviewers' comments, we identified the main concerns focusing on: (1) the off-policy nature of the recycled refinements (Reviewers 8Wvs, 3Jjy, kkUJ); (2) the role of the delta reward, which gives the same advantage as the verifiable outcome (correctness) reward under group normalization, in training self-refinement (Reviewers 8Wvs, 3Jjy, kkUJ); and (3) more baselines and controls (Reviewers 3Jjy, kkUJ).

We have comprehensively addressed these concerns through clarifications and extensive additional experiments in our revision. The new results are reported in a new Appendix E and summarized in a new Sec. 4.7 (Figure 3). Below, we summarize the clarifications and additional experiments contributed during the rebuttal to address the reviewers' concerns:

Off-Policy Approximation for Recycled Refinements (Sec. 3.4, App. C.7): To address concerns about the recycled refinements, we clarified that, when training the solver, a verified refinement is added directly to the solve group as an extra solution to the original question and is trained like any other solve rollout of the GRPO baseline, with no special design. Since it was generated under the self-refinement prompt, training on it in this way without an off-policy correction (e.g., importance weighting for the change of context) is an off-policy approximation widely used in prior work [1, 2, 3, 4] (App. C.7). In particular, RLTF [4] uses this approximation even though its augmented prompt contains feedback from an external model, and finds that an importance-weighting correction increases variance and lowers performance. In contrast, our method relies on no external model, teacher or critique. We therefore adopt this simple approximation, which is also effective in our setting: removing the recycling loses most of our gain over GRPO (Table 4). A dedicated off-policy correction is not the main focus of this paper, and we leave it to future work.

To address the concern about the recycled refinements, we compare our method with the ablation that removes their recycling into the solve group:

DeepSeek-R1-Distill-Qwen-1.5BMATH-500AIME24AIME25AMC23MinervaOlympiadBRUMO25CMIMC25HMMT25Avg.
GRPO (JustRL-DeepSeek-1.5B)91.852.538.391.351.368.051.725.622.554.8
SR-GRPO w/o Self-Distilling Feedback92.351.741.793.156.068.454.225.622.556.2
SR-GRPO (ours)94.361.755.895.660.372.662.533.134.263.3
AuthorsOpenReview ↗

Global Response [Part 2]

Delta Reward under Group Normalization (Sec. 3.2, App. C.5, Figure 1): To address concerns about the delta reward, we clarified that, since all refinements of a query revise the same initial solution, the delta reward and the outcome reward give the same advantage, but this advantage still distinguishes good self-refinements from bad ones in the same way: for the same initial solution, fixing an incorrect solution receives a higher advantage than leaving it incorrect, and keeping a correct solution correct receives a higher advantage than corrupting it. Self-refinement is therefore trained effectively with the verifiable reward alone, as confirmed by our ablation (Table 4, w/o Self-Refine Stage Training), and we have revised Sec. 3.2, App. C.5 and the caption of Figure 1 accordingly.

To address the concern about the delta reward, we compare our method with the ablation that removes the refinement-stage training:

DeepSeek-R1-Distill-Qwen-1.5BMATH-500AIME24AIME25AMC23MinervaOlympiadBRUMO25CMIMC25HMMT25Avg.
GRPO (JustRL-DeepSeek-1.5B)91.852.538.391.351.368.051.725.622.554.8
SR-GRPO w/o Self-Refine Stage Training91.850.840.894.455.469.347.523.821.755.1
SR-GRPO (ours)94.361.755.895.660.372.662.533.134.263.3

Extensive Additional Baselines and Controls (Sec. 4.7, App. D.1, App. E): To address concerns about baselines and controls, we note that the closest published RL baseline, Critique-GRPO [5], is already compared in the submission (App. D.1): even starting from a smaller and weaker base model (DeepSeek-R1-Distill-Qwen-1.5B vs. Qwen3-8B), our method surpasses it on average over the shared benchmarks. The additional experiments, all with the same base model, training data and number of training steps as ours, are: SDPO [6] in its scalar-reward-only setting (App. E.1); two additional independent runs of our method, with significance tests across both training and evaluation seeds (App. E.2); an SFT control that trains on the same verified refinements (App. E.3); and a verified-resampling control that keeps the total number of rollouts the same but replaces self-refinement with resampling from the question alone (App. E.4). Our method outperforms each new baseline and control on every benchmark, and its gain over GRPO is significant (Sec. 4.7, Figure 3).

To address the concern about baselines with self-generated feedback, we train SDPO in its scalar-reward-only setting from the same base model:

DeepSeek-R1-Distill-Qwen-1.5BMATH-500AIME24AIME25AMC23MinervaOlympiadBRUMO25CMIMC25HMMT25Avg.
Backbone85.630.022.566.936.146.131.711.914.238.3
SDPO, scalar-reward-only (new)90.346.728.386.947.661.644.218.819.249.3
SR-GRPO (ours)94.361.755.895.660.372.662.533.134.263.3

To address the concern about rollout-matched controls, we vary how the refinement-stage rollouts are generated (verified resampling) and how the verified refinements are used (SFT):

DeepSeek-R1-Distill-Qwen-1.5BMATH-500AIME24AIME25AMC23MinervaOlympiadBRUMO25CMIMC25HMMT25Avg.
GRPO (JustRL-DeepSeek-1.5B)91.852.538.391.351.368.051.725.622.554.8
Verified resampling (new)92.549.240.095.054.268.148.326.924.255.4
Verified refinements via SFT (new)92.652.543.392.557.467.555.826.323.356.8
SR-GRPO (ours)94.361.755.895.660.372.662.533.134.263.3
AuthorsOpenReview ↗

Global Response [Part 3]

To address the concern about variation across training runs, we train our method with two additional training seeds:

DeepSeek-R1-Distill-Qwen-1.5BMATH-500AIME24AIME25AMC23MinervaOlympiadBRUMO25CMIMC25HMMT25Avg.
SR-GRPO (ours), three training runs94.4 ± 0.261.9 ± 0.756.4 ± 0.695.0 ± 0.461.0 ± 0.471.9 ± 0.562.5 ± 0.533.3 ± 1.334.4 ± 0.763.4 ± 0.3
Gain over GRPO+2.6+9.4+18.1+3.7+9.7+3.9+10.8+7.7+11.9+8.6 (95% CI 7.3 to 10.0)

Mean ± standard error across three independent training runs, each evaluated with pass@1 averaged over four evaluation seeds; the gain is the difference of the printed means from the GRPO row of Table 1, and its CI is a t-interval over the unrounded averages of the three runs (df = 2).

To address the concern about statistical significance, we also test the gain over evaluation seeds on all three backbones:

BackboneGRPO Avg.SR-GRPO Avg.Gain95% CI of gainp
DeepSeek-R1-Distill-Qwen-1.5B54.863.3+8.56.4 to 10.5<0.0001
OpenMath-Nemotron-1.5B64.571.4+6.94.9 to 8.6<0.001
Qwen3-4B-Thinking-250776.080.1+4.12.1 to 5.90.002

These clarifications and extensive additional experiments, together with the existing results on three backbones across nine math benchmarks and two out-of-distribution benchmarks, as well as the component ablations and the G=16 control, sufficiently demonstrate the effectiveness of SR-GRPO. We are grateful for the reviewers' detailed feedback that helped us substantially enhance the paper's rigor and clarity. We sincerely thank all reviewers and area chairs for their time and effort in reviewing our submission.

References

[1] Zelikman, et al. "STaR: Bootstrapping Reasoning With Reasoning." Advances in Neural Information Processing Systems 35, 2022.

[2] Scheurer, et al. "Training Language Models with Language Feedback at Scale." arXiv:2303.16755, 2023.

[3] Xi, et al. "MathCritique: Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision." IJCAI 2026.

[4] Song, et al. "Expanding the Capabilities of Reinforcement Learning via Text Feedback." ICML 2026.

[5] Zhang, et al. "Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback." arXiv:2506.03106, 2025.

[6] Hübotter, et al. "Reinforcement Learning via Self-Distillation." arXiv:2601.20802, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.