acceptodds
Under review as a conference paper at ICLR 2027

Incorrect Answers Can Be Reinforced More: Relative-Reinforcement Policy Optimization for Learning from Verifiable Rewards

Abstract

Group relative policy optimization (GRPO) trains reasoning models from verifiable rewards by favoring correct responses within each sampled group. However, we observe that this local preference does not always carry over to what training accumulates: measured against the initial policy, the incorrect responses to a question can be reinforced more than its correct ones. We call this phenomenon reinforcement inversion and attribute it to the local nature of GRPO's signal, which ranks responses within the current group but does not constrain how accumulated change is divided between correct and incorrect responses. To address this, we propose Relative-Reinforcement Policy Optimization (\rpo), which adds a group-level loss on the reinforcement gap between the two classes, weighted by class balance and shaped by a logistic function. \rpo reuses GRPO's rollouts, verifier labels and reference log-probabilities, requiring no additional sampling. Experiments on mathematical and multimodal reasoning tasks across two backbones show that \rpo consistently improves over GRPO and outperforms alternative policy optimization methods across models and datasets.

Primary area:
reinforcement learning
Keywords:
GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vFg4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper reveals that GRPO sometimes reinforces incorrect responses more than correct ones, even though its advantages favor correct answers within each sampled group. The authors measure reinforcement by the change in mean token log-probability from the initial policy and call this effect reinforcement inversion. They propose R$^2$PO, which adds a logistic loss on the gap between the correct and incorrect class means, with groups weighted by class balance. The loss uses the rollouts and reference probabilities already available in GRPO. Experiments compare R$^2$PO with GRPO and seven variants on mathematical and multimodal reasoning tasks using two 4B Qwen models. The authors report gains over GRPO with little extra training time and use ablations to examine the design choices.

Critical strengths
  1. The method is cheap to add to GRPO. R$^2$PO uses the rollouts and reference log-probabilities already available during training. The update is described in Algorithm 1 and Appendix B.1, and Table 4 reports an increase of only 0.12%–3.55% in median time per update. The low cost makes the reported gains over GRPO practically interesting.

  2. The gains appear across different task settings. Table 1 reports pass@11 improvements of 1.18-3.34 percentage points over GRPO across all six model and task settings. It is encouraging to see this pattern on both mathematics and multimodal reasoning. Table 2 also tests the balance weight, logistic loss, and initial-policy reference separately; removing any of them lowers performance relative to the full method on all three tasks.

Critical weaknesses
  1. AMIR-GRPO [1] is particularly relevant to the algorithmic idea used in this paper. It also adds a logistic preference loss to GRPO using pairs from the same rollout groups and log-probability ratios (normalized by length) against a reference policy. The paper does not discuss or compare with it. I notice that R$^2$PO applies logistic loss to the mean gap in Eq. (5), while a pairwise objective averages the individual logistic losses. The benefit of the former approach is not tested. A comparison with AMIR-GRPO, or with GRPO plus a pairwise logistic loss, would help establish what the group aggregation contributes.

  2. I also find some inconsistency of numerical results in the tables. Let's focus on results of R$^2$PO with Qwen3.5-4B model on Geometry3K dataset. From Table 1, pass@11, pass@44, pass@88 are 77.67, 84.15, and 89.00 respectively, and the author mentioned τ\tau is around 10−310^{-3}; from Table 3, τ=10−3\tau=10^{-3}, pass@11, pass@44, pass@88 are 77.67, 83.42, and 89.00; from Table 3, τ=3×10−3\tau=3\times 10^{-3}, pass@11, pass@44, pass@88 are 75.67, 84.15, and 87.00. This may be a typo or reflect different runs. Either way, it needs clarification: if 83.42 is the pass@4 result for the main setting, it is below CISPO’s 83.79, which affects the claim in Section 5.2 that R$^2$PO is best on pass@44 in every setting.


References

[1] Amir Hossein Yari and Fajri Koto. AMIR-GRPO: Inducing Implicit Preference Signals into GRPO.

Questions for authors
  1. Which exact τ\tau and checkpoint were used for the Geometry3K row in Table 1? Please explain the difference from Table 3 and confirm whether all three metrics in each row were measured from the same selected model. What metrics are used to choose hyperparameters τ\tau?

  2. Could you compare R$^2$PO with AMIR-GRPO or the pairwise logistic loss described above? Reporting variation across training seeds for the main comparisons would also help assess the reliability of the gains.

  3. Table 2 supports the usefulness of the reference-dependent weighting, and Figure 3 shows that R$^2$PO reduces inversion. However, a negative gap in Eq. (3) can coexist with an increase in the total probability of correct answers. Did the authors examine whether questions whose inversion decreases also improve in accuracy? Measuring the gaps on a common response set and evaluating accuracy independently on the same questions would strengthen this connection.

7 comments

We thank the reviewer for the careful review and the concrete suggestions, and we are glad that the reviewer finds R²PO cheap to add to GRPO, the gains across mathematical and multimodal reasoning encouraging, and the separate tests of the balance weight, the logistic loss and the initial-policy reference informative. Below we address each weakness (W) and question (Q). The new experiments use the configurations of the paper and will be added to the revision.

W1: Relation to AMIR-GRPO and the role of group aggregation

1. The connection. We thank the reviewer for pointing us to AMIR-GRPO [1], which we will cite and discuss in Section 2. Both methods add a logistic preference loss to GRPO, computed from the rollout groups GRPO already produces, using length-normalized log-ratios against a reference policy. They differ in what the loss is applied to. Writing mij=ui+−uj−m_{ij}=u_{i}^{+}-u_{j}^{-} for a correct--incorrect pair and ℓτ(z)=τlog⁡(1+e−z/τ)\ell_{\tau}(z)=\tau\log(1+e^{-z/\tau}):

pairwise: 1n+n−∑i,jℓτ(mij),R2PO: ℓτ(1n+n−∑i,jmij)=ℓτ(δg).\text{pairwise: }\frac{1}{n_+n_-}\sum_{i,j}\ell_\tau(m_{ij}),\qquad \text{R}^2\text{PO: }\ell_\tau\Big(\frac{1}{n_+n_-}\sum_{i,j}m_{ij}\Big)=\ell_\tau(\delta_g).

Since the pair margins average to δg\delta_g, the two forms differ only in whether the logistic is applied before or after averaging.

2. What the class-mean form provides. Two properties follow from this aggregation:

  • The objective acts on the quantity the paper measures. δg\delta_g of Eq. (3) is the diagnostic that defines reinforcement inversion, so the loss optimizes exactly the quantity whose behaviour Section 5.4 reports.
  • The class-balance weight is derived rather than chosen. The covariance between correctness and relative reinforcement factors as Covg(c,u)=pg(1−pg)δg\mathrm{Cov}_g(c,u)=p_g(1-p_g)\delta_g (Eq. 4), which yields aga_g. Table 2 shows that removing this weight lowers pass@1 on all three tasks.

Because ℓτ\ell_\tau is convex, Jensen's inequality gives ℓτ(δg)≤1n+n−∑i,jℓτ(mij)\ell_\tau(\delta_g)\le\frac{1}{n_+n_-}\sum_{i,j}\ell_\tau(m_{ij}), with equality only when all pairs have the same margin. The group form therefore responds to whether the two classes are ordered on average, while the pairwise form keeps a large loss for every misordered pair inside an otherwise well-ordered group.

3. The requested comparison. We implemented the pairwise objective in our codebase with the same rollouts, initial-policy reference, masks, class-balance weights and training budget, so that the aggregation is the only difference. On M3CoT (1,108 questions, greedy decoding):

MethodSeed 27182Seed 31415Seed 42
GRPO924924924
R²PO (class-mean gap)943940934
Pairwise logistic931926932

Both forms improve on GRPO on all three seeds. R²PO is higher than the pairwise objective on every seed, with mean accuracy of 84.75% versus 83.90%.

W2: Inconsistent Geometry3K entries and the pass@k ranking

1. It is a typo: we swapped two entries when filling Table 3. We thank the reviewer for the careful reading. When transcribing the τ\tau sweep into Table 3, we inadvertently swapped the pass@4 entries of τ=10−3\tau=10^{-3} and τ=3×10−3\tau=3\times10^{-3}. All other entries are correct:

τ\taupass@1pass@4 (submitted → correct)pass@8
10−310^{-3}77.6783.42 → 84.1589.00
3×10−33\times10^{-3}75.6784.15 → 83.4287.00

The two rows are not different runs of the main setting: Table 1 reports the τ=10−3\tau=10^{-3} run, and with the correct values its row reads 77.67 / 84.15 / 89.00, identical to Table 1. We apologize for the error and will correct Table 3 in the revision.

2. The Section 5.2 claim is unaffected. R²PO's pass@4 on Geometry3K is 84.15, which is above CISPO's 83.79, so R²PO remains best on pass@4 and pass@8 in all six settings.

3. The corrected sweep is consistent with the rest of Table 3. With the correct values, the τ=10−3\tau=10^{-3} run is ahead of the τ=3×10−3\tau=3\times10^{-3} run on all three metrics, and all three peak at τ=10−3\tau=10^{-3}. This matches the role of τ\tau described in Sec. 5.5: the gaps are of order 10−310^{-3} (Fig. 1), which is the scale at which the gap weight fg=σ(−δg/τ)f_g=\sigma(-\delta_g/\tau) separates inverted from correctly ordered groups.

Q1: Checkpoint, τ\tau and hyperparameter selection

1. τ\tau and checkpoint for the Table 1 row. The Geometry3K row of Table 1 is the τ=10−3\tau=10^{-3} run, which is also the τ=10−3\tau=10^{-3} row of Table 3. The checkpoint is the final one after one epoch, that is 525 optimizer updates. No checkpoint is selected during or after training: every method in Table 1 is evaluated at its own final checkpoint under the same rule.

2. The difference from Table 3. It is a transcription error on our side: the pass@4 entries of τ=10−3\tau=10^{-3} and τ=3×10−3\tau=3\times10^{-3} were swapped when Table 3 was filled. The correct τ=10−3\tau=10^{-3} row is 77.67 / 84.15 / 89.00, identical to Table 1, and the correct τ=3×10−3\tau=3\times10^{-3} row is 75.67 / 83.42 / 87.00. We will correct Table 3 in the revision.

3. The three metrics come from the same model. All three are computed from that single final checkpoint. They differ only in decoding: pass@1 uses one greedy response per question, while pass@4 and pass@8 apply the unbiased estimator of Chen et al. (2021) to the same eight sampled responses per question. The greedy and the sampled metrics therefore use different response sets by construction, but the same model.

4. How τ\tau is chosen. τ\tau is selected by pass@1 on each task's evaluation split, the same protocol and the same metric used for the hyperparameters of every baseline in Table 1. The value is set on the scale of the observed reinforcement gaps, which are of order 10−310^{-3} (Fig. 1), so that the gap weight fg=σ(−δg/τ)f_g=\sigma(-\delta_g/\tau) varies across groups rather than saturating; Table 3 reports the sweep behind this choice, and Sec. 5.5 discusses it.

Q2: Pairwise comparison and variation across training seeds

1. The pairwise comparison. We report it under W1: with the same rollouts, reference, masks, class-balance weights and training budget, and only the aggregation changed, both the class-mean and the pairwise form improve on GRPO, and R²PO is higher than the pairwise form on all three training seeds.

2. Variation across training seeds. The submitted results are single runs per setting. We have now repeated the M3CoT comparison with two additional training seeds, keeping the selected configuration and re-tuning nothing (greedy correct answers out of 1,108):

MethodSeed 27182 (Table 1)Seed 31415Seed 42Mean ± s.d. (%)
GRPO92492492483.39 ± 0.00
R²PO94394093484.75 ± 0.41

R²PO is above GRPO on each of the three seeds, by 19, 16 and 10 questions. Its mean accuracy is 84.75%, compared with 83.39% for GRPO, a mean gain of 1.35 percentage points.

3. What the spread shows. The gain varies across seeds while its direction does not. The spread is informative about the size of run-to-run variation on this task: GRPO happens to reach the same total on all three seeds, but the three runs differ on 94 questions between seeds 31415 and 42, so the agreement in totals is a coincidence rather than a sign of low variance. We will report the three seeds and their mean and standard deviation in the revision, and we will state that the submitted tables are single runs.

Reviewer_XR2mOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes R2PO, which augments the GRPO objective with a class-balance-weighted logistic loss, expecting greater average reinforcement of correct responses than incorrect ones to the same question. Experiments across two backbones and three reasoning tasks report improved accuracy over GRPO and fewer groups exhibiting reinforcement inversion.

Critical strengths

1 The paper analyzes reinforcement inversion in GRPO, showing that the local preference for correct responses does not always translate into greater accumulated reinforcement relative to the initial policy.

2 The observation directly motivates the proposed loss on the reinforcement gap, providing a clear connection between the diagnosed issue and the method’s design.

3 R2PO reuses quantities already computed by GRPO, adding only 0.12–3.55% to training time per update and no extra inference cost.

Critical weaknesses

1 The explanation for reinforcement inversion needs more support. The paper suggests that inversion comes from GRPO’s local advantages and the shared structure of correct and incorrect responses. However, that doesn’t directly explain why reinforcement inversion happens.

2 This study seems to be a minor modification of GRPO, which limits the further impact of RL research for LLM society.

Questions for authors

1 For Qwen3.5-4B on Geometry3K, Table 1 reports 77.67/84.15/89.00, while Table 3 reports 77.67/83.42/89.00 at τ = 1e-3. The value 84.15 appears at τ = 3e-3, with different pass@1 and pass@8 values. Could you explain this discrepancy and confirm whether the three metrics in Table 1 come from the same checkpoint and hyperparameter configuration?

2 Since reinforcement inversion is measured using class averages within sampled response groups, the result may depend on which responses are sampled and how often they appear. Were GRPO and R2PO evaluated on exactly the same fixed response groups in Figure 3(a)? How stable is the reported reduction in inversion across different sampling seeds or group sizes?

3 Could this study still work for other RL loss in GRPO family? Such as GSPO,DAPO, DrGRPO...

4 comments

We thank the reviewer for the constructive review, and we are glad that the reviewer finds the analysis of reinforcement inversion informative, the connection between the diagnosed issue and the method's design clear, and the method inexpensive, reusing quantities already computed by GRPO with 0.12–3.55% additional time per update and no inference cost. Below we address each weakness (W) and question (Q). The new experiments use the configurations of the paper and will be added to the revision.

W1: Why does reinforcement inversion happen?

1. A question receives GRPO's preference at a single update. With one epoch and four questions per update, each question is sampled in exactly one of the 525 updates on Geometry3K. GRPO's advantages act on that question's responses only at this update. The other 524 updates change the log-probabilities of its responses through the shared parameters, and the objective places no constraint on how. Since the relative reinforcement ugiu_{gi} of Eq. (2) accumulates all of these changes, a single local preference does not determine the accumulated one.

2. Where the opposing contribution comes from. Consider binary rewards, on-policy updates without clipping or KL, and token-mean normalization. GRPO's gradient for group hh is then kh∇δhk_h\nabla\delta_h with kh=nhph(1−ph)/sh>0k_h=n_hp_h(1-p_h)/s_h>0, the proportionality noted in Sec. 4.4. To first order in the learning rate η\eta, one update changes the gap of group gg by

Δδg=ηkg∥∇δg∥21[g∈batch]+η∑h∈batch,h≠gkh⟨∇δg,∇δh⟩.\Delta\delta_g = \eta k_g \lVert\nabla\delta_g\rVert^2 \mathbb{1}[g\in\text{batch}] + \eta\sum_{h\in\text{batch}, h\ne g} k_h \langle\nabla\delta_g,\nabla\delta_h\rangle.

The first term is nonnegative but occurs at a single update. The cross terms occur at every update and can have either sign: they are positive when the gap directions of two questions agree and negative when they conflict. A group ends up inverted when these cross terms accumulate negatively and outweigh the first term, even though GRPO favoured its correct responses whenever it saw them.

3. A minimal example. Take two questions, each with one correct and one incorrect one-token response, with correct-answer probabilities σ(θ)\sigma(\theta) and σ(−2θ)\sigma(-2\theta). At θ=0\theta=0 the GRPO gradient is +1+1 from the first question and −2-2 from the second, so a joint update decreases θ\theta. The first question's correct response then loses probability relative to the initial policy while its incorrect response gains, and its gap becomes negative although its own advantage favoured the correct response.

4. Why this motivates the method. At the update where a question is sampled, GRPO's advantage depends only on correctness, so it carries no information about whether earlier updates have already inverted that question. The gap δg(θold)\delta_g(\theta_{\text{old}}) measures exactly this accumulated state, which is why R²PO conditions the group weight on it. We will add this explanation to Sections 1 and 4.

W2: Scope and impact of the modification

1. The paper's first contribution is a finding, not a modification. Before proposing anything, the paper reports that GRPO training produces an outcome its objective does not control: measured against the initial policy, about one third of the questions with both correct and incorrect responses end up with their incorrect responses reinforced more (Figure 1). This is not visible in GRPO's advantages, which are recomputed from fresh responses at every step and carry no information about accumulated change (Sec. 3). Section 5.4 and Q2 show that the effect is measurable on shared responses, so it can be checked by anyone training with GRPO, independently of our method.

2. The method follows from the finding rather than from tuning. The quantity to measure, the reinforcement gap, is defined in Eq. (3); the weighting is not chosen by search but derived, since the covariance between correctness and relative reinforcement factors as Covg(c,u)=pg(1−pg)δg\mathrm{Cov}_g(c,u)=p_g(1-p_g)\delta_g (Eq. 4), which yields the class-balance weight. Table 2 shows that removing the balance weight or the initial-policy reference lowers pass@1 on all three tasks.

3. Being a small change is what makes it usable. R²PO leaves GRPO's per-step update unchanged and only reweights how much each group contributes, so it needs no additional sampling, adds 0.12--3.55% to the time per update, and adds no inference cost. It can be dropped into any trainer that provides response groups, verifier labels and reference log-probabilities. In Table 1 it improves on GRPO in all six model--task settings and on every metric, and the gain replicates on two further M3CoT training seeds, with mean accuracy across all three seeds of 84.75% for R²PO versus 83.39% for GRPO, whereas none of the seven alternatives we compare against improves pass@1 on all three tasks with Qwen3.5-4B.

4. The general point. The paper's wider suggestion is methodological: RLVR objectives are normally evaluated by what each update rewards, while the quantity that training accumulates is left unexamined. Measuring that quantity reveals a failure that per-step modifications of GRPO do not address, and it is cheap to measure. We expect this lens to apply to objectives beyond the one we propose.

Q1: Geometry3K entries in Tables 1 and 3

1. We swapped two entries when filling Table 3. We thank the reviewer for catching this. When transcribing the τ\tau sweep into Table 3, we inadvertently swapped the pass@4 entries of τ=10−3\tau=10^{-3} and τ=3×10−3\tau=3\times10^{-3}. All other entries are correct. The correct values are:

τ\taupass@1pass@4 (submitted → correct)pass@8
10−310^{-3}77.6783.42 → 84.1589.00
3×10−33\times10^{-3}75.6784.15 → 83.4287.00

We apologize for this error and will correct Table 3 in the revision.

2. The three metrics in Table 1 share one checkpoint and one configuration. They come from the final checkpoint after one epoch, that is 525 optimizer updates; no checkpoint is selected during training. The configuration is the one reported in Sec. 5.1, and the τ=10−3\tau=10^{-3} row of Table 3 is this same run, which is why the corrected row (77.67 / 84.15 / 89.00) is identical to Table 1. The metrics differ only in how responses are decoded: pass@1 uses one greedy response per question, while pass@4 and pass@8 apply the unbiased estimator to the same eight sampled responses per question.

3. How the correction can be checked against the rest of the sweep. With the correct values, the τ=10−3\tau=10^{-3} run is ahead of the τ=3×10−3\tau=3\times10^{-3} run on all three metrics, and all three peak at τ=10−3\tau=10^{-3}. This is the behaviour the design predicts: the gaps are of order 10−310^{-3} (Fig. 1), which is the scale at which the gap weight fg=σ(−δg/τ)f_g=\sigma(-\delta_g/\tau) separates inverted from correctly ordered groups (Sec. 5.5).

4. The main comparison is unaffected. R²PO's pass@4 on Geometry3K is 84.15, above CISPO's 83.79, so the Table 1 ranking is unchanged.

Q2: Shared response groups, and stability across sampling seeds and group sizes

1. Figure 3(a) does not use the same groups. Each final checkpoint re-scores the response groups saved during its own training run, so the two bars are computed on different responses. We will state this in the caption, and we have repeated the diagnostic under the condition the reviewer describes.

2. The diagnostic on identical response groups. The relative reinforcement of Eq. (2) only requires scoring a fixed response, so both checkpoints can be evaluated on exactly the same groups. We re-scored fixed pools in which the questions, responses, correctness labels, token masks and initial-policy scores are identical for the two checkpoints, so that the only difference is the policy. To check that the conclusion does not depend on whose responses are used, we did this both with a pool sampled from the GRPO checkpoint and with a pool sampled from the R²PO checkpoint.

On these groups, the change R²PO produces falls where the design predicts. Among the groups that GRPO leaves inverted, the gap increases for 85% (181 of 214); among the groups GRPO already orders correctly, it increases for only 34% (161 of 479). The difference between the two strata is significant (Fisher test, p=2×10−37p=2\times10^{-37}).

3. Stability across sampling seeds. We repeated the whole procedure with sampling seeds 1 and 2 in addition to 27182, giving six pools in total. The targeting is present in every one of them:

Sampling seedPool sampled fromInverted under GRPO: gap increasesOrdered under GRPO: gap increases
27182GRPO89%37%
27182R²PO82%31%
1GRPO83%39%
1R²PO87%31%
2GRPO79%32%
2R²PO86%32%

4. Stability across group sizes. Without retraining, we varied the diagnostic group size by drawing 200 random four-response subsets of each group (G=4G=4) and by pooling two and three sampling seeds (G=16G=16 and G=24G=24). The share of mixed groups whose gap increases under R²PO is 55.5--58.9% at G=4G=4, 58.4% at G=8G=8 (693 groups, sign test p=1×10−5p=1\times10^{-5}), 64.1% at G=16G=16 (p=2×10−10p=2\times10^{-10}) and 69.0% at G=24G=24 (p=7×10−6p=7\times10^{-6}). The direction holds at every size and strengthens as groups grow, which is expected because larger groups estimate the class means with less noise.

5. A note on the thresholded count. The number of inverted groups is a thresholded statistic: a group only changes category when its gap crosses zero, and most groups lie further from zero than a typical per-group change. The continuous statistics above test the same hypothesis with more power, and we will report them alongside the counts in the revision.

Q3: Combination with other GRPO-family objectives

1. The term can be added to any of them. R²PO does not modify the per-step update: it computes a coefficient per response from quantities every GRPO-family objective already has — the response groups, the verifier labels and the initial-policy log-probabilities — and adds a surrogate whose gradient is those fixed coefficients (Eq. 10). Nothing in the construction refers to the clipping rule, the importance ratio or the resampling strategy, which is what GSPO, DAPO and Dr.GRPO change. The term can therefore be added to each of them without modification, and it keeps its cost: no additional sampling, and 0.12--3.55% more time per update.

2. Where the added term contributes, and where it overlaps. How much it contributes depends on how the base objective already weights groups. Under binary rewards, writing pp for a group's fraction of correct responses, the total coefficient mass a mixed group receives is proportional to p(1−p)\sqrt{p(1-p)} for objectives that normalize advantages by the group standard deviation, and to p(1−p)p(1-p) for objectives that drop that normalization. The R²PO term is proportional to agfga_gf_g, that is to p(1−p)p(1-p) times the gap weight (Eq. 9). It therefore shifts weight towards balanced groups relative to the first kind of objective, and overlaps with the weighting the second kind already applies. Among the objectives the reviewer names, GSPO and DAPO keep the group-standard-deviation normalization and change the importance ratio and the clipping and resampling rules instead, so the gap-dependent weighting remains new information for them; Dr.GRPO removes that normalization. We will add this analysis to the revision, as it delimits where the term is expected to help.

Reviewer_w3QDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies “reinforcement inversion”: relative to the initial policy, the mean token-level log-probability increase of sampled incorrect responses can exceed that of correct responses to the same question, despite GRPO favoring correct responses within each rollout group. It proposes Relative-Reinforcement Policy Optimization (R²PO), which adds a class-balanced logistic loss on this reinforcement gap. The objective reuses existing rollouts, verifier labels, and reference log-probabilities. Validation includes six model–task settings spanning mathematical and multimodal reasoning with two 4B backbones, comparisons with GRPO and alternative optimization methods, component ablations, training diagnostics, and runtime measurements. The reported results suggest a useful, inexpensive auxiliary objective, although the mechanism interpretation and some headline comparisons remain insufficiently established.

Critical strengths
  1. A simple intervention with a transparent optimization interpretation. Sections 3–4 and Appendix A explain how the auxiliary loss changes group weighting, including its relationship to GRPO under binary correctness rewards. The method reuses existing training information rather than requiring additional policy rollouts. This matters because the proposed improvement is sufficiently concrete and inexpensive to be independently tested and incorporated into existing RLVR systems.

  2. Useful empirical breadth relative to the method’s cost. Table 1 reports pass@1 improvements over GRPO of 1.18–3.34 percentage points across six model–task settings. The component ablations and the reported 0.12%–3.55% additional per-step time support the practical relevance of the objective. These results provide a substantive acceptance case if the comparisons and their stability can be confirmed.

Critical weaknesses
  1. The provenance of a headline comparison is unresolved. For Qwen3.5-4B on Geometry3K, Table 1 on p.6 reports R²PO pass@1/4/8 of 77.67/84.15/89.00. Table 3 on p.9 reports 77.67/83.42/89.00 at tau = 0.001; 84.15 instead appears at tau = 0.003, alongside different pass@1 and pass@8 values. The manuscript does not explain whether these are different checkpoints, evaluation draws, or a reporting error. This matters because using 83.42 changes the pass@4 ranking relative to CISPO’s 83.79. The discrepancy does not establish improper result selection, but it prevents me from accepting the stated dominance without a traceable, consistent evaluation record.

  2. The evidence does not isolate reinforcement inversion as the mechanism behind the gains. The average token log-ratio gap in Sections 3–4 is not the total probability mass of correct answers. Moreover, the paper explains that the update direction within a group is proportional to GRPO’s, with the main difference arising from group weighting. Reducing the optimized gap, or observing weights implied by the logistic function, therefore does not distinguish the proposed mechanism from ordinary changes in difficulty weighting or update magnitude. Existing ablations are useful, but keeping the same loss coefficient does not match gradient strength. This limits both the mechanistic claim in Section 5.4 and the evidence that the proposed weighting is preferable to simpler alternatives.

Questions for authors
  1. Clarification: Which checkpoint, tau, and evaluation sample set produced each Geometry3K entry in Tables 1 and 3? Were all three metrics in the Table 1 row computed from the same checkpoint and sampled responses? Please provide the corrected comparison if applicable.

  2. Clarification: Are the GRPO and R²PO inversion diagnostics evaluated on a shared, fixed response pool or on separately generated pools? What are the mixed-group denominators, and how were hyperparameters and checkpoints selected independently of the reported test results?

  3. Additional evidence that could change my rating: Does the gain remain under a gradient-magnitude-matched constant or shuffled gap-weight control, or a pairwise logistic objective using the same rollouts? A targeted comparison with repeated training would materially clarify whether the proposed gap information adds value beyond generic reweighting. If such evidence is unavailable, which mechanism claims would the authors retain?

4 comments

We thank the reviewer for the careful and constructive review, and we are glad that the reviewer finds R²PO a simple intervention with a transparent optimization interpretation, inexpensive enough to be tested independently and incorporated into existing RLVR systems, and supported by useful empirical breadth. Below we address each weakness (W) and question (Q). The new experiments use the configurations of the paper and will be added to the revision.

W1: Provenance of the Geometry3K comparison

1. We swapped two entries when filling Table 3. We thank the reviewer for catching this. When transcribing the τ\tau sweep into Table 3, we inadvertently swapped the pass@4 entries of τ=10−3\tau=10^{-3} and τ=3×10−3\tau=3\times10^{-3}. All other entries are correct. The correct values are:

τ\taupass@1pass@4 (submitted → correct)pass@8
10−310^{-3}77.6783.42 → 84.1589.00
3×10−33\times10^{-3}75.6784.15 → 83.4287.00

We apologize for this error and will correct Table 3 in the revision.

2. How the correction can be checked. The correct τ=10−3\tau=10^{-3} row is identical to the Table 1 row (77.67 / 84.15 / 89.00), since Table 1 reports this same run. The rest of Table 3 is also consistent with it:

  • The τ=10−3\tau=10^{-3} run is ahead of the τ=3×10−3\tau=3\times10^{-3} run on all three metrics. pass@4 and pass@8 are computed from the same eight samples per question, so they agree in this comparison.
  • All three metrics peak at τ=10−3\tau=10^{-3}.

This is the behavior the design predicts. The gaps are of order 10−310^{-3} (Fig. 1), which is the scale at which the gap weight fg=σ(−δg/τ)f_g=\sigma(-\delta_g/\tau) separates inverted from correctly ordered groups (Sec. 5.5).

3. The comparison with CISPO is unchanged. R²PO's pass@4 on Geometry3K is 84.15, above CISPO's 83.79, so R²PO remains best on pass@4 and pass@8 in all six settings.

W2: Does the evidence isolate reinforcement inversion as the mechanism?

1. The gap is a diagnostic, not the probability mass. We agree that the token-averaged gap δg\delta_g is not the probability mass of the correct class. It measures how the change accumulated since the initial policy is divided between the correct and incorrect responses of a question, and we measure accuracy separately. Inversion is also not an artifact of token averaging: with the summed (sequence-level) log-ratio, 39 of the 107 mixed groups of Figure 1 are still inverted, against 35 with the token mean.

2. A test that separates the gap weight from generic reweighting. R²PO keeps GRPO's direction within a group and changes only the group weight (Sec. 4.4), so the question is whether its gap-dependent factor fgf_g acts as designed. The design makes two predictions:

  • With τ\tau on the scale of the gaps (10−310^{-3}), fg>1/2f_g>1/2 for inverted groups and fg→0f_g\to0 for clearly ordered ones, so the correction should concentrate on inverted groups.
  • With τ=1\tau=1, fg≈1/2f_g\approx1/2 for every group. This is equivalent to a generic reweighting or a larger step, so the shift should be the same for all groups.

We scored fixed Geometry3K response pools under GRPO and under both R²PO checkpoints. The pools cover three sampling seeds and 693 mixed groups, and all scorings use identical responses, labels, masks and initial policy. We then counted the groups whose gap increases:

Inverted under GRPO (214)Ordered under GRPO (479)
R²PO, τ=10−3\tau=10^{-3}181 (85%)161 (34%)
R²PO, τ=1\tau=1 (fg≈1/2f_g\approx1/2)128 (60%)277 (58%)

At τ=10−3\tau=10^{-3} the increase is concentrated on inverted groups, and this holds on every sampling seed (82--86% vs. 32--35%; pooled Fisher test p=2×10−37p=2\times10^{-37}). At τ=1\tau=1 the shift is the same for both kinds of group (p=0.68p=0.68). A generic reweighting or a larger update predicts the τ=1\tau=1 pattern, not the targeted one, so the targeting comes from the gap weight. The τ=1\tau=1 checkpoint also rules out a selection effect on the groups labeled inverted.

3. Matched-strength controls. For accuracy, we compared R²PO against three alternatives with repeated training (Q3). The first is a constant force matched to R²PO's total force, which keeps the update magnitude the same. The other two are a shuffled force and a pairwise loss. All three retain a gain over GRPO, but R²PO has the highest mean accuracy (84.75%, versus 83.97%, 83.81% and 83.90%, respectively). These results support the shared balance-weighted loss on the reference-relative gap and show an additional observed benefit from R²PO's gap-dependent force assignment. The shared-pool diagnostic attributes the targeting of inverted groups to the gap weight fgf_g. We will state the mechanism claims of Section 5.4 in this form.

Q1: Checkpoint, τ\tau and evaluation samples of the Geometry3K entries

1. Which run produced each entry. The Table 1 row and the τ=10−3\tau=10^{-3} row of Table 3 are the same run; the other rows of Table 3 are separate runs that differ only in τ\tau. Here τ\tau is the temperature of the loss in Eq. (5), not a decoding temperature.

2. Checkpoint and samples. All three metrics in the Table 1 row come from the same checkpoint, the final one after one epoch (525 updates); no checkpoint is selected. Pass@1 uses one greedy response per question. Pass@4 and pass@8 use the unbiased estimator on the same eight sampled responses per question (Sec. 5.1), so the two sampled metrics share one response set, while the greedy metric uses its own deterministic response.

3. The corrected comparison. With the two swapped pass@4 entries of Table 3 corrected (W1):

Entrypass@1pass@4pass@8
Table 1, R²PO77.6784.1589.00
Table 3, τ=10−3\tau=10^{-3} (same run)77.6784.1589.00
Table 3, τ=3×10−3\tau=3\times10^{-3}75.6783.4287.00
Table 1, CISPO76.3383.7988.33

Q2: Shared response pools, denominators and model selection

1. Figure 3(a) uses separately generated pools. Each final checkpoint re-scores the response groups saved during its own training run, so the two bars are computed on different responses. We will state this in the caption.

2. We have repeated the diagnostic on shared pools. Because the relative reinforcement of Eq. (2) only requires scoring a fixed response, both checkpoints can be evaluated on the same responses. We therefore re-scored fixed pools in which the questions, responses, correctness labels, token masks and initial-policy scores are identical for the two checkpoints, so that the only difference is the policy. To check that the conclusion does not depend on whose responses are used, we did this with a pool sampled from the GRPO checkpoint and with a pool sampled from the R²PO checkpoint, and repeated both with three sampling seeds.

On these shared pools, the change R²PO produces is concentrated where the design predicts. Among the groups that GRPO leaves inverted, the gap increases for 85% (181 of 214), while among the groups that GRPO already orders correctly it increases for only 34% (161 of 479); the difference between the two strata is significant (Fisher test, p=2×10−37p=2\times10^{-37}). The pattern holds in each of the six pools separately (79--89% versus 31--39%). A control checkpoint whose gap weight is nearly constant shifts the two strata alike (60% versus 58%, p=0.68p=0.68), as a generic reweighting would.

3. Mixed-group denominators. Each pool is drawn from the 300 Geometry3K validation questions, and a group is mixed when it contains both correct and incorrect responses. For the GRPO-sampled and R²PO-sampled pools, the denominators are 107 and 123 at sampling seed 27182, 118 and 132 at seed 1, and 106 and 107 at seed 2, giving 693 mixed groups in total.

4. Checkpoint and hyperparameter selection. No checkpoint is selected: every method is evaluated at its final checkpoint after one epoch. λ\lambda and τ\tau of R²PO and the hyperparameters of every baseline were selected with the same protocol on each task's evaluation split (Sec. 5.1). To show that the gain does not depend on that selection, we trained two additional seeds with the selected configuration and no re-tuning; R²PO stays above GRPO on every seed (Q3).

Q3: Matched constant, shuffled and pairwise controls with repeated training

1. The reported gain over GRPO replicates across training seeds. In the submitted paper, R²PO improves on GRPO in all six model--task settings and on every metric, raising pass@1 by 1.18--3.34 points (Table 1). We have now repeated the M3CoT comparison with two additional training seeds, using the selected configuration with no re-tuning (greedy correct answers out of 1,108):

MethodSeed 27182 (Table 1)Seed 31415Seed 42Mean accuracy (%)
GRPO92492492483.39
R²PO94394093484.75

R²PO is above GRPO on every seed, by 19, 16 and 10 questions. Its mean accuracy is 84.75%, compared with 83.39% for GRPO, a gain of 1.35 percentage points.

2. The requested controls. For these controls, we kept the rollouts, the initial-policy reference, the masks and the class-balance weights fixed, and changed only how the force is assigned to groups: a constant force matched to R²PO's total force, a shuffled force, and a pairwise logistic objective on the same rollouts.

MethodSeed 27182Seed 31415Seed 42Mean accuracy (%)
GRPO92492492483.39
R²PO94394093484.75
Constant force, matched92793093483.97
Shuffled force92992593283.81
Pairwise logistic93192693283.90

Every variant improves on GRPO on all three seeds. R²PO has the highest mean accuracy. It is higher than all three controls on seeds 27182 and 31415; on seed 42, it ties the constant-force control at 934 correct answers and exceeds the shuffled and pairwise controls, both at 932.

3. What we conclude, and which claims we retain. We read this as evidence for both the shared design and R²PO's gap-dependent force assignment. All four objectives share the two components the paper identifies: a loss on the reinforcement gap measured against the initial policy, and a weighting of groups by class balance. Table 2 shows that removing either component lowers pass@1 on all three tasks, and Eq. (4) derives the balance weight from the covariance between correctness and relative reinforcement. What the controls vary is only how the remaining force is distributed across groups. All three variations improve on GRPO, while R²PO achieves the highest mean accuracy. We therefore retain:

  • Reinforcement inversion occurs in GRPO training and is measurable on shared responses (Figure 1, Q2).
  • Adding a balance-weighted loss on the reinforcement gap against the initial policy improves GRPO; this is the claim the paper makes, and it replicates across three training seeds and three force profiles.
  • The logistic form concentrates the correction on inverted groups when τ\tau is on the scale of the gaps (Q2); in the matched-control comparison, R²PO also achieves the highest mean accuracy.

These controls show that the shared components improve over GRPO and that R²PO gives the highest mean accuracy among the tested alternatives. We will report these controls in Section 5.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.