- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper reveals that GRPO sometimes reinforces incorrect responses more than correct ones, even though its advantages favor correct answers within each sampled group. The authors measure reinforcement by the change in mean token log-probability from the initial policy and call this effect reinforcement inversion. They propose R$^2$PO, which adds a logistic loss on the gap between the correct and incorrect class means, with groups weighted by class balance. The loss uses the rollouts and reference probabilities already available in GRPO. Experiments compare R$^2$PO with GRPO and seven variants on mathematical and multimodal reasoning tasks using two 4B Qwen models. The authors report gains over GRPO with little extra training time and use ablations to examine the design choices.
Critical strengths
-
The method is cheap to add to GRPO. R$^2$PO uses the rollouts and reference log-probabilities already available during training. The update is described in Algorithm 1 and Appendix B.1, and Table 4 reports an increase of only 0.12%–3.55% in median time per update. The low cost makes the reported gains over GRPO practically interesting.
-
The gains appear across different task settings. Table 1 reports pass@ improvements of 1.18-3.34 percentage points over GRPO across all six model and task settings. It is encouraging to see this pattern on both mathematics and multimodal reasoning. Table 2 also tests the balance weight, logistic loss, and initial-policy reference separately; removing any of them lowers performance relative to the full method on all three tasks.
Critical weaknesses
-
AMIR-GRPO [1] is particularly relevant to the algorithmic idea used in this paper. It also adds a logistic preference loss to GRPO using pairs from the same rollout groups and log-probability ratios (normalized by length) against a reference policy. The paper does not discuss or compare with it. I notice that R$^2$PO applies logistic loss to the mean gap in Eq. (5), while a pairwise objective averages the individual logistic losses. The benefit of the former approach is not tested. A comparison with AMIR-GRPO, or with GRPO plus a pairwise logistic loss, would help establish what the group aggregation contributes.
-
I also find some inconsistency of numerical results in the tables. Let's focus on results of R$^2$PO with Qwen3.5-4B model on Geometry3K dataset. From Table 1, pass@, pass@, pass@ are 77.67, 84.15, and 89.00 respectively, and the author mentioned is around ; from Table 3, , pass@, pass@, pass@ are 77.67, 83.42, and 89.00; from Table 3, , pass@, pass@, pass@ are 75.67, 84.15, and 87.00. This may be a typo or reflect different runs. Either way, it needs clarification: if 83.42 is the pass@4 result for the main setting, it is below CISPO’s 83.79, which affects the claim in Section 5.2 that R$^2$PO is best on pass@ in every setting.
References
[1] Amir Hossein Yari and Fajri Koto. AMIR-GRPO: Inducing Implicit Preference Signals into GRPO.
Questions for authors
-
Which exact and checkpoint were used for the Geometry3K row in Table 1? Please explain the difference from Table 3 and confirm whether all three metrics in each row were measured from the same selected model. What metrics are used to choose hyperparameters ?
-
Could you compare R$^2$PO with AMIR-GRPO or the pairwise logistic loss described above? Reporting variation across training seeds for the main comparisons would also help assess the reliability of the gains.
-
Table 2 supports the usefulness of the reference-dependent weighting, and Figure 3 shows that R$^2$PO reduces inversion. However, a negative gap in Eq. (3) can coexist with an increase in the total probability of correct answers. Did the authors examine whether questions whose inversion decreases also improve in accuracy? Measuring the gaps on a common response set and evaluating accuracy independently on the same questions would strengthen this connection.
7 comments
W2: Inconsistent Geometry3K entries and the pass@k ranking
1. It is a typo: we swapped two entries when filling Table 3. We thank the reviewer for the careful reading. When transcribing the sweep into Table 3, we inadvertently swapped the pass@4 entries of and . All other entries are correct:
| pass@1 | pass@4 (submitted → correct) | pass@8 | |
|---|---|---|---|
| 77.67 | 83.42 → 84.15 | 89.00 | |
| 75.67 | 84.15 → 83.42 | 87.00 |
The two rows are not different runs of the main setting: Table 1 reports the run, and with the correct values its row reads 77.67 / 84.15 / 89.00, identical to Table 1. We apologize for the error and will correct Table 3 in the revision.
2. The Section 5.2 claim is unaffected. R²PO's pass@4 on Geometry3K is 84.15, which is above CISPO's 83.79, so R²PO remains best on pass@4 and pass@8 in all six settings.
3. The corrected sweep is consistent with the rest of Table 3. With the correct values, the run is ahead of the run on all three metrics, and all three peak at . This matches the role of described in Sec. 5.5: the gaps are of order (Fig. 1), which is the scale at which the gap weight separates inverted from correctly ordered groups.
Q1: Checkpoint, and hyperparameter selection
1. and checkpoint for the Table 1 row. The Geometry3K row of Table 1 is the run, which is also the row of Table 3. The checkpoint is the final one after one epoch, that is 525 optimizer updates. No checkpoint is selected during or after training: every method in Table 1 is evaluated at its own final checkpoint under the same rule.
2. The difference from Table 3. It is a transcription error on our side: the pass@4 entries of and were swapped when Table 3 was filled. The correct row is 77.67 / 84.15 / 89.00, identical to Table 1, and the correct row is 75.67 / 83.42 / 87.00. We will correct Table 3 in the revision.
3. The three metrics come from the same model. All three are computed from that single final checkpoint. They differ only in decoding: pass@1 uses one greedy response per question, while pass@4 and pass@8 apply the unbiased estimator of Chen et al. (2021) to the same eight sampled responses per question. The greedy and the sampled metrics therefore use different response sets by construction, but the same model.
4. How is chosen. is selected by pass@1 on each task's evaluation split, the same protocol and the same metric used for the hyperparameters of every baseline in Table 1. The value is set on the scale of the observed reinforcement gaps, which are of order (Fig. 1), so that the gap weight varies across groups rather than saturating; Table 3 reports the sweep behind this choice, and Sec. 5.5 discusses it.
Q2: Pairwise comparison and variation across training seeds
1. The pairwise comparison. We report it under W1: with the same rollouts, reference, masks, class-balance weights and training budget, and only the aggregation changed, both the class-mean and the pairwise form improve on GRPO, and R²PO is higher than the pairwise form on all three training seeds.
2. Variation across training seeds. The submitted results are single runs per setting. We have now repeated the M3CoT comparison with two additional training seeds, keeping the selected configuration and re-tuning nothing (greedy correct answers out of 1,108):
| Method | Seed 27182 (Table 1) | Seed 31415 | Seed 42 | Mean ± s.d. (%) |
|---|---|---|---|---|
| GRPO | 924 | 924 | 924 | 83.39 ± 0.00 |
| R²PO | 943 | 940 | 934 | 84.75 ± 0.41 |
R²PO is above GRPO on each of the three seeds, by 19, 16 and 10 questions. Its mean accuracy is 84.75%, compared with 83.39% for GRPO, a mean gain of 1.35 percentage points.
3. What the spread shows. The gain varies across seeds while its direction does not. The spread is informative about the size of run-to-run variation on this task: GRPO happens to reach the same total on all three seeds, but the three runs differ on 94 questions between seeds 31415 and 42, so the agreement in totals is a coincidence rather than a sign of low variance. We will report the three seeds and their mean and standard deviation in the revision, and we will state that the submitted tables are single runs.
We thank the reviewer for the careful review and the concrete suggestions, and we are glad that the reviewer finds R²PO cheap to add to GRPO, the gains across mathematical and multimodal reasoning encouraging, and the separate tests of the balance weight, the logistic loss and the initial-policy reference informative. Below we address each weakness (W) and question (Q). The new experiments use the configurations of the paper and will be added to the revision.
W1: Relation to AMIR-GRPO and the role of group aggregation
1. The connection. We thank the reviewer for pointing us to AMIR-GRPO [1], which we will cite and discuss in Section 2. Both methods add a logistic preference loss to GRPO, computed from the rollout groups GRPO already produces, using length-normalized log-ratios against a reference policy. They differ in what the loss is applied to. Writing mij=ui+−uj− for a correct--incorrect pair and ℓτ(z)=τlog(1+e−z/τ):
pairwise: n+n−1∑i,jℓτ(mij),R2PO: ℓτ(n+n−1∑i,jmij)=ℓτ(δg).
Since the pair margins average to δg, the two forms differ only in whether the logistic is applied before or after averaging.
2. What the class-mean form provides. Two properties follow from this aggregation:
Because ℓτ is convex, Jensen's inequality gives ℓτ(δg)≤n+n−1∑i,jℓτ(mij), with equality only when all pairs have the same margin. The group form therefore responds to whether the two classes are ordered on average, while the pairwise form keeps a large loss for every misordered pair inside an otherwise well-ordered group.
3. The requested comparison. We implemented the pairwise objective in our codebase with the same rollouts, initial-policy reference, masks, class-balance weights and training budget, so that the aggregation is the only difference. On M3CoT (1,108 questions, greedy decoding):
Both forms improve on GRPO on all three seeds. R²PO is higher than the pairwise objective on every seed, with mean accuracy of 84.75% versus 83.90%.