- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces RAC-GRPO, a stateless modification of GRPO for binary rewards. Given a rollout group's success count and size, it adds symmetric success/failure pseudo-counts to calibrate the reward mean and standard deviation. The correction is chosen using a reference Beta posterior and a constraint derived from probability-estimation MSE. A precomputed lookup table supplies the resulting advantages during training. The method both modifies mixed-reward groups and assigns small nonzero advantages to all-correct and all-wrong groups.
The analysis decomposes the advantages into a rescaled within-group contrast and a shared offset, and characterizes expected updates and endpoint noise under explicit on-policy assumptions. Three-seed experiments on Qwen3-1.7B, 4B, and 8B report average improvements of 2.53, 2.08, and 2.82 percentage points over GRPO across seven mathematical reasoning benchmarks. Additional experiments examine component contributions, correction strength, smaller rollout groups, and integration with GSPO.
Critical strengths
-
The empirical evidence is substantial for a lightweight optimization change. The main comparisons use three training seeds at three model scales, and RAC-GRPO improves the mean on every reported model-benchmark combination. The matched Qwen3-8B runs in Table 10 improve by approximately 2.71, 2.60, and 3.13 points, so the average gain is not driven by one favorable run. Table 2 separately tests endpoint recovery and mixed-group calibration, while Figure 3 holds mixed-group calibration fixed when varying endpoint strength. These are useful controls that directly address the proposed mechanism.
-
The method is simple to implement, and the analysis makes its behavior unusually transparent. Training requires only a lookup indexed by group size and success count, with no additional rollouts, persistent prompt history, or critic. The contrast-offset decomposition explains how the method affects both homogeneous and mixed groups. I also appreciate the treatment of locally saturated prompts: the paper explicitly shows that recovered updates can be pure sampling noise there. This qualification gives a more credible account of the method than an argument based solely on eliminating zero advantages.
Critical weaknesses
-
The main GRPO comparison changes the normalization convention as well as introducing calibration. Appendix C.3 states that GRPO uses the sample standard deviation, whereas RAC-GRPO uses a Bernoulli plug-in standard deviation. With negligible numerical stabilization, the population-normalized GRPO advantages are larger than the sample-normalized ones by , approximately 1.069 at and 1.155 at . Consequently, even balanced groups are changed relative to the implemented baseline, although they are unchanged relative to the theoretical reference. This does not establish that the reported gains are caused by scaling, and the endpoint-recovery evidence remains useful. Nevertheless, a GRPO control using the same population convention is needed to cleanly attribute the mixed-group gains to calibration.
-
The evidence for the particular risk-constrained schedule is weaker than the evidence for modest endpoint recovery plus mixed-group adjustment. The paper correctly acknowledges that the risk constraint concerns probability estimation, and that fixed has lower integrated estimation MSE despite worse training performance. Thus, the constraint provides a defensible construction, but does not explain why the largest admissible correction should be the preferred optimization choice. Table 3 varies endpoint and interior corrections together, while Figure 3 retains the proposed interior schedule. Neither comparison establishes whether a simpler rule with the same endpoint magnitude and one fixed interior pseudo-count would perform comparably. This matters because choosing the correction as a function of the observed count is the central methodological contribution beyond existing virtual-reward and Bayesian-estimation approaches (He et al., 2026; Kim et al., 2026).
Questions for authors
-
Additional evidence affecting the assessment: How do the main comparison and mixed-group ablation change when GRPO uses the population standard deviation, matching the convention underlying RAC-GRPO? A representative three-seed comparison would help determine whether the normalization mismatch materially affects the conclusions.
-
Additional evidence affecting the assessment: With both endpoint advantages fixed to RAC-GRPO's magnitude of approximately 0.0376, how does the method compare against a validation-selected fixed pseudo-count applied only to mixed groups? This would directly test the value of the count-dependent interior schedule while keeping endpoint recovery unchanged.
-
Clarification: What is the source, size, and selection metric of the validation set used in Appendix C.5, and how frequently are checkpoints evaluated? Was the 95% reference credible level fixed before experimentation or selected using this validation set? The paper states that test benchmarks were excluded from selection; these details would make that protocol reproducible.
4 comments
Response to Reviewer bc39(2/2)
W2 & Q2: Contribution of the count-dependent schedule beyond a fixed interior correction
Response: RAC does not minimize MSE; MSE only limits calibration strength. We first calibrate the current empirical success probability, then use the MSE constraint to restrict the correction. For each observed success count , we select the largest pseudo-count whose fixed- estimator has no greater pointwise MSE than the empirical MLE throughout the 95% equal-tailed credible interval of the reference posterior .
The rationale for selecting the largest admissible correction follows from the advantage decomposition: $ A_i^{\mathrm{RAC}}=\lambda_K A_i^{\mathrm{GRPO}}+b_K$.
For unbalanced groups, increasing decreases the contrast factor and strictly increases the magnitude of the shared offset . Thus, selecting the largest admissible maximizes the shared-offset correction within the MSE-constrained range, while controlling the associated probability-estimation distortion.
Estimation risk and policy-learning performance are evaluated separately:
| Method | MSE | Acc. (%) |
|---|---|---|
| Fixed | 0.01667 | 65.58 ± 0.76 |
| RAC | 0.01989 | 68.62 ± 0.50 |
Integrated MSE summarizes probability-estimation error over a distribution. RAC uses the pointwise risk constraint to limit the correction at each observed success count; the table separately evaluates estimation risk and policy-learning performance.
The fixed-interior comparison is an ablation of RAC. It retains RAC's endpoint advantages (approximately ) and replaces only the count-dependent mixed-group pseudo-count with a constant. Both methods use the same normalization and remaining training settings.
We screened using the first training seed and selected on the independent validation set. The reported three-seed comparison includes the screening run and two additional matched-seed runs.
| Interior rule | Seed 1 | Seed 2 | Seed 3 | Avg (%) |
|---|---|---|---|---|
| Fixed | 67.80 | 67.71 | 68.55 | 68.02 ± 0.46 |
| RAC | 68.21 | 68.46 | 69.18 | 68.62 ± 0.50 |
RAC improves over the validation-selected fixed correction by 0.60 pp, with positive paired gains on all three matched seeds (+0.41, +0.75, and +0.63 pp), calculated from the displayed seed-level means. Since endpoint advantages are identical, this comparison directly quantifies the benefit of adapting the interior correction to the observed success count relative to the validation-selected constant correction. The fixed-interior control also exceeds uniform by 0.43 pp, supporting different calibration strengths for mixed and endpoint groups.
Q3: Validation set and credible-level protocol
Response: All methods and ablations use the same validation protocol:
| Item | Protocol |
|---|---|
| Validation set | Omni-MATH-Rule |
| Size | 2,821 problems |
| Checkpoint evaluation | Every 20 training steps |
| Decoding | Greedy |
| Selection metric | Highest validation accuracy |
| Test-set use | Seven test benchmarks excluded from selection and tuning |
| Reference credible level | 95%, fixed before experimentation |
| Sensitivity settings | 80%, 85%, and 90%; exploratory single-seed runs |
The reference credible level was fixed independently of validation or test performance.
Thank you for the detailed response. The responses address most of my concerns. I would suggest including these results in the final version and polishing the figures and tables for readability. I will maintain my positive rating.
Thank you for your positive assessment and constructive suggestions. We are currently consolidating feedback from all reviewers and will upload a revised PDF once the ongoing discussions have concluded. The revision will incorporate the additional experimental results and validation details presented in our responses, along with improvements to the readability of the figures and tables, as you suggested.
We hope these revisions will further improve the clarity and completeness of the paper. Thank you again for your time and thoughtful feedback.
Response to Reviewer bc39(1/2)
Thank you for highlighting the controls needed to attribute RAC's gains. We added a population-normalized GRPO baseline and an endpoint-matched comparison against a fixed interior correction. We also report the validation protocol and credible-level setting.
Unless otherwise stated, results use Qwen3-8B with eight rollouts per prompt. Average accuracy is the unweighted mean across seven benchmarks; ± denotes the sample standard deviation across three matched training seeds. Aggregate gains compare method averages; paired gains compare matched seeds.
W1 & Q1: Population-normalized GRPO control and attribution of calibration gains
Response: The original GRPO baseline uses the sample standard deviation, whereas RAC-GRPO uses a Bernoulli population plug-in standard deviation. Ignoring numerical stabilization, population-normalized advantages are larger than sample-normalized ones by a factor of G/(G−1): approximately 1.069 at G=8 and 1.155 at G=4. For balanced groups, RAC's calibration leaves population-normalized GRPO advantages unchanged; however, the normalization difference still changes their magnitudes relative to the original sample-normalized GRPO baseline.
To isolate calibration from this normalization difference, we added GRPO-pop using the same population normalization as RAC-GRPO, while retaining the other training settings and matched seeds. The results are:
Under the common population-normalization convention, mixed-group calibration improves over GRPO-pop by 1.06 pp, and endpoint recovery adds a further 1.54 pp. Thus, full RAC-GRPO improves over GRPO-pop by 2.60 pp. The full RAC-GRPO gains are positive on all three matched seeds (+1.96, +2.54, and +3.29 pp), calculated from the displayed seed-level means.
Our theoretical statement that balanced groups are unchanged is relative to the population-normalized GRPO reference. At K=G/2, the calibrated success probability remains 1/2, and the resulting advantages coincide with those of GRPO-pop.
The next response addresses calibration strength and the validation protocol.