acceptodds
Under review as a conference paper at ICLR 2027

RAC-GRPO: RISK-CONSTRAINED ADVANTAGE CALIBRATION FOR FINITE ROLLOUT GROUPS

Abstract

Group Relative Policy Optimization (GRPO) constructs advantages from reward statistics estimated within finite rollout groups. As a result, its training weights are sensitive to finite-group sampling. With binary rewards, this issue is most extreme when all sampled responses are correct or all are wrong, in which case GRPO assigns zero advantage to the entire group. Mixed groups are also affected because their reward means and standard deviations are estimated from only a few samples. We introduce Risk-Constrained Advantage Calibration for GRPO (RAC-GRPO), a stateless calibration algorithm based on the observed success count and group size. It adjusts the reward statistics by adding symmetric virtual success and failure counts. A reference posterior describes uncertainty in the underlying success probability. A mean squared error constraint limits the allowable correction strength. Within this constraint and a fixed cap, RAC-GRPO selects the largest admissible correction. The resulting advantages decompose into an attenuated within-group contrast and a shared offset. This recovers small updates for all-correct and all-wrong groups while also calibrating mixed groups. Under independent on-policy sampling with equally weighted responses, we characterize the expected update. We show that endpoint recovery is aligned with the success gradient. The recovered component’s mean and directional standard deviation both scale linearly with recovery magnitude. Training requires only a precomputed lookup table, with no additional responses, reward history, or learned critic. Across seven mathematical reasoning benchmarks, RAC-GRPO improves over GRPO by 2.53, 2.08, and 2.82 percentage points on Qwen3-1.7B, 4B, and 8B, respectively.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Reinforcement Learning with Verifiable Rewards, Group Relative Policy Optimization, Advantage Estimation, Finite-Sample Calibration, Large Language Model Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bc39OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RAC-GRPO, a stateless modification of GRPO for binary rewards. Given a rollout group's success count and size, it adds symmetric success/failure pseudo-counts to calibrate the reward mean and standard deviation. The correction is chosen using a reference Beta posterior and a constraint derived from probability-estimation MSE. A precomputed lookup table supplies the resulting advantages during training. The method both modifies mixed-reward groups and assigns small nonzero advantages to all-correct and all-wrong groups.

The analysis decomposes the advantages into a rescaled within-group contrast and a shared offset, and characterizes expected updates and endpoint noise under explicit on-policy assumptions. Three-seed experiments on Qwen3-1.7B, 4B, and 8B report average improvements of 2.53, 2.08, and 2.82 percentage points over GRPO across seven mathematical reasoning benchmarks. Additional experiments examine component contributions, correction strength, smaller rollout groups, and integration with GSPO.

Critical strengths
  1. The empirical evidence is substantial for a lightweight optimization change. The main comparisons use three training seeds at three model scales, and RAC-GRPO improves the mean on every reported model-benchmark combination. The matched Qwen3-8B runs in Table 10 improve by approximately 2.71, 2.60, and 3.13 points, so the average gain is not driven by one favorable run. Table 2 separately tests endpoint recovery and mixed-group calibration, while Figure 3 holds mixed-group calibration fixed when varying endpoint strength. These are useful controls that directly address the proposed mechanism.

  2. The method is simple to implement, and the analysis makes its behavior unusually transparent. Training requires only a lookup indexed by group size and success count, with no additional rollouts, persistent prompt history, or critic. The contrast-offset decomposition explains how the method affects both homogeneous and mixed groups. I also appreciate the treatment of locally saturated prompts: the paper explicitly shows that recovered updates can be pure sampling noise there. This qualification gives a more credible account of the method than an argument based solely on eliminating zero advantages.

Critical weaknesses
  1. The main GRPO comparison changes the normalization convention as well as introducing calibration. Appendix C.3 states that GRPO uses the sample standard deviation, whereas RAC-GRPO uses a Bernoulli plug-in standard deviation. With negligible numerical stabilization, the population-normalized GRPO advantages are larger than the sample-normalized ones by G/(G−1)\sqrt{G/(G-1)}, approximately 1.069 at G=8G=8 and 1.155 at G=4G=4. Consequently, even balanced groups are changed relative to the implemented baseline, although they are unchanged relative to the theoretical reference. This does not establish that the reported gains are caused by scaling, and the endpoint-recovery evidence remains useful. Nevertheless, a GRPO control using the same population convention is needed to cleanly attribute the mixed-group gains to calibration.

  2. The evidence for the particular risk-constrained schedule is weaker than the evidence for modest endpoint recovery plus mixed-group adjustment. The paper correctly acknowledges that the risk constraint concerns probability estimation, and that fixed c=1c=1 has lower integrated estimation MSE despite worse training performance. Thus, the constraint provides a defensible construction, but does not explain why the largest admissible correction should be the preferred optimization choice. Table 3 varies endpoint and interior corrections together, while Figure 3 retains the proposed interior schedule. Neither comparison establishes whether a simpler rule with the same endpoint magnitude and one fixed interior pseudo-count would perform comparably. This matters because choosing the correction as a function of the observed count is the central methodological contribution beyond existing virtual-reward and Bayesian-estimation approaches (He et al., 2026; Kim et al., 2026).

Questions for authors
  1. Additional evidence affecting the assessment: How do the main comparison and mixed-group ablation change when GRPO uses the population standard deviation, matching the convention underlying RAC-GRPO? A representative three-seed comparison would help determine whether the normalization mismatch materially affects the conclusions.

  2. Additional evidence affecting the assessment: With both endpoint advantages fixed to RAC-GRPO's magnitude of approximately 0.0376, how does the method compare against a validation-selected fixed pseudo-count applied only to mixed groups? This would directly test the value of the count-dependent interior schedule while keeping endpoint recovery unchanged.

  3. Clarification: What is the source, size, and selection metric of the validation set used in Appendix C.5, and how frequently are checkpoints evaluated? Was the 95% reference credible level fixed before experimentation or selected using this validation set? The paper states that test benchmarks were excluded from selection; these details would make that protocol reproducible.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bc39(1/2)

Thank you for highlighting the controls needed to attribute RAC's gains. We added a population-normalized GRPO baseline and an endpoint-matched comparison against a fixed interior correction. We also report the validation protocol and credible-level setting.

Unless otherwise stated, results use Qwen3-8B with eight rollouts per prompt. Average accuracy is the unweighted mean across seven benchmarks; ± denotes the sample standard deviation across three matched training seeds. Aggregate gains compare method averages; paired gains compare matched seeds.


W1 & Q1: Population-normalized GRPO control and attribution of calibration gains

Response: The original GRPO baseline uses the sample standard deviation, whereas RAC-GRPO uses a Bernoulli population plug-in standard deviation. Ignoring numerical stabilization, population-normalized advantages are larger than sample-normalized ones by a factor of G/(G−1)\sqrt{G/(G-1)}: approximately 1.069 at G=8G=8 and 1.155 at G=4G=4. For balanced groups, RAC's calibration leaves population-normalized GRPO advantages unchanged; however, the normalization difference still changes their magnitudes relative to the original sample-normalized GRPO baseline.

To isolate calibration from this normalization difference, we added GRPO-pop using the same population normalization as RAC-GRPO, while retaining the other training settings and matched seeds. The results are:

MethodSeed 1Seed 2Seed 3Avg (%)
GRPO (sample)65.5065.8666.0565.80 ± 0.28
GRPO-pop66.2565.9265.8966.02 ± 0.20
RAC (mixed only)66.5767.1767.5067.08 ± 0.47
RAC-GRPO68.2168.4669.1868.62 ± 0.50

Under the common population-normalization convention, mixed-group calibration improves over GRPO-pop by 1.06 pp, and endpoint recovery adds a further 1.54 pp. Thus, full RAC-GRPO improves over GRPO-pop by 2.60 pp. The full RAC-GRPO gains are positive on all three matched seeds (+1.96, +2.54, and +3.29 pp), calculated from the displayed seed-level means.

Our theoretical statement that balanced groups are unchanged is relative to the population-normalized GRPO reference. At K=G/2K=G/2, the calibrated success probability remains 1/21/2, and the resulting advantages coincide with those of GRPO-pop.

The next response addresses calibration strength and the validation protocol.

AuthorsOpenReview ↗

Response to Reviewer bc39(2/2)

W2 & Q2: Contribution of the count-dependent schedule beyond a fixed interior correction

Response: RAC does not minimize MSE; MSE only limits calibration strength. We first calibrate the current empirical success probability, then use the MSE constraint to restrict the correction. For each observed success count KK, we select the largest pseudo-count c≤2c\leq 2 whose fixed-cc estimator has no greater pointwise MSE than the empirical MLE K/GK/G throughout the 95% equal-tailed credible interval of the reference posterior Beta(K+1,G−K+1)\mathrm{Beta}(K+1,G-K+1).

The rationale for selecting the largest admissible correction follows from the advantage decomposition: $ A_i^{\mathrm{RAC}}=\lambda_K A_i^{\mathrm{GRPO}}+b_K$.

For unbalanced groups, increasing cc decreases the contrast factor λK\lambda_K and strictly increases the magnitude of the shared offset ∣bK(c)∣|b_K(c)|. Thus, selecting the largest admissible cc maximizes the shared-offset correction within the MSE-constrained range, while controlling the associated probability-estimation distortion.

Estimation risk and policy-learning performance are evaluated separately:

MethodMSEAcc. (%)
Fixed c=1c=10.0166765.58 ± 0.76
RAC0.0198968.62 ± 0.50

Integrated MSE summarizes probability-estimation error over a distribution. RAC uses the pointwise risk constraint to limit the correction at each observed success count; the table separately evaluates estimation risk and policy-learning performance.

The fixed-interior comparison is an ablation of RAC. It retains RAC's endpoint advantages (approximately ±0.0376\pm 0.0376) and replaces only the count-dependent mixed-group pseudo-count with a constant. Both methods use the same normalization and remaining training settings.

We screened cint∈0,0.0113,0.1,0.5,1,2c_{\mathrm{int}}\in\\{0,0.0113,0.1,0.5,1,2\\} using the first training seed and selected cint=0.5c_{\mathrm{int}}=0.5 on the independent validation set. The reported three-seed comparison includes the screening run and two additional matched-seed runs.

Interior ruleSeed 1Seed 2Seed 3Avg (%)
Fixed cint=0.5c_{\mathrm{int}}=0.567.8067.7168.5568.02 ± 0.46
RAC68.2168.4669.1868.62 ± 0.50

RAC improves over the validation-selected fixed correction by 0.60 pp, with positive paired gains on all three matched seeds (+0.41, +0.75, and +0.63 pp), calculated from the displayed seed-level means. Since endpoint advantages are identical, this comparison directly quantifies the benefit of adapting the interior correction to the observed success count relative to the validation-selected constant correction. The fixed-interior control also exceeds uniform c=0.01c=0.01 by 0.43 pp, supporting different calibration strengths for mixed and endpoint groups.


Q3: Validation set and credible-level protocol

Response: All methods and ablations use the same validation protocol:

ItemProtocol
Validation setOmni-MATH-Rule
Size2,821 problems
Checkpoint evaluationEvery 20 training steps
DecodingGreedy
Selection metricHighest validation accuracy
Test-set useSeven test benchmarks excluded from selection and tuning
Reference credible level95%, fixed before experimentation
Sensitivity settings80%, 85%, and 90%; exploratory single-seed runs

The reference credible level was fixed independently of validation or test performance.

Thank you for the detailed response. The responses address most of my concerns. I would suggest including these results in the final version and polishing the figures and tables for readability. I will maintain my positive rating.

Thank you for your positive assessment and constructive suggestions. We are currently consolidating feedback from all reviewers and will upload a revised PDF once the ongoing discussions have concluded. The revision will incorporate the additional experimental results and validation details presented in our responses, along with improvements to the readability of the figures and tables, as you suggested.

We hope these revisions will further improve the clarity and completeness of the paper. Thank you again for your time and thoughtful feedback.

Reviewer_NsDEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GRPO constructs response advantages from the reward mean and standard deviation estimated within finite rollout groups. With binary verifier rewards, groups in which all responses share the same reward (K = 0 or K = G) receive zero advantages, and mixed groups estimate their reward statistics from only a few samples. RAC-GRPO addresses this by adding a count-dependent pseudo-count c(K, G) to both successes and failures: p̃ = (K + c)/(G + 2c). The correction strength is chosen as the largest c ≤ 2 such that the 95% equal-tailed credible interval of a Beta(K+1, G−K+1) reference posterior is contained in the set of success probabilities for which shrinkage has pointwise MSE no larger than the MLE — a closed-form rule (Prop. 4) with a bound guaranteeing no worse conditional posterior risk than K/G (Prop. 2). The resulting advantages decompose into λK · GRPO-advantage plus a shared offset bK (sign agrees with the majority reward). The theory characterizes the expected update E[ĝ] = α(p)∇θp (Prop. 1), shows endpoint recovery is aligned with the success gradient, and analyzes the second moments of the recovered endpoint component (mean and noise both scale linearly with the endpoint magnitude; pure sampling noise at locally saturated prompts). Validation: Qwen3-1.7B/4B/8B trained for two epochs on DAPO-Math-17K, G = 8, three seeds, seven mathematical benchmarks; average gains of 2.53/2.08/2.82 percentage points over GRPO, plus component ablations, a fixed-pseudo-count sweep, an endpoint-strength sweep, credible-level sensitivity, a G = 4 comparison, and a GSPO transfer. Baselines are GRPO, AVSPO, and DBB (both reimplemented by the authors).

Critical strengths
  1. Honest and unusually complete empirical analysis. The component ablations (endpoint-only vs. mixed-only vs. their combination), the fixed-pseudo-count sweep, and the endpoint-strength sweep (Fig. 3) mutually corroborate the paper's central empirical claim that "stronger correction is not uniformly better," and the per-seed tables and full training configuration support reproducibility. This level of experimental honesty is a genuine asset: even if the mechanism is simple, the reader can see exactly where the gains come from and how sensitive they are.
  2. Extremely lightweight, stateless method with a verifiable core theory. Deployment is a precomputed lookup table — no extra rollouts, no reward history, no critic — and the appendix analysis (expected update alignment with ∇p; the endpoint magnitude decaying as G⁻¹ under the risk constraint; the signal/noise co-scaling at saturated prompts) is correct as far as I verified. The G⁻¹ argument is a principled reason for keeping endpoint corrections small.
Critical weaknesses
  1. Limited novelty and a weak link between the central design principle and the training objective. Endpoint recovery through shrinkage / pseudo-counts / virtual rewards / fixed offsets already exists (e.g., AERO's fixed Beta(1,1) posterior-mean endpoints, AVSPO's virtual rewards, margin/fixed-sign baselines such as Dr. GRPO and Nie et al.). The genuinely new component — count-dependent correction strength selected by an MSE risk constraint — delivers only +1.03 pp over the best fixed pseudo-count (c = 0.01, Table 3) on Qwen3-8B, and the paper itself demonstrates (Table 5) that the MSE-optimal estimator (fixed c = 1) is the worst for RL. The risk constraint is therefore a heuristic with no established connection to policy-optimization quality, and the closest conceptual baselines (fixed-sign recovery, margin-based GRPO, AERO-style posterior means) are absent from the experiments. As a result, the marginal contribution over "use a small pseudo-count" is modest and not causally explained.
  2. Comparison credibility and attribution are not airtight. AVSPO and DBB are reimplementations from published descriptions with no validation against the original papers' reported numbers (AVSPO underperforms GRPO by 3.6 pp at 1.7B, which is surprising for a method that should recover collapsed groups — this raises reimplementation-quality concerns). In addition, the GRPO baseline uses the sample standard deviation (torch.std default) while RAC-GRPO uses the population standard deviation of the shrunken mean (Appendix C.3), a systematic ~6.9% difference in mixed-group advantage magnitudes that confounds the attribution of the mixed-group gains to calibration per se. Statistical power is limited: three seeds, no significance testing, and several per-benchmark margins (e.g., Minerva at 1.7B, +0.12 pp) are within noise.
Questions for authors
  1. AVSPO and DBB baselines were reimplemented from the papers' descriptions, and AVSPO underperforms GRPO by 3.6 points on the 1.7B model. Have you validated your reimplementations against the numbers reported in the original papers, and if so, where? If not, how can readers be confident the comparison is fair? Relatedly, why did you not include the closest existing approaches — e.g., Dr. GRPO's reward margin, the fixed-sign advantage recovery of Nie et al., or AERO's posterior-mean endpoints — as baselines?
  2. Table 3 shows that a fixed pseudo-count of c = 0.01 already captures most of the improvement (67.59 vs. 68.62 for the full method). How much of the final gain comes specifically from the count-dependent schedule, rather than from adding any small shrinkage at all? What do you expect with an even smaller fixed value, and is the count-dependence worth its complexity?
  3. In a real training run, how often do zero-variance and unbalanced groups actually occur (beyond the single trace in Fig. 5), what fraction of the total update do the recovered endpoint signals contribute, and is the endpoint-only gain in Table 2 stable across seeds?
  4. Beyond math reasoning with Qwen3 models on a single corpus: what is the actual wall-clock / compute overhead of RAC-GRPO relative to GRPO, and do you have any evidence on other domains or model families?
6 comments
AuthorsOpenReview ↗

Response to Reviewer NsDE(1/4)

Thank you for the careful assessment and for recognizing the lightweight implementation, theoretical analysis, and comprehensive empirical evaluation. We clarify the role of the MSE constraint, its connection to policy optimization, and the distinction between fixed and count-dependent calibration. We also provide additional baseline reproductions, matched-normalization controls, direct comparisons with related recovery mechanisms, and analyses of group composition and gradient contributions.

Unless otherwise stated, Avg denotes the unweighted average accuracy over seven benchmarks; the DBB reproduction averages the nine listed benchmarks. Values after ± denote the sample standard deviation across three training seeds; gains are reported in percentage points (pp). Aggregate gains compare method averages; paired gains compare matched seeds.


W1: Limited novelty and the connection between the calibration principle and policy optimization

Response: RAC addresses the finite-sample estimation problem in binary-reward rollout groups. Given GG responses and KK successes, standard GRPO estimates the group success probability using K/GK/G, which leads to zero advantages for zero-variance groups and potentially unreliable reward statistics for mixed groups.

Existing methods provide related but distinct recovery mechanisms. AVSPO uses virtual rewards and analyzes their gradient bias; AERO uses fixed Beta(1,1) posterior means for residual zero-variance groups alongside adaptive sampling; and Nie et al. use fixed sign-based advantages. Dr. GRPO addresses normalization bias by removing standard-deviation and response-length normalization, but retains group-mean centering. Consequently, its advantages still vanish when all rewards in a group are identical.

Unlike these approaches, RAC applies count-dependent calibration to every observed success count and constrains the calibration strength through an explicit MSE-based risk criterion. For each observed count KK, RAC selects the largest cK≤2c_K\leq 2 such that the associated fixed-cc estimator's pointwise MSE does not exceed that of the empirical MLE K/GK/G for every success probability within the 95% equal-tailed credible interval of the reference posterior Beta(K+1,G−K+1)\mathrm{Beta}(K+1,G-K+1). Proposition 2 further establishes a bound on conditional posterior squared-error risk that is no worse than the empirical estimate. This provides a unified treatment of endpoint recovery and mixed-group calibration while controlling the estimation error introduced by calibration.

RAC does not minimize MSE; MSE only limits calibration strength. We first calibrate the current group's empirical statistics, then constrain the admissible correction to avoid excessive estimation distortion. Fixed c=1c=1 minimizes integrated probability-estimation MSE under uniform weighting, which solves a different estimation objective and is not the criterion used to select RAC's correction. Its lower integrated MSE therefore does not contradict our design. Its endpoint advantage magnitude is also much larger than RAC's (approximately 0.3333 versus 0.0376), and the endpoint-strength ablation shows that stronger recovery is not uniformly better. The training experiments assess the practical value of the constrained calibration.

The calibrated statistics enter the training advantages directly: $ \tilde p_K=\frac{K+c_K}{G+2c_K}, $ $ \hat A_i^{\mathrm{RAC}} = \frac{r_i-\tilde p_K} {\sqrt{\tilde p_K(1-\tilde p_K)}+\epsilon_{\mathrm{num}}}. $ The theoretical connection to policy optimization comes from these calibrated advantages. Under the on-policy sampling and weighting assumptions of Proposition 1,$ \mathbb E[\hat g] = \alpha_{\mathrm{RAC}}(p)\nabla_\theta p, \alpha_{\mathrm{RAC}}(p)>0, $ and the isolated zero-variance recovery contribution satisfies $ \mathbb E[\hat g-\hat g_{\mathrm{mix}}] = a_{0,\epsilon} \left[p^{G-1}+(1-p)^{G-1}\right] \nabla_\theta p. $ These results establish the alignment of the expected policy update with the success-probability gradient and characterize the contribution of endpoint recovery. The matched experiments further evaluate how the calibration strength affects practical training performance.

The next response addresses normalization, statistical evidence, and baseline reproduction.

AuthorsOpenReview ↗

Response to Reviewer NsDE(2/4)

W2: Comparison credibility, normalization, and statistical evidence

Response: Our main comparison evaluates calibration under common sampling and optimization settings. To isolate calibration from the difference between sample and population standard deviations, we added GRPO-pop, which uses the same Bernoulli population normalization as RAC. At G=8G=8, population normalization changes mixed-group advantage magnitudes by a factor of G/(G−1)≈1.069\sqrt{G/(G-1)}\approx1.069.

The improvement persists after matching normalization: RAC gains 2.60 pp over GRPO-pop, comprising 1.06 pp from mixed-group calibration and a further 1.54 pp from endpoint recovery.

MethodCalibrationAcc. (%)
GRPO (sample)None; original normalization65.80 ± 0.28
GRPO-popNone; matched population normalization66.02 ± 0.20
RAC (mixed only)Mixed-group calibration; zero endpoint advantages67.08 ± 0.47
RAC-GRPOMixed-group calibration and endpoint recovery68.62 ± 0.50

The gains over GRPO-pop are positive on all three matched seeds (+1.96, +2.54, and +3.29 pp). The gain over the original GRPO baseline is 2.82 pp, also positive on all three seeds (+2.71, +2.60, and +3.13 pp).

We use Avg across seven benchmarks as the primary measure of overall performance and report individual benchmark results, sample standard deviations, and all paired seed-level differences. The positive aggregate gains on every matched seed support RAC's effectiveness across both GRPO references. Within the available rebuttal compute budget, we prioritized these matched three-seed controls; additional seeds are planned as part of the broader evaluation.


Q1: Validation of AVSPO/DBB implementations and comparisons with nearby recovery approaches

Response: We added reproduction checks against the original AVSPO and DBB results, direct controls for fixed-sign and posterior-mean recovery, and a full-pipeline comparison with AERO.

AVSPO reproduction.

Following AVSPO's original data-selection procedure, we evaluated AVSPO and GRPO on the same selected training subset with matched training and evaluation configurations. Our reproduction on Qwen2.5-Math-7B obtains a positive aggregate improvement of 3.90 pp, with gains on six of seven benchmarks:

BenchmarkPaper GRPO (%)Paper AVSPO (%)Our GRPO (%)Our AVSPO (%)Our gain (pp)
MATH50065.0074.1074.0075.80+1.80
GSM8K65.3069.7086.2888.40+2.12
MinervaMath25.7029.4031.6234.93+3.31
OlympiadBench36.2043.6039.4738.72−0.75
AMC40.9043.8052.6662.19+9.53
AIME2420.6023.2016.6726.46+9.79
MMLU-Pro38.9040.1037.7839.25+1.47
Avg41.8046.2748.3552.25+3.90

Recovering AVSPO's positive improvement under its original data-selection procedure supports the correctness of our implementation.

DBB reproduction.

Following DBB's original four-epoch protocol, our Qwen3-8B results broadly reproduce the reported performance across nine benchmarks:

BenchmarkOur DBB (%)Published DBB (%)Difference (pp)
MATH50089.1888.71+0.47
MinervaMath42.3039.46+2.84
AIME2430.5833.60−3.02
AIME2527.9229.02−1.10
AMC2464.0060.54+3.46
OlympiadBench58.7756.53+2.24
MMLU-Pro66.0163.12+2.89
GPQA-Diamond49.1249.43−0.31
BBH64.4567.09−2.64
Avg54.7054.17+0.53

Our DBB implementation averages 54.70%, compared with 54.17% for the nine published benchmark scores (+0.53 pp). These original-protocol reproductions support the correctness of both baseline implementations; AVSPO's weaker performance in our main setting may reflect algorithm-independent differences in model initialization, training data, or training horizon. Our main comparison retains a shared two-epoch budget.

Dr. GRPO.

Dr. GRPO addresses normalization bias by removing standard-deviation and response-length normalization while retaining group-mean centering: $ A_i=r_i-\frac{1}{G}\sum_j r_j $. When all rewards in a group are identical, every reward equals the group mean, and all residuals vanish. Thus, Dr. GRPO does not recover nonzero advantages for zero-variance binary-reward groups.

Recovery controls and the fixed-strength ablations follow in Part 3.

AuthorsOpenReview ↗

Response to Reviewer NsDE(3/4)

Q1 (continued)

Response: We report the recovery controls below.

Fixed-sign and AERO-style recovery.

We directly evaluated fixed-sign recovery and Beta(1,1) posterior-mean endpoint recovery. The AERO-style endpoint control retains RAC's mixed-group calibration rule and changes only the endpoint recovery mechanism:

Advantage ruleEndpoint magnitudeAcc. (%)
GRPO-pop066.02 ± 0.20
Fixed sign, Ai=2ri−1A_i=2r_i-1, in every group164.10 ± 1.16
AERO-style Beta(1,1) endpoints; RAC mixed-group rule≈0.333366.05 ± 0.39
RAC-GRPO≈0.037668.62 ± 0.50

RAC outperforms both tested controls, including a 2.57 pp gain over the AERO-style endpoint control. This comparison isolates the effect of endpoint recovery strength while retaining the same mixed-group rule.

Full AERO comparison.

AERO additionally changes rollout allocation through rescue and rejection sampling. We evaluated the complete AERO pipeline under an approximately 200 GPU-hour budget. The comparison is:

MethodAcc. (%)Optimization stepsGPU-hGenerated tokensWall-clock time (h)
AERO57.83 ± 1.31161200.172500.46M25.461
RAC-GRPO68.62 ± 0.50542181.191.115B22.65

AERO's three run-level accuracies are 56.37%, 58.19%, and 58.93%. RAC achieves 10.79 pp higher Avg, with fewer GPU-hours and shorter wall-clock time under comparable compute budgets.

AERO completes 161 optimization steps, whereas RAC-GRPO completes 542 steps. This difference reflects the higher per-step cost of AERO's adaptive rollout allocation in our experimental setting. In contrast, RAC performs calibration through a precomputed lookup table without introducing additional rollout generation or model forward passes. This comparison evaluates end-to-end performance under comparable compute budgets.


Q2: What count dependence adds beyond a small fixed pseudo-count

Response: The fixed-pseudo-count settings are fixed-strength ablations of RAC's calibration construction. They use the same advantage formula with a constant pseudo-count; GRPO is our main baseline. In particular, c=0.01c=0.01 was included with reference to RAC's derived endpoint pseudo-count of approximately 0.01130.0113. At G=8G=8, its endpoint advantage magnitude is $ \sqrt{\frac{0.01}{8.01}}\approx0.0353 $, which is close to RAC's derived endpoint magnitude of approximately 0.03760.0376. Thus, fixed c=0.01c=0.01 already provides nearly the same small endpoint-recovery strength as RAC, explaining its strong performance.

RAC further improves Avg by 1.03 pp over uniform c=0.01c=0.01. To evaluate the additional benefit of adapting the calibration strength to the observed success count, we conducted an endpoint-matched comparison. We fixed the endpoint advantages to RAC's original values (approximately ±0.0376\pm0.0376) and varied only the mixed-group pseudo-count.

Mixed-group rule, with RAC endpoints fixedRun 1Run 2Run 3Avg (%)
Fixed cint=0.5c_{\mathrm{int}}=0.567.8067.7168.5568.02 ± 0.46
RAC count-dependent rule68.2168.4669.1868.62 ± 0.50
Paired gain (pp)+0.41+0.75+0.63+0.60

We screened cint∈0,0.0113,0.1,0.5,1,2c_{\mathrm{int}}\in\\{0,0.0113,0.1,0.5,1,2\\} with one training seed, selected cint=0.5c_{\mathrm{int}}=0.5 using an independent validation set, and conducted two additional matched-seed runs. Both methods use identical endpoint advantages, normalization, numerical stabilization, and remaining training settings.

With endpoint recovery held fixed, RAC improves Avg by a further 0.60 pp over the validation-selected fixed interior correction, with positive gains on all three matched seeds. This directly demonstrates the additional benefit of adapting the mixed-group correction to the observed success count. The fixed-interior control also exceeds uniform c=0.01c=0.01 by 0.43 pp, supporting different calibration strengths for mixed and zero-variance groups.

We also evaluated a smaller uniform pseudo-count to determine whether an arbitrarily small correction is sufficient:

Calibration ruleAcc. (%)
GRPO-pop66.02 ± 0.20
Uniform c=0.001c=0.00165.92 ± 0.53
Uniform c=0.01c=0.0167.59 ± 0.52
RAC68.62 ± 0.50

Uniform c=0.001c=0.001 is 0.10 pp below the corrected GRPO-pop reference, whereas c=0.01c=0.01 is 1.57 pp higher. Together, these results show that the effectiveness of fixed calibration depends on its magnitude, while adapting the calibration strength to the observed success count provides additional gains beyond a fixed correction.

The additional complexity is negligible: for G=8G=8, RAC uses only a precomputed nine-entry lookup table, without additional rollouts, model evaluations, reward history, or a critic.

Part 4 covers training diagnostics and compute cost.

Reviewer_1yvUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RAC-GRPO replaces the empirical binary-reward statistics in group-relative advantages with a success-count-dependent pseudocount rule. It gives small updates to homogeneous groups and recalibrates mixed groups without extra rollouts or historical state. The paper derives the lookup from an estimation-risk criterion, analyzes idealized gradient and noise effects, and compares the complete method with GRPO, AVSPO, DBB and fixed-pseudocount alternatives. Component and endpoint-strength studies, two group sizes and a GSPO experiment examine the implemented design.

Critical strengths
  1. Low-overhead advantage calibration. The method adjusts advantages through a precomputed lookup table, without additional rollouts, historical statistics or a learned critic. It improves average accuracy over GRPO, AVSPO and DBB on Qwen3-1.7B, 4B and 8B. Applying the calibration to GSPO yields a 1.80-point improvement.

  2. Component and correction-strength analysis. The ablations quantify the contributions of endpoint recovery and mixed-group calibration. On Qwen3-8B, the full method reaches 68.62, compared with 67.59 for the best fixed pseudocount in the experiments. Figure 3 shows how performance changes with endpoint recovery strength.

Critical weaknesses
  1. Limited significance and unsubstantiated value for large-scale RL. The contribution is an incremental calibration of group reward statistics, with a 1.03-point improvement over the best fixed pseudocount in the Qwen3-8B experiments. The paper does not establish that recovering all-correct or all-incorrect groups addresses an important bottleneck when large training batches already contain many informative prompts. It also leaves the value of this correction relative to difficulty-aware data selection and curriculum adjustment unresolved as model capability increases. The experiments use models up to 8B, a prompt batch size of 64 and group sizes of 4/8, leaving the effect of larger prompt batches and changing task difficulty unexamined. The risk criterion constrains success-probability estimation error; the link to practical training gains remains unclear.

  2. Missing comparisons with stronger sampling and advantage-estimation methods. The evaluation omits dynamic filtering (DAPO), adaptive rollout allocation (AERO), batch-level normalization (REINFORCE++) and tree-sampling methods [1][2][3][4]. These approaches address the use of rollout computation and the construction of informative advantages. The central practical question is whether c_K calibration offers greater benefit than filtering, resampling or improving the sampling structure. Comparisons under matched total compute budgets should report final accuracy, generated tokens and training time. The paper also needs to measure the incremental benefit of adding c_K calibration to a stronger sampling or filtering pipeline. Without this evidence, its competitive value and training efficiency remain unestablished.

References

[1] TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling.

[2] TreeRPO: Tree Relative Policy Optimization.

[3] Tree Search for LLM Agent Reinforcement Learning.

[4] Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO.

Questions for authors
  1. What evidence shows that homogeneous groups remain a material training bottleneck as model capability and prompt batch size increase? How do the gains from c_K calibration depend on the proportion of easy, intermediate and difficult prompts?

  2. Under matched total compute budgets, how does RAC-GRPO compare with representative dynamic-filtering, adaptive-sampling and tree-based advantage-estimation methods in final accuracy and training time? Does c_K calibration retain a meaningful benefit when stronger sampling or filtering methods are already used?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1yvU(1/3)

Thank you for recognizing RAC-GRPO's low overhead and component analyses. We first clarify the problem RAC addresses. In group-based RL with binary verifiable rewards, each prompt has a finite number of sampled responses. When all responses in a group are correct or all are incorrect, standard GRPO assigns zero advantages to the entire group, even though these outcomes do not establish that the prompt is always solvable or unsolvable. RAC calibrates the observed success count within each prompt's rollout group, recovering small updates for homogeneous groups and adjusting advantages for mixed groups. It operates on already sampled responses and requires no additional rollouts, prompt history, or critic. It therefore addresses a different part of the training pipeline from methods that change prompt selection, rollout allocation, or sampling structure, and can be combined with them.

Results use Qwen3-8B. Accuracy is the unweighted seven-benchmark mean ± sample SD over three seeds; gains are in percentage points (pp).


W1 & Q1: Practical significance, larger prompt batches, and prompt difficulty

Response: RAC addresses finite-group statistics at the prompt level. We show that homogeneous groups remain common at larger prompt batch sizes, identify where the accuracy gains occur across prompt difficulties, and quantify the contribution of adaptive calibration beyond fixed-strength alternatives.

Larger prompt batches.

RAC calibrates the reward statistics within each prompt's rollout group. Increasing the number of prompts in a batch does not increase the number of responses used to estimate the success probability of an individual prompt. For a fixed policy and prompt with success probability pp, the probability of obtaining a homogeneous group is$P(\mathrm{homogeneous})=p^G+(1-p)^G$, which depends on the number of rollouts per prompt, GG, rather than the number of prompts in the batch. A larger batch may provide more informative groups overall, but it does not change the zero-advantage assignment to a homogeneous group under standard GRPO. This is why we also tested RAC with a larger prompt batch.

At prompt batch size 256, we counted 34,304 response groups from training steps 1–134:

Group typeRAC groupsShare (%)
All-incorrect5,01914.63
All-correct14,19941.39
Homogeneous, total19,21856.02

The corresponding homogeneous-group proportion for GRPO is 56.45%. Thus, homogeneous groups remain common even at prompt batch size 256.

We also increased the prompt batch size from 64 to 256, retained eight rollouts per prompt, and trained both methods for two epochs:

Method at prompt batch size 256Acc. (%)GPU-hGenerated tokens
GRPO60.19 ± 0.77112.970.969B
RAC-GRPO61.54 ± 0.47110.480.937B

RAC improves accuracy by 1.35 pp, with gains of 1.16, 1.67, and 1.22 pp across the three seeds, while using fewer GPU-hours and generated tokens. Its three-seed mean is higher on every benchmark. Together with the model-scale results below, these experiments demonstrate that RAC remains effective as both prompt batch size and model capability increase.

Prompt difficulty.

RAC calibrates reward statistics after prompts are selected, making it complementary to difficulty-aware selection and curriculum adjustment. Our difficulty analysis shows that RAC's largest gain is on difficult problems (+1.92 pp), while both methods already achieve high accuracy on intermediate and easy problems.

To examine where the gains occur, we assigned 1,576 test problems to fixed difficulty categories using 32 responses per problem from the original Qwen3-8B model: 0–8, 9–24, and 25–32 correct responses. Both trained methods were evaluated on the same categories using eight responses per problem.

DifficultyProblemsGRPO (%)RAC (%)Gain (pp)
Difficult50729.29 ± 0.8831.21 ± 0.33+1.92
Intermediate27790.48 ± 0.4590.76 ± 0.09+0.28
Easy79299.22 ± 0.0999.29 ± 0.10+0.07

These accuracies are response-weighted within each category, whereas the main metric averages seven benchmarks equally. The results show that RAC's gains are concentrated on difficult problems, where both methods have more room for improvement.

An all-incorrect sampled group does not establish that a prompt is unsolvable, just as an all-correct group does not establish that it is always solved. This finite-sampling distinction motivates calibrating their advantages. Small recovery magnitudes also limit stochastic updates on already saturated prompts, as analyzed in Section 4.4.

The next response discusses calibration strength and the expected policy update.

AuthorsOpenReview ↗

Response to Reviewer 1yvU(2/3)

W1 & Q1 (continued): Calibration strength

Fixed-strength calibration.

GRPO is our main baseline. Fixed-pseudocount settings are ablations of RAC's calibration construction, rather than independent competing methods. On Qwen3-8B, RAC achieves 68.62%, compared with 67.59% for the best fixed-pseudocount setting, a difference of 1.03 pp. This difference measures the additional benefit of adaptive calibration over a strong fixed-calibration variant; RAC's full improvement over GRPO is 2.82 pp.

Under a common rollout budget, RAC improves over GRPO by 2.53, 2.08, and 2.82 pp on Qwen3-1.7B, 4B, and 8B, respectively.

The fixed value c=0.01c=0.01 was included with reference to RAC's derived endpoint pseudocount of 0.0113 (Figure 2 and Table 6). It therefore already provides nearly the same homogeneous-group recovery magnitude as RAC: approximately 0.0353, compared with RAC's 0.0376. Its strong performance is consistent with the fact that it already captures almost the same endpoint-recovery effect.

To isolate the contribution of count dependence, we conducted an additional endpoint-matched control. We fixed the homogeneous-group advantages to RAC's original values and varied only the pseudocount used for mixed groups. The count-dependent schedule adds 0.60 pp over the validation-selected fixed value, with positive differences on all three matched seeds. The fixed-interior control also exceeds uniform c=0.01c=0.01 by 0.43 pp, supporting different calibration strengths for mixed and homogeneous groups. The comparison is:

Mixed-group calibrationHomogeneous-group advantagesAcc. (%)
Fixed cint=0.5c_{\mathrm{int}}=0.5RAC values68.02 ± 0.46
RAC count-dependent ruleRAC values68.62 ± 0.50

We screened cint∈0,0.0113,0.1,0.5,1,2c_{\mathrm{int}}\in\\{0,0.0113,0.1,0.5,1,2\\} with one seed using the independent validation set, then evaluated the selected value on two additional matched seeds. The paired differences are +0.41, +0.75, and +0.63 pp. At eight rollouts per prompt, the adaptive rule requires only a nine-entry precomputed table.

The MSE criterion.

RAC does not minimize MSE; MSE only limits calibration strength. We first calibrate the current group's empirical reward statistics, then use the MSE constraint to bound the allowed correction. For each observed success count KK, the calibrated estimate uses a pseudo-count cK≤2c_K\leq 2:p~K=K+cKG+2cK\tilde p_K=\frac{K+c_K}{G+2c_K}.

We select the largest value whose fixed-cc estimator has no greater pointwise MSE than the empirical MLE K/GK/G throughout the 95% equal-tailed credible interval of the reference posterior Beta(K+1,G−K+1)\mathrm{Beta}(K+1,G-K+1). Thus, the MSE criterion determines the admissible strength of the correction.

Under independent on-policy sampling, a binary verifier independent of the policy parameters, standard score-function regularity conditions, and stop-gradient advantages, Proposition 1 establishes $ \mathbb{E}[\hat g] = \alpha_{\mathrm{RAC}}(p)\nabla_\theta p, \alpha_{\mathrm{RAC}}(p)>0. $

This establishes the expected update direction under the stated assumptions. The matched GRPO controls and experiments above assess the resulting practical training gains, while the MSE constraint controls the admissibility of the probability calibration.


W2 & Q2: Relationship to stronger sampling and advantage-estimation methods

Response: The cited methods do not all address the same training setting or operate at the same level. Dynamic filtering and adaptive sampling change which responses are generated or retained; batch-level normalization changes how training weights are constructed; tree-based methods change the sampling structure and often construct step- or thought-level learning signals. RAC instead calibrates prompt-level binary-reward statistics after sampling. These approaches are therefore not necessarily substitutes. RAC can be added to compatible training pipelines, and we directly test this with DAPO.

The requested compute-budget comparisons follow in Part 3.

AuthorsOpenReview ↗

Response to Reviewer 1yvU(3/3)

W2 & Q2 (continued)

Response: The requested method comparisons are reported below.

DAPO: Filtering and calibration are complementary.

DAPO's dynamic sampling discards homogeneous groups and continues sampling to fill the training batch with informative groups. RAC takes a different approach: it assigns small calibrated advantages to already sampled homogeneous groups and also calibrates mixed groups. Thus, RAC can complement filtering by calibrating the mixed groups retained by DAPO.

Adding RAC after dynamic filtering improves accuracy by 1.62 pp.

We applied RAC after dynamic filtering in the same binary-reward DAPO pipeline, under approximately 200 training GPU-hours per run:

Filtering pipelineAcc. (%)Generated tokensWall-clock time (h)
DAPO64.69 ± 0.321.578B25.78
DAPO + RAC66.31 ± 0.451.589B25.77

The paired gains are 1.53, 1.78, and 1.57 pp, with nearly identical training time and approximately 0.70% more generated tokens. Since DAPO has already removed homogeneous groups, this experiment directly demonstrates the additional benefit of calibrating the retained mixed groups. Token counts include discarded rollouts; time includes evaluation and checkpoint saving.

AERO: Adaptive sampling and calibration operate at different stages.

AERO allocates additional rollouts to rescue low-success prompts, curates responses through rejection sampling, and uses fixed Beta(1,1) posterior means for residual homogeneous groups. RAC instead calibrates existing groups across observed success counts without an additional rescue-sampling stage.

RAC achieves 10.79 pp higher accuracy in our full-pipeline comparison under comparable GPU-hour budgets.

MethodAcc. (%)GPU-hGenerated tokensWall-clock time (h)
AERO57.83 ± 1.31200.172500.46M25.461
RAC-GRPO68.62 ± 0.50181.191.115B22.65

The different sampling strategies produce different token counts under comparable GPU-hour budgets. The table compares the complete pipelines' end-to-end accuracy and computational cost.

REINFORCE++: Batch-level normalization.

REINFORCE++ normalizes advantages across prompts in the batch. Our evaluated configuration uses one rollout per prompt, whereas RAC retains group-based estimation with multiple rollouts per prompt. The two methods therefore use different sampling configurations.

Under approximately 200 training GPU-hours, DAPO + RAC achieves 2.13 pp higher accuracy than REINFORCE++.

This is a full-pipeline reference with the sampling configurations shown below:

MethodPrompt batch × rollouts per promptAcc. (%)Generated tokensWall-clock time (h)
REINFORCE++256 × 164.18 ± 0.941.214B26.77
DAPO + RAC64 × 866.31 ± 0.451.589B25.77

The DAPO versus DAPO + RAC comparison above provides the controlled evidence for RAC's incremental contribution within the same pipeline.

Tree-based methods: Different sampling structures and credit-assignment levels.

The four cited tree-based methods primarily improve rollout efficiency, exploration, or finer-grained credit assignment:

TreePO uses uncertainty-guided tree sampling with segment-level advantage estimation. TreeRPO uses tree sampling to estimate reward expectations at intermediate reasoning steps and constructs step-level groups. Tree Search for LLM Agent Reinforcement Learning organizes multi-turn agent interactions into a search tree, shares common prefixes, and derives step-wise learning signals from outcome rewards. Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO studies thought-level advantage estimation and shows how branching into multiple continuations can reduce estimation variance.

These methods change how responses are sampled or how learning signals are constructed, whereas RAC addresses finite-sample calibration of prompt-level binary-reward statistics. They are therefore complementary directions rather than direct replacements for RAC. When a tree-based pipeline retains compatible prompt-level binary-reward groups, RAC can be added as a post-sampling calibration step. For methods that construct advantages at the segment, step, or thought level, the calibration unit would need to be aligned with that level.

Given the available computational resources, we prioritized the requested comparisons with dynamic filtering (DAPO), adaptive rollout allocation (AERO), and batch-level normalization (REINFORCE++), and evaluated RAC within the DAPO pipeline. These experiments report accuracy, generated tokens, and training time under comparable compute budgets, while the DAPO + RAC control directly measures calibration's incremental benefit.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We thank the reviewers for their constructive feedback. We have uploaded a revised PDF incorporating the additional experiments and clarifications from the discussion. All additions and substantive edits are highlighted in blue. The core method, theoretical analysis, and original experimental results are retained.

1. Controlled attribution of gains (Section 5.3, Table 3).

We add a population-normalized GRPO control (GRPO-pop) and an endpoint-matched fixed-interior control. Under matched normalization, mixed-group calibration contributes 1.06 percentage points, and endpoint recovery adds another 1.54 points, improving the average from 66.02% to 68.62%. With endpoint advantages fixed, the count-dependent rule further improves over the fixed-interior control from 68.02 ± 0.46% to 68.62 ± 0.50%, with positive gains on all three matched seeds. Appendix C.5 documents the control selection protocol.

2. Integration with dynamic filtering (Section 5.5, Appendix F.1).

We add a DAPO versus DAPO + RAC comparison. RAC improves average accuracy from 64.69 ± 0.32% to 66.31 ± 0.45%, with positive gains across all three matched runs and nearly identical wall-clock time, demonstrating its benefit after homogeneous groups have been filtered.

3. Larger batches, model scale, and difficulty (Section 5.5, Appendices F.2–F.3).

At prompt batch size 256, RAC improves the three-seed average by 1.35 points. On Qwen3.8-27B, one-epoch LoRA training (rank 128) improves the average from 76.46% to 77.81%, with gains on all seven benchmarks. Difficulty-stratified results show the largest gains on difficult problems at both model scales.

4. Sampling comparisons and additional diagnostics (Appendices F.1, F.4–F.6).

We add full-pipeline comparisons with AERO, REINFORCE++, and TreePO. Under the approximately 200 GPU-hour budget-control protocol, RAC achieves 68.62 ± 0.50%, compared with 61.97 ± 0.92% for TreePO, using 181.19 versus 203.70 GPU-hours. We also include additional recovery controls, an AERO-style Beta(1, 1) endpoint reference, group-composition statistics, endpoint gradient measurements, end-to-end costs, and AVSPO/DBB reproduction checks.

5. Reproducibility and presentation (Appendix C.5 and Figure 3).

We provide the complete validation protocol, including the 2,821-problem Omni-MATH-Rule validation set, evaluation every 20 steps, checkpoint selection by validation accuracy, and the 95% reference credible level fixed before experimentation. The seven test benchmarks remain excluded from checkpoint selection and hyperparameter tuning. We also add relevant citations, clarify the balanced-group reference, improve figure and table readability, and retain the full fixed-strength and G=4/GSPO benchmark tables in Appendix E.1, with their analyses preserved in the main text.

We appreciate the reviewers' suggestions, which have helped us further clarify the contribution and strengthen its empirical support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.