acceptodds
Under review as a conference paper at ICLR 2027

Calibrated Bernoulli Policy Optimization for Single-Sample RLVR

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as the core paradigm for enhancing reasoning capabilities in Large Language Models (LLMs). Currently, multi-sample algorithms like GRPO dominate this field despite their prohibitive computational costs. While single sample sequence level methods offer a theoretically efficient alternative, they suffer from notorious training instability. This paper reveals the mechanism underlying the empirical robustness of GRPO: at the infinite sample limit, it implicitly approximates Bernoulli variance normalization. However, due to the inherent calibration deficit of critic networks, direct introduction of this variance normalization triggers training instability. To address this, we propose Calibrated Bernoulli Policy Optimization (CBPO). By integrating Expected Calibration Error (ECE) regularization and lightweight L-BFGS temperature scaling, CBPO aligns the critic probability estimations. This calibration enables stable analytical variance normalization, yielding Fisher information aware advantage scaling without the need for multi-sampling. Extensive experiments demonstrate that CBPO matches the training stability and SOTA performance of multi-sample baselines, requiring only a single sample computational budget.

Primary area:
reinforcement learning
Keywords:
RLVR, Policy Optimization, Probability Calibration, Bernoulli, Single-Sample

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CKuXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CBPO reduces rollout cost by replacing empirical group statistics with a learned success-probability critic. For binary rewards, the critic provides both the reward baseline and the Bernoulli variance used to normalize advantages. The method combines this normalization with an ECE-style training penalty and repeated temperature scaling. The analysis connects the resulting residual to the large-group limit of GRPO and characterizes how probability errors affect its mean and variance. Experiments on mathematical reasoning benchmarks report competitive accuracy at lower training cost, with component ablations, equal-budget comparisons, and three-seed results for one model setting.

Critical strengths
  1. The critic controls both advantage centering and scale. An inaccurate prediction near a probability boundary can therefore affect both quantities. The paper derives this dependence explicitly and uses it to motivate calibration, giving the method a clear statistical rationale.
  2. The equal-GPU-hour comparison directly addresses the intended efficiency benefit. The component ablations further examine the contributions of normalization, ECE regularization, and temperature scaling. Together, these experiments provide useful evidence for the proposed combination.
Critical weaknesses
  1. On Qwen3-8B without thinking, Table 1 reports Avg scores of 49.0 for CBPO and 46.2 for GRPO, a difference of 2.8 points. In Table 2, their three-seed means are 48.50 and 47.29, reducing the difference to 1.21 points. The repeated evaluation covers only this model setting and excludes DAPO, a strong baseline in the main comparison. The seed and checkpoint selection rules are therefore important to interpreting the reported gains. The efficiency comparison also depends on the baseline configuration. The equal-budget experiment uses eight rollouts per prompt for GRPO, while CBPO uses one. It remains unclear whether CBPO retains the same advantage against smaller GRPO group sizes under the same compute budget. These comparisons are needed to determine how much of the observed benefit persists across training runs and reasonable baseline configurations.
  2. The theoretical bounds require controlled probability error and predictions away from the boundaries. The paper acknowledges that its calibration procedures do not guarantee these conditions. Appendix F reports minimum advantages of approximately −9,986 despite calibration. It also provides gradient-norm traces, but does not establish how these events affect subsequent held-out performance or how their frequency compares with GRPO and SPPO across independent runs. This matters because stable single-rollout training is a central motivation for CBPO. Lower aggregate calibration error and competitive final scores leave open whether the method consistently controls rare, large updates. The stability claim needs direct evidence connecting these events to training outcomes.
Questions for authors
  1. How were the runs and checkpoints in Table 1 selected? How do they relate to the runs in Table 2, and were any runs excluded from the reported comparisons?
  2. Can you provide individual-seed results and uncertainty estimates for the performance differences between CBPO and the strongest relevant baselines, including DAPO, under a common selection and compute-budget protocol?
  3. Were smaller GRPO group sizes evaluated at the same GPU-hour budget? How does CBPO compare with the strongest budget-matched configuration?
  4. What happened to gradient norms and subsequent held-out performance around the extreme-advantage events in Appendix F? How frequently did comparable events or training failures occur for CBPO, GRPO, and SPPO across independent runs?
  5. Could you compare CBPO with GRPO on a downstream agentic RL task involving very long trajectories and multiple rounds of environment interaction? Under matched training budgets and comparable tuning effort, please report task success, variation across independent training seeds, and the frequency of extreme advantages, gradient spikes, and training failures. This would help assess whether the observed efficiency gains remain useful when training requires sustained interaction with a changing environment.
4 comments

Dear Reviewer CKuX,

Your close reading of our experimental protocol, baseline configurations, and training stability exposed gaps in the evidence that we could not see ourselves, and we are grateful for the time and care it took.


Weakness1:

Both tables use the same Avg metric and selection rule, and each Table 1 run is the seed 1 run of Table 2. The two margins differ only because of seed variance, since GRPO's seed 1 run, at 46.2, lies below its three seed mean of 47.29.

We ran GRPO with G = 4, using 256 prompts × 4 rollouts, and G = 2, using 128 prompts × 2 rollouts to match CBPO's 256 responses per step, on Qwen3-8B without thinking. The table reports each configuration at 100 GPU hours, as in Table 6, and after 500 steps (seed 1), where Steps @100 GPU-h is the number of steps completed within 100 GPU hours.

MethodPrompts × GGPU-hours, 500 stepsSteps @100 GPU-hAvg @100Pass @100Hard Avg @100Hard Pass @100AvgPassHard AvgHard Pass
GRPO256 × 8480.23~10440.7167.2727.5559.3346.273.033.266.7
GRPO256 × 4301.5~16641.6268.0328.4160.1245.972.532.866.0
GRPO128 × 2138.9~36042.4868.7629.1760.8545.171.632.064.4
SPPO256 × 1169.76~29544.1270.0131.1763.1446.772.433.966.3
CBPO256 × 1174.75~28645.6472.8332.9866.1549.076.536.670.8

At 100 GPU hours. As the reviewer anticipated, smaller groups help GRPO because cheaper steps allow more updates, and GRPO-2 becomes its strongest configuration at 1.77 Avg above GRPO-8. CBPO still leads GRPO-2 by 3.16 Avg, because the calibrated critic gives every single response a reliable baseline, letting CBPO spend its budget on more distinct prompts, and the Bernoulli normalization keeps hard prompts at full weight.

After full training. Smaller groups lose accuracy, with Avg falling from 46.2 at G = 8 to 45.1 at G = 2, because fewer responses give a noisier contrast and more groups with identical rewards. CBPO exceeds GRPO-8 by 2.8 Avg with 36% of its GPU hours and GRPO-2 by 3.9 Avg with the same responses per step, so the gain comes from replacing noisy group statistics with calibrated probabilities, the role of online ECE regularization and temperature scaling. SPPO, which also uses one rollout but lacks these components, trails CBPO by 2.3 Avg at almost the same cost, which confirms this.


Weakness2:

The reviewer is right that the bounds require predictions away from the boundaries, and the Appendix F events are exactly where this fails, for a numerical reason. All five steps below −100 have the identical value −9,986.44, which follows from A^=(r−s)/s(1−s)+ϵ\hat A=(r-s)/\sqrt{s(1-s)+\epsilon}. For an incorrect response, r=0r=0, and

\begin{cases} -\sqrt{\frac{s}{1-s}}, & s < 1, \\\\ -\frac{1}{\sqrt{\epsilon}}, & s = 1. \end{cases}$$ For $s<1$ the advantage is $-\sqrt{s/(1-s)}$, and for $s=1$ only $\epsilon$ remains in the denominator. In bf16, probabilities just below 1 lie on the grid $s=1-k\cdot2^{-8}$, and $k=1,2,3$ give −15.97, −11.27, and −9.18, almost identical to the other observed minima −16.0, −11.29, and −9.16. The next grid point, $s=1$, gives $-1/\sqrt{\epsilon}\approx-10^{4}$, almost identical to −9,986.44, so each event is an incorrect response whose probability was rounded to exactly 1.0. Because the cause is rounding rather than a failure of calibration, the events are rare and leave no lasting effect. They occur at 5 of 1,500 steps, gradient clipping keeps each within an ordinary update, and both the gradient norm and MATH500 accuracy are normal at the next step and evaluation. No new run in this rebuttal has an advantage below −100, and clamping $s$ to $[10^{-3},1-10^{-3}]$, a safeguard we add to the released code, bounds $|\hat A|$ by 31.6. Q4 gives the detailed statistics.

Question1 and 2:

Selection rule. Each Table 1 entry is a single seed 1 run, reported at the checkpoint with the highest six benchmark Avg among evaluations every 5 steps, with the same rule for all methods. Table 2 repeats this with seeds 1–3, so the Table 1 runs are its seed 1 runs, and no run or seed was excluded.

Individual seeds and uncertainty. We added three DAPO seeds under the same protocol, and Table R1 reports every seed with the mean and standard deviation.

Table R1. Individual seeds at the best checkpoint, Qwen3-8B without thinking.

MethodSeed 1 AvgSeed 2 AvgSeed 3 AvgAvg, mean ± SDPassHard AvgHard Pass
PPO45.6046.1844.8745.55 ± 0.6670.79 ± 1.1632.25 ± 0.8363.59 ± 1.26
GRPO46.2048.0647.6147.29 ± 0.9774.06 ± 0.9634.51 ± 1.1968.05 ± 1.19
SPPO46.7047.8447.2747.27 ± 0.5773.30 ± 0.8534.48 ± 0.6467.21 ± 0.86
DAPO47.7048.2846.5847.52 ± 0.8674.61 ± 1.0434.72 ± 1.0268.66 ± 1.21
CBPO49.0148.5747.9248.50 ± 0.5575.52 ± 0.8835.83 ± 0.7469.45 ± 1.16

CBPO has the highest mean on every metric, leading DAPO by 0.98 Avg, and the smallest seed variance. Paired by seed, it wins all 12 comparisons against PPO, GRPO, SPPO, and DAPO, so the gain does not depend on a favorable seed. It also ranks first at the final checkpoint, with 47.96 Avg against 46.88 for DAPO, so the result is independent of the selection rule. We will state the rule in Section 4.1 and report both protocols.


Question3:

We evaluated GRPO with smaller groups under the same budget, and the full table is in our response to W1. At 100 GPU hours, G = 2 is the strongest GRPO configuration at 42.48 Avg, and CBPO reaches 45.64, ahead of it by 3.16 and of SPPO, the strongest budget matched baseline, by 1.52. After 500 steps (seed 1), CBPO leads every GRPO configuration by 2.8–3.9 Avg.


Question4:

We traced all five archived events in the training logs, and none of them had a lasting effect. At each event the gradient norm before clipping jumps to between 85 and 1,001, but clipping caps the update at norm 1.0, and at the very next step the norm is back to 0.8–1.3 and the minimum advantage to between −1.1 and −2.4. Held out MATH500 accuracy is barely affected, rising after three events, for example from 0.734 to 0.760, and changing by at most 0.014 after the other two, as from 0.872 to 0.864, which is within the normal variation between evaluations.

We also counted the steps whose most negative advantage or gradient norm exceeds ten times its run median. Every method was trained in three runs of 500 steps each, 1,500 training steps in total. As Table R2 shows, the baselines have one or two of each, the archived CBPO runs have five, the new CBPO runs have one, and no run diverged.

Table R2. Extreme advantage steps and gradient spikes over all training steps.

MethodExtreme advantage stepsGradient spikes
GRPO10
DAPO12
GSPO21
RLOO10
PPO22
REINFORCE++21
CBPO55
CBPO, new runs in this rebuttal11

Question5:

We trained Qwen2.5-7B-Instruct on ALFWorld and WebShop with the GiGPO protocol [1], with binary task success and trajectories of several thousand tokens over up to 50 environment steps. All methods use the same tasks per step and the same iterations, but GRPO and GiGPO sample eight trajectories per task and CBPO one. The baselines use GiGPO's tuned configurations while CBPO reuses its math hyperparameters, so they receive at least comparable tuning. Results are over three seeds.

MethodALFWorld AllWebShop ScoreWebShop SuccessSeed std, ALFWorld / WebShop Success
Prompting [1]14.826.47.8–
GRPO [1]77.679.366.15.2 / 3.7
GiGPO [1]90.884.472.81.5 / 2.9
CBPO93.287.576.91.2 / 2.1

With one eighth of the trajectories, CBPO improves over GRPO by 15.6 points on ALFWorld and 10.8 on WebShop success, and exceeds GiGPO, which adds step level credit assignment, by 2.4 and 4.1.

CBPO also has the smallest seed variance, 1.2 and 2.1 points against 5.2 and 3.7 for GRPO. Across all runs, no method has an advantage beyond 100 in magnitude or a failed run, and each method has a single gradient spike, so with one trajectory per task CBPO trains as stably as the group baselines.

The efficiency gain thus carries over to long interaction with a changing environment, where each trajectory needs dozens of environment calls and CBPO removes seven of every eight rollouts.


References:

[1] Group-in-Group Policy Optimization for LLM Agent Training. arXiv:2505.10978.

The revised manuscript is now available on OpenReview, and we would like to briefly summarize how it reflects your review. Your observation of the rare large updates in our training logs led us to trace their source and give a clearer account of the training stability of CBPO, and the changes related to your comments are as follows.

  • Seeds and checkpoint selection (W1, Q1, Q2). Table 1 is now marked as seed 1, and Appendix E states the selection rule, which applies the same criterion to every method and excludes no run or seed. The comparison across three seeds in Section 4.2 now includes DAPO, and Appendix E reports Pass metrics and the final checkpoint at step 500 (Table 12), where CBPO keeps the highest mean on every metric.
  • Smaller GRPO groups (W1, Q3). Section 4.6 adds GRPO with groups of 2 and 4 alongside 8 and 16 (Table 5), at 100 GPU hours and after full training. Smaller groups help GRPO at a fixed budget, yet CBPO leads GRPO-2 by 3.16 Avg with the same 256 responses per step.
  • Rare large updates (W2, Q4). Appendix I traces every extreme advantage to bf16 rounding of a critic probability to exactly one (Eq. 18), reports the gradient norms and held out accuracy around each event, and counts comparable events for all methods (Table 18). The events occur at 5 of 1,500 steps, gradient clipping bounds each update, and clamping the critic probability, now included in the released implementation, bounds every advantage by 31.6.
  • Agentic tasks (Q5). Section 4.6 and Appendix G add ALFWorld and WebShop (Table 8), where CBPO outperforms GRPO and GiGPO with one trajectory per task and has the smallest variation across seeds.

We hope these revisions address your concerns, and we would be grateful if you could take them into account in your final assessment. We are happy to clarify any remaining questions.

Thank you for your clarification and repsonse. But I still have some concerns.

Reviewer_JYGDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CBPO, a single-rollout method that calibrates its critic with ECE-style regularization and temperature scaling, and uses the critic's predicted success probability in place of GRPO's group statistics to normalize advantages.

Critical strengths

The paper addresses a practically important problem: the high generation cost of group-based RLVR methods such as GRPO. The core observation is clear and intuitive. Under binary rewards, the mean of the reward determines its variance, so GRPO's group statistics can be viewed as estimating a single quantity, the prompt's success probability. This motivates replacing repeated sampling with a learned prompt-level critic. The experimental results are strong: across multiple model settings, CBPO matches or outperforms strong multi-sample baselines while using only one rollout per prompt and substantially less compute.

Critical weaknesses
  1. I suggest rerunning GRPO and DAPO with a larger group size (e.g., G = 16 or 32) instead of G = 8. G = 8 is below standard practice: DeepSeek-R1 and the original DAPO paper use 16, and the original GRPO used 64. Under binary rewards, a small G produces many all-correct or all-incorrect groups that give no learning signal, which weakens both baselines. Since CBPO is motivated as an approximation of GRPO's large-group limit, comparing against larger G is also the most direct test of the paper's main claim. Please report both accuracy and GPU-hours so the cost–accuracy trade-off is clear.

  2. The premise about why GRPO works is not established. CBPO assumes that GRPO is effective because its group statistics estimate each prompt's success probability, so a critic can replace the group. However, under binary rewards, the GRPO update of each group is exactly proportional to the difference between the average gradients of the correct and incorrect responses; the quality of the probability estimate only affects the weight of each group, not the update direction. Existing evidence also supports this view: GRPO with only two rollouts per prompt retains about 98% of the performance of 16-rollout GRPO, and Dr. GRPO reports that removing the standard-deviation normalization does not hurt performance. This suggests that within-prompt contrast, rather than accurate probability estimation, is the key ingredient, and CBPO removes it by design. I suggest comparing against 2-rollout GRPO at the same generation budget (128 prompts × 2 rollouts, matching CBPO's 256 responses per step), which requires no critic.

Questions for authors
  1. The paper is split into too many short subsections (e.g., seven in Section 3 and eleven in Appendix G, several only one paragraph long), which fragments the narrative; consolidating them would make the method and its motivation easier to follow.

  2. For other issues, please see the weaknesses listed above.

5 comments

Dear Reviewer JYGD,

Thank you for the time you spent on our paper and for your comments on the baselines, the theoretical analysis, and the organization of the text, which will make both the argument and its presentation stronger.


Weakness1:

We ran GRPO and DAPO with G = 16 on Qwen3-8B without thinking, with 256 prompts per step and all other settings unchanged. Table R1 reports full training (seed 1) and Table R2 an equal budget of 100 GPU hours.

G = 8 is the common setting at the 7B–8B scale, also used by SimpleRL-Zoo [1] and Dr. GRPO [2]. The larger values cited by the reviewer come from much larger budgets, with 64 responses per question in DeepSeekMath [3] and 512 prompts × 16 responses for a 32B model in DAPO [4]. Since cost grows roughly linearly in G, the group size is a cost–accuracy tradeoff, which is what CBPO is designed to improve.

Table R1. Full training for 500 steps, Qwen3-8B without thinking.

MethodGResponses / stepGPU-hoursAvgPassHard AvgHard Pass
GRPO82,048480.2346.273.033.266.7
GRPO164,096924.448.376.134.670.8
DAPO82,048584.4547.774.934.769.3
DAPO164,0961,125.249.176.536.270.9
CBPO1256174.7549.076.536.670.8

Table R2. Equal budget of 100 GPU hours, same setting.

MethodGSteps @100 GPU-hAvgPassHard AvgHard Pass
GRPO8~10440.7167.2727.5559.33
GRPO16~5439.6266.0526.4857.91
DAPO8~8640.8467.5227.6659.70
DAPO16~4439.8566.3126.7058.24
CBPO1~28645.6472.8332.9866.15

Both baselines improve with G = 16, by 2.1 Avg for GRPO and 1.4 Avg for DAPO, while their GPU hours almost double. DAPO-16 leads CBPO by only 0.1 on Avg and Hard Pass at 6.4 times the GPU hours, and CBPO ties it on Pass, leads it by 0.4 Hard Avg, and is ahead of or level with GRPO-16 on every metric. One rollout per prompt thus matches training with 16 rollouts at a fraction of the cost, which is the efficiency the calibrated critic and the Bernoulli normalization provide.

At 100 GPU hours, larger groups become a disadvantage. A G = 16 step costs about twice a G = 8 step, so GRPO-16 and DAPO-16 complete only about 54 and 44 steps against about 286 for CBPO, fall below their G = 8 versions, and trail CBPO by 5.79 Avg.

This trend follows Eq. (6), by which the group normalized advantage converges to the Bernoulli residual as G grows, so a larger G moves GRPO toward the target CBPO estimates directly, at a cost linear in G. Larger groups also still waste generation, since a group with identical rewards, and hence zero advantages, occurs with probability 75.0% at G = 2, 39.3% at G = 8, and still 28.0% at G = 16 when success probabilities follow the U-shaped distribution Beta(1/2, 1/2) typical of RLVR, whereas every CBPO response has a nonzero advantage.

Weakness2:

We agree with the identity, and CBPO preserves it. Because the critic depends only on the prompt, the expected CBPO update for one prompt is Eo[A^ ∇θlog⁡πθ(o∣q)]=p(1−p)D(s) (gˉ+−gˉ−),\mathbb E_o\big[\hat A\,\nabla_\theta\log\pi_\theta(o\mid q)\big]=\frac{p(1-p)}{\sqrt{D(s)}}\,\big(\bar g^{+}-\bar g^{-}\big), where gˉ+\bar g^{+} and gˉ−\bar g^{-} are the mean score functions of the correct and incorrect responses. This is the same correct minus incorrect contrast as in GRPO, and with a calibrated critic, s=ps=p, each prompt gets the weight p(1−p)\sqrt{p(1-p)} of the GRPO update in the large group limit. CBPO therefore does not remove the contrast. It estimates it across the batch with one response per prompt, and we will add this derivation to Appendix G.

Our view differs on why the probability estimate matters. In GRPO the group mean is the baseline, so the estimate only reweights groups. Without a group, the critic itself is the baseline, and a baseline error adds the variance term (s−b⋆)2E[∥∇θlog⁡πθ∥2]/D(s)(s-b^{\star})^2 \mathbb{E}[\lVert\nabla_{\theta}\log\pi_{\theta}\rVert^2]/D(s), with b⋆≈pb^{\star}\approx p the variance optimal baseline [5, 6]. An accurate estimate is therefore what makes one response per prompt a low variance estimate of the contrast, as the ablations confirm. ECE regularization alone raises SPPO from 46.73 to 47.75 Avg in Table 3, and removing temperature refitting lowers CBPO from 49.01 to 46.97 in Table 9.

Dr. GRPO [2] removes the standard deviation but keeps eight rollouts, so its group mean is still an accurate baseline. The weight still changes the objective, since p(1−p)\sqrt{p(1-p)} makes the expected update the gradient of ∑q2arcsin⁡p(q)\sum_q 2\arcsin\sqrt{p(q)}, while the raw residual ascends ∑qp(q)\sum_q p(q) and gives hard prompts a smaller share. With one sample per prompt the difference is measurable, and Bernoulli normalization alone raises Avg from 46.73 to 47.59 in Table 3.

This analysis also predicts the behavior of GRPO-2. With G = 2, every nondegenerate group has the same standard deviation, so GRPO-2 keeps the contrast but weights each prompt by p(1−p)/2p(1-p)/\sqrt2, losing the difficulty adaptive weight, and only mixed pairs, with probability 2p(1−p)≤1/22p(1-p)\le 1/2, carry signal. We ran the suggested comparison at 256 responses per step (seed 1).

MethodPrompts × rolloutsCriticNon-zero-advantage responsesGPU-hoursAvgPassHard AvgHard PassAvg @100 GPU-h
GRPO-8256 × 8✗57.9%480.2346.273.033.266.740.71
GRPO-2128 × 2✗23.8%138.945.171.632.064.442.48
CBPO256 × 1✓100%174.7549.076.536.670.845.64

The results agree with both the reviewer's evidence and our analysis. GRPO-2 retains 97.6% of GRPO-8's Avg and is 1.77 Avg above it at 100 GPU hours, yet CBPO outperforms it on every metric with the same responses per step, by 3.9 Avg after full training and 3.16 Avg at 100 GPU hours, including the critic's cost. Since GRPO-2 has the same contrast, these gains come from the calibrated critic and the Bernoulli normalization, and they are largest on the Hard metrics, where the normalization matters most.


Question1:

We will consolidate the structure in the revision without removing any content.

  • Section 3 will have three subsections, covering the statistical target and the calibrated critic, including the bound in Eq. (8); dual calibration, merging online ECE regularization and temperature scaling; and the CBPO objective, merging the advantage and the clipped surrogate.
  • Appendix G will be reduced from eleven subsections to four, covering the GRPO limit and affine uniqueness; the miscalibration identities and calibration bounds, including the unit variance case; the Fisher interpretation and the calibration mechanisms; and extensions beyond binary rewards.
  • The analysis in W2, which links the CBPO update to the correct minus incorrect contrast, will be added to the first appendix subsection.

References:

[1] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild. arXiv:2503.18892.

[2] Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO). arXiv:2503.20783.

[3] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.

[4] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476.

[5] Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning. JMLR 5:1471–1530, 2004.

[6] Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning 8:229–256, 1992.

Thank you for your clarification and repsonse.

Reviewer_rpP1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Calibrated Bernoulli Policy Optimization (CBPO), a single-sample RLVR method designed to reduce the computational overhead of group-based policy optimization. The authors theoretically show that GRPO's normalized advantage converges to a standardized Bernoulli residual as the group size approaches infinity. Building on this observation, CBPO uses a prompt-level critic to estimate success probabilities and construct variance-normalized advantages from a single rollout. To address critic miscalibration, the method introduces online ECE-style regularization and iterative temperature scaling. Experiments on mathematical reasoning benchmarks demonstrate competitive or superior performance compared to multi-sample baselines, while substantially reducing computational costs.

Critical strengths
  • The connection between GRPO's group normalization and Bernoulli variance normalization provides a principled motivation for single-sample RLVR. The analysis of how critic miscalibration affects advantage estimation further supports the proposed calibration mechanisms.
  • CBPO achieves competitive or superior performance across multiple mathematical reasoning benchmarks while reducing GPU-hours by approximately 61–70% compared to multi-sample methods. The ablation studies and equal-budget comparisons further support the effectiveness and practical value of the proposed approach.
Critical weaknesses
  • Although the paper acknowledges that score-level calibration does not guarantee accurate prompt-level success probabilities, the proposed calibration mechanisms primarily optimize aggregate calibration metrics. It remains unclear whether these improvements translate into sufficiently accurate prompt-level estimates, which are central to the theoretical motivation of CBPO. Moreover, the reported extreme advantage values suggest that calibration does not fully eliminate potential instability.
  • The experiments primarily focus on Qwen2.5-7B and Qwen3-8B models on mathematical reasoning tasks with binary rewards. Although different critic configurations are explored, the influence of backbone capabilities on calibration quality and final performance is not systematically isolated. The effectiveness of CBPO across substantially different model scales, pretrained capabilities, and other reasoning tasks therefore remains unclear.
Questions for authors
  • Since score-level calibration does not necessarily guarantee accurate prompt-level success probability estimation, have the authors investigated how closely the calibrated critic estimates the true success probability of individual prompts? How does this estimation error affect advantage normalization and training stability, particularly near probability boundaries?
  • Could the gap between calibration quality and prompt-level probability accuracy depend on the underlying backbone's capabilities? Have the authors investigated how actor or critic backbone strength affects calibration quality, prompt-level estimation accuracy, and the resulting performance of CBPO? In particular, can the different critic configurations in Appendix F help clarify this relationship?
  • How well does CBPO generalize to models with substantially different scales or pretrained capabilities, and to other tasks such as coding or tool use? While partial-credit rewards would require extending the current Bernoulli formulation, discussing this limitation and the applicability of the proposed calibration strategy would also help clarify the method's broader scope.
5 comments

Dear Reviewer rpP1,

We are grateful for your constructive and detailed feedback on the calibration analysis, model scale, and task coverage, which showed us where the method needed more direct evidence.


Weakness1:

Score level calibration is the property that CBPO needs. The advantage depends on the critic only through its score ss, so by Eq. (26) in Appendix G.5 its mean and variance at each score are (ps−s)/D(s)(p_s-s)/\sqrt{D(s)} and ps(1−ps)/D(s)p_s(1-p_s)/D(s). A calibrated critic, ps=sp_s=s, thus gives zero mean and unit variance at every score, even when prompts sharing a score differ in true success probability, so the normalization works without an exact estimate for every prompt.

Prompt level accuracy is linked to aggregate calibration by the standard decomposition [1, 2],

E[(s−p(q))2]=E[(s−ps)2]⏟calibration+E[{Var}(p(q)∣s)]⏟refinement.\mathbb E\big[(s-p(q))^2\big]=\underbrace{\mathbb E\big[(s-p_s)^2\big]}_{\text{calibration}}+\underbrace{\mathbb E\big[\mathrm\{Var\}(p(q)\mid s)\big]}_{\text{refinement}}.

ECE regularization and temperature scaling remove the calibration term, and BCE training improves the refinement term, which measures how well the critic separates easy and hard prompts. Better calibration therefore lowers the prompt level error directly.

To measure this directly, we estimated p(q)p(q) for 512 held out DeepScaleR prompts with 32 rollouts each from the final Qwen3-8B actor at step 500, the checkpoint in Table 1, and compared it with the critic at the same step.

CriticHeld out ECE ↓Prompt MAE ∣s−p(q)∣\lvert s-p(q)\rvert ↓Brier ↓Spearman ρ(s,p)\rho(s,p) ↑s∉[0.02,0.98]s\notin[0.02,0.98] ↓DvarD_{\mathrm{var}} ↓
BCE (SPPO)0.1620.2210.1710.719.6%0.52
BCE + ECE0.1180.1930.1630.726.3%0.43
BCE + ECE + Temp. (CBPO)0.0470.1620.1520.722.1%0.17

ECE regularization lowers the prompt level MAE from 0.221 to 0.193 and temperature scaling to 0.162, a 27% reduction, and boundary predictions fall from 9.6% to 2.1%. Spearman correlation stays at 0.71–0.72 because temperature scaling preserves ranking, so the gain comes entirely from correcting the probability scale that enters the advantage denominator.

The extreme advantages have a different cause, the numerical precision of the critic output. All five steps in Appendix F below −100 have the identical value −9,986.44, which follows from A^=(r−s)/s(1−s)+ϵ\hat A=(r-s)/\sqrt{s(1-s)+\epsilon}. For an incorrect response, r=0r=0, and

\begin{cases} -\sqrt{\frac{s}{1-s}}, & s < 1, \\\\ -\frac{1}{\sqrt{\epsilon}}, & s = 1. \end{cases}$$ For $s<1$ the advantage is $-\sqrt{s/(1-s)}$, and for $s=1$ only $\epsilon$ remains in the denominator. In bf16, probabilities just below 1 lie on the grid $s=1-k\cdot2^{-8}$, and $k=1,2,3$ give −15.97, −11.27, and −9.18, almost identical to the other observed minima −16.0, −11.29, and −9.16. The next grid point, $s=1$, gives $-1/\sqrt{\epsilon}\approx-10^{4}$, almost identical to −9,986.44, so each event is an incorrect response whose probability was rounded to exactly 1.0. Because the cause is rounding, the events are rare and leave no lasting effect. They occur at 5 of 1,500 steps, gradient clipping keeps each within the size of an ordinary update, and training returns to normal at the next step. No new run in this rebuttal has such an event, and clamping $s$ to $[10^{-3},1-10^{-3}]$, a safeguard we add to the released code, bounds $|\hat A|$ by 31.6. --- **Weakness2:** We added a smaller Qwen3-4B and a larger Qwen3-14B, both without thinking, with the data, schedule, and CBPO hyperparameters of the paper, a critic of the actor's size, and eight rollouts for GRPO. For these runs we report Avg@12 and Pass@12 on every benchmark. | Backbone | Method | Avg | Pass | Hard Avg | Hard Pass | Eval ECE ↓ | Prompt MAE ↓ | GPU-hours | | --- | --- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | | Qwen3-4B | Base | 39.6 | 53.4 | 24.0 | 42.5 | – | – | – | | | GRPO | 47.2 | 62.0 | 33.4 | 53.5 | – | – | 290.4 | | | **CBPO** | **48.5** | **64.7** | **34.5** | **56.5** | **0.14** | **0.172** | 106.8 | | Qwen3-14B | Base | 43.6 | 72.1 | 30.0 | 61.1 | – | – | – | | | GRPO | 65.9 | 81.4 | 55.8 | 74.4 | – | – | 812.7 | | | **CBPO** | **69.7** | **83.5** | **60.2** | **77.2** | **0.10** | **0.149** | 299.8 | CBPO is best on all four metrics for both backbones with 37% of GRPO's GPU hours, gaining 1.3 Avg on Qwen3-4B and 3.8 Avg on Qwen3-14B. With Table 1, it now outperforms GRPO on four backbones from 4B to 14B in two model families, and per benchmark results are listed at the end of this response. Calibration also improves with backbone strength, with Eval ECE of 0.14, 0.12, and 0.10 from 4B to 8B to 14B, because a stronger actor gives more consistent outcomes and a cleaner target for the critic.

Question1:

We carried out this study, and the measurement is reported in W1. Building on it, the table below breaks the results down by difficulty, grouping the same 512 prompts by p(q)p(q). The overconfident share is the fraction of prompts whose ss lies closer to the nearest boundary than p(q)p(q).

p(q)p(q) binShareMean ssMAE ∣s−p∣\lvert s-p\rvertOverconfident shareVar^(A^)\widehat{\mathrm{Var}}(\hat A)Max ∣A^∣\lvert\hat A\rvert
[0,0.1)[0,0.1)27%0.1280.10718%0.426.8
[0.1,0.4)[0.1,0.4)21%0.2680.21431%1.124.9
[0.4,0.6)[0.4,0.6)12%0.4930.24544%1.092.6
[0.6,0.9)[0.6,0.9)19%0.7210.20733%1.144.4
[0.9,1][0.9,1]21%0.8820.09317%0.457.3

Near the boundaries the critic errs on the conservative side, which keeps the advantage bounded. By Eq. (25), E[A^2∣q]=[p(1−p)+(p−s)2]/D(s)\mathbb E[\hat A^2\mid q]=\big[p(1-p)+(p-s)^2\big]/D(s) is large only when ss is closer to the boundary than pp, whereas a conservative ss between pp and 1/21/2 gives D(s)≥p(1−p)D(s)\ge p(1-p) and an advantage of ideal scale. Temperature scaling moves predictions in this direction, with fitted temperatures above 1 at 75–85% of steps in Appendix F, so only 17–18% of prompts near the boundary end up overconfident.

This conservativeness lowers the weight of the outer bins, whose advantage variance is 0.42–0.45. For p=0.03p=0.03 and s=0.13s=0.13, the weight p(1−p)/D(s)p(1-p)/\sqrt{D(s)} is 0.087, half of the ideal 0.171, but still three times the raw residual weight of 0.029 and above the 0.069 of GRPO with eight rollouts, so hard prompts keep a larger share than without normalization. The largest advantage is 7.3, far below the saturation value in W1, and we will add this analysis with a second moment bound to Appendix G.


Question2:

Since the decomposition in W1 separates the error into a calibration term and a refinement term, we examined how each term changes with backbone strength. Calibration removes the calibration term on every backbone, so the dependence on the backbone enters through the refinement term, which reflects critic capacity and how predictable the actor's outcomes are.

Appendix F points the same way but changes several factors at once. With the same Qwen2.5-7B actor, the 1.5B critic run has three extreme advantage steps and the 7B critic run has one, so we ran a controlled comparison that varies only the critic size.

ActorCriticHeld out ECE ↓Prompt MAE ↓Spearman ρ(s,p)\rho(s,p) ↑DvarD_{\mathrm{var}} ↓AvgHard Avg
Qwen2.5-7BQwen2.5-1.5B0.0690.2080.610.2329.115.2
Qwen2.5-7BQwen2.5-7B0.0520.1720.680.1729.816.3
Qwen3-8BQwen3-1.7B0.0630.1970.640.2148.135.5
Qwen3-8BQwen3-8B0.0470.1620.720.1749.036.6

As the decomposition predicts, a larger critic mainly improves ranking, the part temperature scaling cannot change, raising Spearman correlation from 0.61 to 0.68 and from 0.64 to 0.72 and Avg by 0.7–0.9. Calibration error stays at 0.05–0.07 for both sizes, so even a 1.5B or 1.7B critic keeps CBPO above GRPO with eight rollouts.

A stronger actor helps in the same way, lowering Eval ECE from 0.14 to 0.10 across the W2 backbones while CBPO leads GRPO by 1.3–2.8 Avg. We will add the critic size comparison to Appendix F.

Question3:

W2 and Q2 cover other model scales. For other tasks, we trained Qwen2.5-7B-Instruct on ALFWorld [3] and WebShop [4] with the GiGPO protocol [5], with binary task success as the reward. All methods use the same tasks per step and the same iterations, but GRPO and GiGPO sample eight trajectories per task and CBPO one. CBPO reuses its math hyperparameters, and results are averaged over three seeds.

MethodALFWorld AllWebShop ScoreWebShop Success
Prompting [5]14.826.47.8
GRPO [5]77.679.366.1
GiGPO [5]90.884.472.8
CBPO93.287.576.9

With one eighth of the trajectories, CBPO improves over GRPO by 15.6 points on ALFWorld and 10.8 on WebShop success, and exceeds GiGPO [5], which adds step level credit assignment to eight trajectories per task, by 2.4 and 4.1. Since each trajectory needs dozens of environment calls, removing seven of every eight rollouts is especially valuable here. Stability statistics are in our response to Reviewer CKuX, Q5.

Coding tasks rewarded by unit tests are binary and fit CBPO directly. For partial credit, Appendix G.10 gives the moment normalized form A=(R−μ(q))/v(q)+ϵA=(R-\mu(q))/\sqrt{v(q)+\epsilon}, and a categorical critic can be calibrated with the same binwise matching and temperature scaling. We will describe this in the Limitations section.


References:

[1] A New Vector Partition of the Probability Score. Journal of Applied Meteorology 12(4):595–600, 1973.

[2] Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association 102(477):359–378, 2007.

[3] ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. arXiv:2010.03768.

[4] WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. arXiv:2207.01206.

[5] Group-in-Group Policy Optimization for LLM Agent Training. arXiv:2505.10978.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.