- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CBPO reduces rollout cost by replacing empirical group statistics with a learned success-probability critic. For binary rewards, the critic provides both the reward baseline and the Bernoulli variance used to normalize advantages. The method combines this normalization with an ECE-style training penalty and repeated temperature scaling. The analysis connects the resulting residual to the large-group limit of GRPO and characterizes how probability errors affect its mean and variance. Experiments on mathematical reasoning benchmarks report competitive accuracy at lower training cost, with component ablations, equal-budget comparisons, and three-seed results for one model setting.
Critical strengths
- The critic controls both advantage centering and scale. An inaccurate prediction near a probability boundary can therefore affect both quantities. The paper derives this dependence explicitly and uses it to motivate calibration, giving the method a clear statistical rationale.
- The equal-GPU-hour comparison directly addresses the intended efficiency benefit. The component ablations further examine the contributions of normalization, ECE regularization, and temperature scaling. Together, these experiments provide useful evidence for the proposed combination.
Critical weaknesses
- On Qwen3-8B without thinking, Table 1 reports Avg scores of 49.0 for CBPO and 46.2 for GRPO, a difference of 2.8 points. In Table 2, their three-seed means are 48.50 and 47.29, reducing the difference to 1.21 points. The repeated evaluation covers only this model setting and excludes DAPO, a strong baseline in the main comparison. The seed and checkpoint selection rules are therefore important to interpreting the reported gains. The efficiency comparison also depends on the baseline configuration. The equal-budget experiment uses eight rollouts per prompt for GRPO, while CBPO uses one. It remains unclear whether CBPO retains the same advantage against smaller GRPO group sizes under the same compute budget. These comparisons are needed to determine how much of the observed benefit persists across training runs and reasonable baseline configurations.
- The theoretical bounds require controlled probability error and predictions away from the boundaries. The paper acknowledges that its calibration procedures do not guarantee these conditions. Appendix F reports minimum advantages of approximately −9,986 despite calibration. It also provides gradient-norm traces, but does not establish how these events affect subsequent held-out performance or how their frequency compares with GRPO and SPPO across independent runs. This matters because stable single-rollout training is a central motivation for CBPO. Lower aggregate calibration error and competitive final scores leave open whether the method consistently controls rare, large updates. The stability claim needs direct evidence connecting these events to training outcomes.
Questions for authors
- How were the runs and checkpoints in Table 1 selected? How do they relate to the runs in Table 2, and were any runs excluded from the reported comparisons?
- Can you provide individual-seed results and uncertainty estimates for the performance differences between CBPO and the strongest relevant baselines, including DAPO, under a common selection and compute-budget protocol?
- Were smaller GRPO group sizes evaluated at the same GPU-hour budget? How does CBPO compare with the strongest budget-matched configuration?
- What happened to gradient norms and subsequent held-out performance around the extreme-advantage events in Appendix F? How frequently did comparable events or training failures occur for CBPO, GRPO, and SPPO across independent runs?
- Could you compare CBPO with GRPO on a downstream agentic RL task involving very long trajectories and multiple rounds of environment interaction? Under matched training budgets and comparable tuning effort, please report task success, variation across independent training seeds, and the frequency of extreme advantages, gradient spikes, and training failures. This would help assess whether the observed efficiency gains remain useful when training requires sustained interaction with a changing environment.
4 comments
Question1 and 2:
Selection rule. Each Table 1 entry is a single seed 1 run, reported at the checkpoint with the highest six benchmark Avg among evaluations every 5 steps, with the same rule for all methods. Table 2 repeats this with seeds 1–3, so the Table 1 runs are its seed 1 runs, and no run or seed was excluded.
Individual seeds and uncertainty. We added three DAPO seeds under the same protocol, and Table R1 reports every seed with the mean and standard deviation.
Table R1. Individual seeds at the best checkpoint, Qwen3-8B without thinking.
| Method | Seed 1 Avg | Seed 2 Avg | Seed 3 Avg | Avg, mean ± SD | Pass | Hard Avg | Hard Pass |
|---|---|---|---|---|---|---|---|
| PPO | 45.60 | 46.18 | 44.87 | 45.55 ± 0.66 | 70.79 ± 1.16 | 32.25 ± 0.83 | 63.59 ± 1.26 |
| GRPO | 46.20 | 48.06 | 47.61 | 47.29 ± 0.97 | 74.06 ± 0.96 | 34.51 ± 1.19 | 68.05 ± 1.19 |
| SPPO | 46.70 | 47.84 | 47.27 | 47.27 ± 0.57 | 73.30 ± 0.85 | 34.48 ± 0.64 | 67.21 ± 0.86 |
| DAPO | 47.70 | 48.28 | 46.58 | 47.52 ± 0.86 | 74.61 ± 1.04 | 34.72 ± 1.02 | 68.66 ± 1.21 |
| CBPO | 49.01 | 48.57 | 47.92 | 48.50 ± 0.55 | 75.52 ± 0.88 | 35.83 ± 0.74 | 69.45 ± 1.16 |
CBPO has the highest mean on every metric, leading DAPO by 0.98 Avg, and the smallest seed variance. Paired by seed, it wins all 12 comparisons against PPO, GRPO, SPPO, and DAPO, so the gain does not depend on a favorable seed. It also ranks first at the final checkpoint, with 47.96 Avg against 46.88 for DAPO, so the result is independent of the selection rule. We will state the rule in Section 4.1 and report both protocols.
Question3:
We evaluated GRPO with smaller groups under the same budget, and the full table is in our response to W1. At 100 GPU hours, G = 2 is the strongest GRPO configuration at 42.48 Avg, and CBPO reaches 45.64, ahead of it by 3.16 and of SPPO, the strongest budget matched baseline, by 1.52. After 500 steps (seed 1), CBPO leads every GRPO configuration by 2.8–3.9 Avg.
Question4:
We traced all five archived events in the training logs, and none of them had a lasting effect. At each event the gradient norm before clipping jumps to between 85 and 1,001, but clipping caps the update at norm 1.0, and at the very next step the norm is back to 0.8–1.3 and the minimum advantage to between −1.1 and −2.4. Held out MATH500 accuracy is barely affected, rising after three events, for example from 0.734 to 0.760, and changing by at most 0.014 after the other two, as from 0.872 to 0.864, which is within the normal variation between evaluations.
We also counted the steps whose most negative advantage or gradient norm exceeds ten times its run median. Every method was trained in three runs of 500 steps each, 1,500 training steps in total. As Table R2 shows, the baselines have one or two of each, the archived CBPO runs have five, the new CBPO runs have one, and no run diverged.
Table R2. Extreme advantage steps and gradient spikes over all training steps.
| Method | Extreme advantage steps | Gradient spikes |
|---|---|---|
| GRPO | 1 | 0 |
| DAPO | 1 | 2 |
| GSPO | 2 | 1 |
| RLOO | 1 | 0 |
| PPO | 2 | 2 |
| REINFORCE++ | 2 | 1 |
| CBPO | 5 | 5 |
| CBPO, new runs in this rebuttal | 1 | 1 |
Question5:
We trained Qwen2.5-7B-Instruct on ALFWorld and WebShop with the GiGPO protocol [1], with binary task success and trajectories of several thousand tokens over up to 50 environment steps. All methods use the same tasks per step and the same iterations, but GRPO and GiGPO sample eight trajectories per task and CBPO one. The baselines use GiGPO's tuned configurations while CBPO reuses its math hyperparameters, so they receive at least comparable tuning. Results are over three seeds.
| Method | ALFWorld All | WebShop Score | WebShop Success | Seed std, ALFWorld / WebShop Success |
|---|---|---|---|---|
| Prompting [1] | 14.8 | 26.4 | 7.8 | – |
| GRPO [1] | 77.6 | 79.3 | 66.1 | 5.2 / 3.7 |
| GiGPO [1] | 90.8 | 84.4 | 72.8 | 1.5 / 2.9 |
| CBPO | 93.2 | 87.5 | 76.9 | 1.2 / 2.1 |
With one eighth of the trajectories, CBPO improves over GRPO by 15.6 points on ALFWorld and 10.8 on WebShop success, and exceeds GiGPO, which adds step level credit assignment, by 2.4 and 4.1.
CBPO also has the smallest seed variance, 1.2 and 2.1 points against 5.2 and 3.7 for GRPO. Across all runs, no method has an advantage beyond 100 in magnitude or a failed run, and each method has a single gradient spike, so with one trajectory per task CBPO trains as stably as the group baselines.
The efficiency gain thus carries over to long interaction with a changing environment, where each trajectory needs dozens of environment calls and CBPO removes seven of every eight rollouts.
References:
[1] Group-in-Group Policy Optimization for LLM Agent Training. arXiv:2505.10978.
The revised manuscript is now available on OpenReview, and we would like to briefly summarize how it reflects your review. Your observation of the rare large updates in our training logs led us to trace their source and give a clearer account of the training stability of CBPO, and the changes related to your comments are as follows.
- Seeds and checkpoint selection (W1, Q1, Q2). Table 1 is now marked as seed 1, and Appendix E states the selection rule, which applies the same criterion to every method and excludes no run or seed. The comparison across three seeds in Section 4.2 now includes DAPO, and Appendix E reports Pass metrics and the final checkpoint at step 500 (Table 12), where CBPO keeps the highest mean on every metric.
- Smaller GRPO groups (W1, Q3). Section 4.6 adds GRPO with groups of 2 and 4 alongside 8 and 16 (Table 5), at 100 GPU hours and after full training. Smaller groups help GRPO at a fixed budget, yet CBPO leads GRPO-2 by 3.16 Avg with the same 256 responses per step.
- Rare large updates (W2, Q4). Appendix I traces every extreme advantage to bf16 rounding of a critic probability to exactly one (Eq. 18), reports the gradient norms and held out accuracy around each event, and counts comparable events for all methods (Table 18). The events occur at 5 of 1,500 steps, gradient clipping bounds each update, and clamping the critic probability, now included in the released implementation, bounds every advantage by 31.6.
- Agentic tasks (Q5). Section 4.6 and Appendix G add ALFWorld and WebShop (Table 8), where CBPO outperforms GRPO and GiGPO with one trajectory per task and has the smallest variation across seeds.
We hope these revisions address your concerns, and we would be grateful if you could take them into account in your final assessment. We are happy to clarify any remaining questions.
Thank you for your clarification and repsonse. But I still have some concerns.
Dear Reviewer CKuX,
Your close reading of our experimental protocol, baseline configurations, and training stability exposed gaps in the evidence that we could not see ourselves, and we are grateful for the time and care it took.
Weakness1:
Both tables use the same Avg metric and selection rule, and each Table 1 run is the seed 1 run of Table 2. The two margins differ only because of seed variance, since GRPO's seed 1 run, at 46.2, lies below its three seed mean of 47.29.
We ran GRPO with G = 4, using 256 prompts × 4 rollouts, and G = 2, using 128 prompts × 2 rollouts to match CBPO's 256 responses per step, on Qwen3-8B without thinking. The table reports each configuration at 100 GPU hours, as in Table 6, and after 500 steps (seed 1), where Steps @100 GPU-h is the number of steps completed within 100 GPU hours.
At 100 GPU hours. As the reviewer anticipated, smaller groups help GRPO because cheaper steps allow more updates, and GRPO-2 becomes its strongest configuration at 1.77 Avg above GRPO-8. CBPO still leads GRPO-2 by 3.16 Avg, because the calibrated critic gives every single response a reliable baseline, letting CBPO spend its budget on more distinct prompts, and the Bernoulli normalization keeps hard prompts at full weight.
After full training. Smaller groups lose accuracy, with Avg falling from 46.2 at G = 8 to 45.1 at G = 2, because fewer responses give a noisier contrast and more groups with identical rewards. CBPO exceeds GRPO-8 by 2.8 Avg with 36% of its GPU hours and GRPO-2 by 3.9 Avg with the same responses per step, so the gain comes from replacing noisy group statistics with calibrated probabilities, the role of online ECE regularization and temperature scaling. SPPO, which also uses one rollout but lacks these components, trails CBPO by 2.3 Avg at almost the same cost, which confirms this.
Weakness2:
The reviewer is right that the bounds require predictions away from the boundaries, and the Appendix F events are exactly where this fails, for a numerical reason. All five steps below −100 have the identical value −9,986.44, which follows from A^=(r−s)/s(1−s)+ϵ. For an incorrect response, r=0, and
\begin{cases} -\sqrt{\frac{s}{1-s}}, & s < 1, \\\\ -\frac{1}{\sqrt{\epsilon}}, & s = 1. \end{cases}$$ For $s<1$ the advantage is $-\sqrt{s/(1-s)}$, and for $s=1$ only $\epsilon$ remains in the denominator. In bf16, probabilities just below 1 lie on the grid $s=1-k\cdot2^{-8}$, and $k=1,2,3$ give −15.97, −11.27, and −9.18, almost identical to the other observed minima −16.0, −11.29, and −9.16. The next grid point, $s=1$, gives $-1/\sqrt{\epsilon}\approx-10^{4}$, almost identical to −9,986.44, so each event is an incorrect response whose probability was rounded to exactly 1.0. Because the cause is rounding rather than a failure of calibration, the events are rare and leave no lasting effect. They occur at 5 of 1,500 steps, gradient clipping keeps each within an ordinary update, and both the gradient norm and MATH500 accuracy are normal at the next step and evaluation. No new run in this rebuttal has an advantage below −100, and clamping $s$ to $[10^{-3},1-10^{-3}]$, a safeguard we add to the released code, bounds $|\hat A|$ by 31.6. Q4 gives the detailed statistics.