- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies the reward normalization in GRPO. By interpreting GRPO normalization as prompt reweighting: it strengthens low-variance prompt contributions while controlling aggregate gradient noise and exposes a tradeoff under more aggressive normalization.
Critical strengths
- The theoretical analysis is sufficient.
Critical weaknesses
-
The experiments are too weak. The paper only conducts experiments on models of 0.5B and tasks only in math, which is much weaker than other papers.
-
Missing comparison with other strong baseline methods, such as GSPO,GMPO,GPG,CISPO,Dr.GRPO.
-
The paper uses roughly 5× more rollouts, while sequential doubling and variable group sizes may hinder batching and parallelism and complicate scheduling. Matched-budget and wall-clock comparisons are needed to justify these costs.
Questions for authors
Under the same total sampling budget, does adaptive allocation outperform a larger fixed group size?
2 comments
Adaptive sampling during training: comparisons with larger fixed groups at matched rollout budgets
Thank you for suggesting a comparison with larger fixed groups under the same total rollout budget. We have now completed three end-to-end training comparisons. Adaptive sampling is applied throughout training, and the reported test rewards evaluate the resulting trained policies. Adaptive sampling achieves essentially the same test reward at 2.5 epochs and higher test reward at 4.5 and 6 epochs, while the fixed-group baselines receive slightly more generated training rollouts.
All comparisons use Qwen2.5-0.5B-Instruct on GSM8K, normalization exponent (standard GRPO), sequence-mean loss aggregation, the same data processing order, AdamW optimizer, cosine learning-rate schedule, and evaluation procedure. We evaluate expected reward on the test set using four responses per prompt. All runs use training seed 20260811.
We compare three fixed-group baselines with the same adaptive run. Adaptive sampling starts with and doubles the group size, reusing all previously generated responses, until the group contains both reward outcomes or reaches . At each training horizon, we choose the fixed group size by rounding up the adaptive run’s realized average group size:
| Training horizon | Adaptive prompt visits | Adaptive training rollouts | Adaptive mean group size | Fixed group size |
|---|---|---|---|---|
| 2.5 epochs | 18,722 | 259,976 | 13.8861 | 14 |
| 4.5 epochs | 33,668 | 515,460 | 15.3101 | 16 |
| 6 epochs | 44,838 | 724,896 | 16.1670 | 17 |
The test rewards at these training horizons are:
| Training horizon | Adaptive | Fixed group | Difference (percentage points) |
|---|---|---|---|
| 2.5 epochs | 0.443518 | 0.443897 | −0.04 |
| 4.5 epochs | 0.472707 | 0.468726 | +0.40 |
| 6 epochs | 0.463419 | 0.461524 | +0.19 |
We also evaluate every quarter epoch. The highest observed test reward within each comparison window is slightly higher for adaptive sampling:
| Comparison window | Adaptive best | Fixed-group best | Epoch attaining both maxima |
|---|---|---|---|
| Through 2.5 epochs, fixed | 0.451857 | 0.451478 | 2.25 |
| Through 4.5 epochs, fixed | 0.472707 | 0.468726 | 4.5 |
| Through 6 epochs, fixed | 0.472707 | 0.470243 | 4.5 |
These results support the practical relevance of the sampling rule motivated by our analysis: adaptive allocation remains competitive with larger uniform groups under controlled rollout budgets, with modest improvements at the longer training horizons.
Response to reviewer Khoj
Thank you for again for your questions about sampling cost and the baseline comparisons. We have revised the paper to address these points more directly. The additions are marked in blue.
We have clarified the connection to Dr.GRPO: removing reward standard-deviation normalization is the α=0 setting in our experiments. The comparisons hold response-length averaging fixed across normalization settings. The experiments include both 0.5B and 1.5B models; the latter is evaluated on MATH-500, Minerva Math, and OlympiadBench in Table 2 of the revision (Table 1 in the submission).
We have also added a controlled reward-profile comparison at the initial 0.5B policy, using 128 responses on each of 320 GSM8K training prompts (new Table 3, Appendix E). Exact averages over reward orderings show that doubling uses 13.41 rollouts per prompt: 54.2% of groups stop at four responses and 8.4% reach 64. It produces mixed-reward groups for 96.1% of prompts, versus 81.6% with uniform groups of 14 and 67.3% when the doubling size distribution is assigned independently of rewards at the same expected budget. A uniform-size control matched to that expected budget reaches 80.9%. Uniform groups of 64 reach the same 96.1% coverage as doubling while using about 4.8 times as many rollouts. The table also quantifies the precision tradeoff: at α=1, the conditional-variance factor with unit within-class score variances is 2.88 times the uniform-14 value.
For sampling cost, Proposition 8 and Table 1 now give the expected update and rollout count of the experimental rule, which starts with four responses and doubles up to 64 until both reward outcomes appear. For standard GRPO and binary success probability p∈[0.01,0.99], its response-level expected-update multiplier lies in [0.9139,1.2259], compared with approximately 0.1719 to 0.8080 for a fixed group of four. The adaptive multiplier is closer to one at every probability in this range. At p=0.01, for example, it retains about 97% of the population-normalized update using 52.05 responses on average, compared with 17% for four responses. The table shows how the additional sampling recovers prompt contributions that would otherwise be strongly suppressed.
The revised binary theorem (Theorem 4; Theorem 3 in the submission) shows that, for each fixed α>0, group size of order 1/σ2 is necessary and sufficient to preserve a prescribed fraction of the population-normalized update. Proposition 3 establishes the same inverse-variance scaling for accuracy and sampling cost under a confidence-based doubling rule.
We also added an exact second-moment identity in Proposition 8. For a fixed set of prompts and a total rollout budget, Proposition 9 uses it to identify the allocation minimizing leading large-budget gradient variance and mean-squared error. The allocation accounts for both reward variance and how policy scores vary within and between the two reward classes.
Finally, new Theorem 3 establishes matching lower bounds for estimation error, bias, and rollout variance. The sharp worst-case mean-squared-error and variance rate is K−min1,2−α. Standard GRPO is thus the strongest normalization power in this family that retains the 1/K rate. This identifies the statistical cost of giving low-variance prompts greater relative weight.