acceptodds
Under review as a conference paper at ICLR 2027

Understanding Reward Normalization in GRPO

Abstract

Group Relative Policy Optimization (GRPO) has become a widely used and effective method for post-training large language models, yet the role of its within-group reward normalization remains underexplored. We identify reward discreteness as a key structure and develop a finite-sample theory for a family of normalized policy-gradient estimators. For binary rewards, we further show that the expected finite-group estimator is exactly collinear with its population counterpart. Our analysis reveals a group-size law that controls finite-group accuracy and motivates an adaptive sampler that assigns more rollouts to unresolved low-variance prompts. We interpret GRPO normalization as prompt reweighting: it strengthens low-variance prompt contributions while controlling aggregate gradient noise and exposes a tradeoff under more aggressive normalization. Finally, we analyze Balanced Aggregation, an existing token-aggregation rule, and prove that for binary rewards it preserves the population policy-gradient direction and obeys an analogous group-size law. Experiments on mathematical reasoning tasks compare normalization schemes and evaluate adaptive sampling.

Primary area:
reinforcement learning
Keywords:
GRPO, RLVR, Policy Optimization, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_KhojOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the reward normalization in GRPO. By interpreting GRPO normalization as prompt reweighting: it strengthens low-variance prompt contributions while controlling aggregate gradient noise and exposes a tradeoff under more aggressive normalization.

Critical strengths
  1. The theoretical analysis is sufficient.
Critical weaknesses
  1. The experiments are too weak. The paper only conducts experiments on models of 0.5B and tasks only in math, which is much weaker than other papers.

  2. Missing comparison with other strong baseline methods, such as GSPO,GMPO,GPG,CISPO,Dr.GRPO.

  3. The paper uses roughly 5× more rollouts, while sequential doubling and variable group sizes may hinder batching and parallelism and complicate scheduling. Matched-budget and wall-clock comparisons are needed to justify these costs.

Questions for authors

Under the same total sampling budget, does adaptive allocation outperform a larger fixed group size?

2 comments
AuthorsOpenReview ↗

Response to reviewer Khoj

Thank you for again for your questions about sampling cost and the baseline comparisons. We have revised the paper to address these points more directly. The additions are marked in blue.

We have clarified the connection to Dr.GRPO: removing reward standard-deviation normalization is the α=0\alpha=0 setting in our experiments. The comparisons hold response-length averaging fixed across normalization settings. The experiments include both 0.5B and 1.5B models; the latter is evaluated on MATH-500, Minerva Math, and OlympiadBench in Table 2 of the revision (Table 1 in the submission).

We have also added a controlled reward-profile comparison at the initial 0.5B policy, using 128 responses on each of 320 GSM8K training prompts (new Table 3, Appendix E). Exact averages over reward orderings show that doubling uses 13.41 rollouts per prompt: 54.2%54.2\% of groups stop at four responses and 8.4%8.4\% reach 64. It produces mixed-reward groups for 96.1%96.1\% of prompts, versus 81.6%81.6\% with uniform groups of 14 and 67.3%67.3\% when the doubling size distribution is assigned independently of rewards at the same expected budget. A uniform-size control matched to that expected budget reaches 80.9%80.9\%. Uniform groups of 64 reach the same 96.1%96.1\% coverage as doubling while using about 4.8 times as many rollouts. The table also quantifies the precision tradeoff: at α=1\alpha=1, the conditional-variance factor with unit within-class score variances is 2.88 times the uniform-14 value.

For sampling cost, Proposition 8 and Table 1 now give the expected update and rollout count of the experimental rule, which starts with four responses and doubles up to 64 until both reward outcomes appear. For standard GRPO and binary success probability p∈[0.01,0.99]p\in[0.01,0.99], its response-level expected-update multiplier lies in [0.9139,1.2259][0.9139,1.2259], compared with approximately 0.17190.1719 to 0.80800.8080 for a fixed group of four. The adaptive multiplier is closer to one at every probability in this range. At p=0.01p=0.01, for example, it retains about 97%97\% of the population-normalized update using 52.0552.05 responses on average, compared with 17%17\% for four responses. The table shows how the additional sampling recovers prompt contributions that would otherwise be strongly suppressed.

The revised binary theorem (Theorem 4; Theorem 3 in the submission) shows that, for each fixed α>0\alpha>0, group size of order 1/σ21/\sigma^2 is necessary and sufficient to preserve a prescribed fraction of the population-normalized update. Proposition 3 establishes the same inverse-variance scaling for accuracy and sampling cost under a confidence-based doubling rule.

We also added an exact second-moment identity in Proposition 8. For a fixed set of prompts and a total rollout budget, Proposition 9 uses it to identify the allocation minimizing leading large-budget gradient variance and mean-squared error. The allocation accounts for both reward variance and how policy scores vary within and between the two reward classes.

Finally, new Theorem 3 establishes matching lower bounds for estimation error, bias, and rollout variance. The sharp worst-case mean-squared-error and variance rate is K−min⁡1,2−αK^{-\min\\{1,2-\alpha\\}}. Standard GRPO is thus the strongest normalization power in this family that retains the 1/K1/K rate. This identifies the statistical cost of giving low-variance prompts greater relative weight.

AuthorsOpenReview ↗

Adaptive sampling during training: comparisons with larger fixed groups at matched rollout budgets

Thank you for suggesting a comparison with larger fixed groups under the same total rollout budget. We have now completed three end-to-end training comparisons. Adaptive sampling is applied throughout training, and the reported test rewards evaluate the resulting trained policies. Adaptive sampling achieves essentially the same test reward at 2.5 epochs and higher test reward at 4.5 and 6 epochs, while the fixed-group baselines receive slightly more generated training rollouts.

All comparisons use Qwen2.5-0.5B-Instruct on GSM8K, normalization exponent α=1\alpha=1 (standard GRPO), sequence-mean loss aggregation, the same data processing order, AdamW optimizer, cosine learning-rate schedule, and evaluation procedure. We evaluate expected reward on the test set using four responses per prompt. All runs use training seed 20260811.

We compare three fixed-group baselines with the same adaptive run. Adaptive sampling starts with K=4K=4 and doubles the group size, reusing all previously generated responses, until the group contains both reward outcomes or reaches Kmax⁡=64K_{\max}=64. At each training horizon, we choose the fixed group size by rounding up the adaptive run’s realized average group size:

Training horizonAdaptive prompt visitsAdaptive training rolloutsAdaptive mean group sizeFixed group size
2.5 epochs18,722259,97613.886114
4.5 epochs33,668515,46015.310116
6 epochs44,838724,89616.167017

The test rewards at these training horizons are:

Training horizonAdaptiveFixed groupDifference (percentage points)
2.5 epochs0.4435180.443897−0.04
4.5 epochs0.4727070.468726+0.40
6 epochs0.4634190.461524+0.19

We also evaluate every quarter epoch. The highest observed test reward within each comparison window is slightly higher for adaptive sampling:

Comparison windowAdaptive bestFixed-group bestEpoch attaining both maxima
Through 2.5 epochs, fixed K=14K=140.4518570.4514782.25
Through 4.5 epochs, fixed K=16K=160.4727070.4687264.5
Through 6 epochs, fixed K=17K=170.4727070.4702434.5

These results support the practical relevance of the sampling rule motivated by our analysis: adaptive allocation remains competitive with larger uniform groups under controlled rollout budgets, with modest improvements at the longer training horizons.

Reviewer_Q5qvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents some estimation-error and bias bounds for finite-group reward normalization in GRPO for discrete rewards. The results identify the quantity Kσ2K\sigma^2 governing accuracy relative to the signal scale for the bias and mean-squared error of the gradient estimator and suggest an adaptive strategy of rollout number allocation. Besides, for the binary rewards the results identify that small groups can substantially suppress the intended normalized learning signal. Validation consists of numerical illustrations of the attenuation effect and mathematical-reasoning experiments with 0.5B and 1.5B models comparing normalization exponents and fixed versus adaptive sampling.

Critical strengths

None

Critical weaknesses

Below are the concerns related to some of the contributions that the authors identify as main (as stated in the introduction):

  1. Finite-group estimation and bias guarantees for σα\sigma^\alpha normalization. This is a legitimate statistical contribution but its practical implications are insufficiently established. The results identify how estimation difficulty depends on reward variance and normalization strength, but that doesn't establish whether spending more samples to reduce that error is preferable to, for example, obtaining updates from more prompts or allocating resources differently. The experiments do not establish whether reducing this estimation error improves training under a fixed sampling budget.
  2. A group-size law motivating a doubling sampler that allocates more rollouts to unresolved prompts. This contribution has a plausible motivation, but weak evidence for the proposed allocation strategy. More specifically, the experimental part reports higher performance when applying the proposed allocation strategy, however, as noted by the authors, that strategy leads to 5x more rollouts on average. That raises an obvious question: would uniformly increasing the number of rollouts achieve comparable gains? To demonstrate the effectiveness of the proposed strategy, the compute budget should have been fixed for all the methods.
  3. Interpreting normalization as prompt reweighting and analyzing aggregate gradient noise to explain a potential benefit of GRPO. The reweighting interpretation is already acknowledged as existing in prior work, therefore the interpretation itself is not a novel contribution of this paper. The paper's additional results characterize aggregate noise and establish stationarity of finite-group objectives, but neither result establishes when this reweighting benefits optimization of the average-reward objective.

A separate concern is related to the reliability of the experimental differences. The demonstrated empirical gains look mostly marginal, therefore the lack of uncertainty estimates (e.g. running several random seed for each run) calls the interpretation into question.

Questions for authors

How reproducible are the differences between normalization exponents? Can the authors report results across multiple training seeds and uncertainty estimates?

Reviewer_TkwGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the role of normalization in GRPO. It generalizes the reward normalization family to a σα\sigma^{\alpha} family for α∈[0,2]\alpha \in [0, 2] and derives finite-group estimation error and bias bounds for the resulting estimators. Both bounds in the general discrete reward setting and the binary reward setting depend on a single quantity Kσ2K\sigma^{2}, which is further used to motivate a doubling sampler that assigns more rollouts to prompts with low Kσ2K \sigma^2. The paper also shows how normalization can be beneficial to the optimization and extends the analysis to balanced aggregation. Empirically, the proposed methods are evaluated on GSM8K with a 0.5B model and on MATH-500, Minerva Math, and OlympiadBench with a 1.5B model.

Critical strengths
  • The main strength of the paper is a unified finite-group theory of reward normalization in GRPO. This paper analyzes the whole σα\sigma^{\alpha} family on both estimation error and bias bounds for general discrete rewards, and further extends the analysis to the mechanism behind normalization in the optimization process. This is a valuable contribution to the community.
Critical weaknesses
  • My primary concern is that the paper’s theoretical may be more reflective of the structure of model with small size than the general properties of GRPO. Since all experiments are conducted on 0.5B and 1.5B models, the results may not fully reflect the scale and complexity of real LLMs.
  • The theory suggests that α\alpha in (1,2)(1, 2) will be better than α=0,1\alpha =0, 1, but the experiments only try α=32\alpha = \frac{3}{2} and α=2\alpha = 2 and do not explain why we should choose α=32\alpha = \frac{3}{2}. Thus, the results in the theory part are not fully connected to the empirical results.
Questions for authors
  1. What are the results of the proposed method on larger models (e.g., 7B or above)? Do the benefits of different α\alpha and sampling still appear at larger model scales?
  2. What are the results of the proposed method on different random seeds? If the gains are stable across seeds, I will have more confidence in the theory and empirical connection.
  3. Is it possible to try different choices of α∈(1,2)\alpha \in (1, 2) and explain the results with the proposed theory?
3 comments
AuthorsOpenReview ↗

Response to reviewer TkwG

Thank you for your questions about the normalization exponent and its connection to the experiments. We have revised the paper to make this connection more explicit. The additions are marked in blue.

New Theorem 3 establishes matching lower bounds, giving the sharp worst-case mean-squared-error and rollout-variance rate K−min⁡1,2−α.K^{-\min\\{1,2-\alpha\\}}. Standard GRPO, α=1\alpha=1, is therefore the strongest power in this family that retains the worst-case 1/K1/K rate. For α=3/2\alpha=3/2, the rate is K−1/2K^{-1/2}; at α=2\alpha=2, worst-case error and variance can remain bounded away from zero. This gives a precise interpretation of the α=3/2\alpha=3/2 setting in our experiments: it gives low-variance prompts greater relative weight while remaining in the regime where estimation error vanishes as the group grows. These bounds are stated for general policies under the paper's assumptions and apply across model sizes.

The revision now includes an initial-model reward profile that connects the exponent to the sampler on 320 GSM8K training prompts, with 128 responses per prompt (new Table 3, Appendix E). At α=3/2\alpha=3/2, the relative squared error of the expected prompt weights after a common rescaling is 8.64%8.64\% for groups of four, 2.48%2.48\% for uniform groups of 14, and 0.25%0.25\% for doubling, which uses 13.41 rollouts per prompt on average. These are exact averages over reward orderings, with reference weights estimated from each prompt's full reward record. At α=1\alpha=1, uniform groups of 14 and doubling both have errors below 1%1\%. The improvement in preserving relative prompt weights is therefore much larger at α=3/2\alpha=3/2 in this profile. At this exponent, doubling's conditional-variance factor, with both within-class score variances set to one, is 2.64 times the uniform-14 value. The measurements make the connection between stronger normalization, allocation, and within-prompt precision concrete.

We also strengthened the group-size result. The revised binary theorem (Theorem 4; Theorem 3 in the submission) shows that, for each fixed α>0\alpha>0, group size of order 1/σ21/\sigma^2 is necessary and sufficient to preserve a prescribed fraction of the population-normalized update. Small groups can therefore suppress the very prompt contributions that stronger normalization is intended to emphasize.

Proposition 8 and Table 1 now quantify how the experimental sampler addresses this effect. The sampler doubles from four through 64 responses until both reward outcomes appear. For standard GRPO and p∈[0.01,0.99]p\in[0.01,0.99], its response-level expected-update multiplier lies in [0.9139,1.2259][0.9139,1.2259], compared with approximately 0.17190.1719 to 0.80800.8080 for four fixed responses. The adaptive multiplier is closer to one at every probability in this range, and the table reports the corresponding expected rollout costs.

The revision also connects α\alpha to allocation under a fixed budget. The common variance upper bound is minimized by Ki∝σi1−α.K_i\propto\sigma_i^{1-\alpha}. This allocation is uniform at α=1\alpha=1, the same exponent identified by the sharp 1/K1/K boundary. The exact second-moment calculation in Proposition 8 further accounts for how policy scores vary within and between the two reward classes; Proposition 9 uses it to identify the allocation minimizing leading large-budget gradient variance and mean-squared error. These results explain how normalization strength, group size, and sampling allocation jointly determine the expected update and its noise.

Thank you to the authors for their thorough rebuttal. All of my concerns have been addressed, and I am increasing my rating accordingly.

AuthorsOpenReview ↗

Additional empirical evidence for adaptive sampling

We have completed an additional training comparison connecting our sampling analysis to empirical performance at the standard GRPO normalization exponent, α=1\alpha=1. On GSM8K with the 0.5B model, we compare adaptive sampling against fixed group sizes of 14, 16, and 17 at comparable rollout budgets. Adaptive sampling achieves essentially the same test reward at 2.5 epochs and higher reward at 4.5 and 6 epochs, with improvements of 0.40 and 0.19 percentage points despite using slightly fewer rollouts. We provide the full setup and tables in our detailed results post.

Reviewer_ErkkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyzes how finite rollout groups estimate reward-normalized policy updates. It studies discrete rewards and a family of normalization strengths. For binary rewards, the expected fixed-group update preserves the population direction but changes its magnitude. The analysis identifies how reward variance and group size affect estimation, then studies prompt reweighting, adaptive sampling, and Balanced Aggregation. Mathematical reasoning experiments illustrate the behavior of the normalization choices.

Critical strengths

S1. The finite-group analysis addresses a meaningful regime. Reward discreteness controls the random normalization denominator. The results cover general discrete rewards and several normalization powers, beyond the known binary identity for standard GRPO. They distinguish vanishing absolute error for powers below two from the nonvanishing worst-case error at two. These bounds cover prompts with reward variance approaching zero.

S2. The Balanced Aggregation result explains what length averaging preserves. For binary rewards, the population rule preserves the policy-gradient direction even when correct and incorrect responses have different average lengths. The finite-group analysis then controls the additional bias and error from estimating those class-specific ratios. This gives theoretical understanding of an existing method. The paper appropriately separates that result from exact finite-group collinearity and from the behavior of standard per-response token averaging.

Critical weaknesses

W1. The adaptive experiments do not isolate the allocation benefit. Adaptive runs use about 20 responses per prompt on average, compared with four for fixed sampling. Figure 2 and Table 1 compare epochs, which therefore represent different rollout budgets. The improved scores support those tested configurations, but cannot separate adaptive allocation from the additional samples. The paper discloses the difference, and its main contribution is theoretical. This limits the empirical evidence for the sampler rather than undermining the fixed-group analysis.

Questions for authors

No additional questions. The retained sampling-budget limitation and the theoretical scope are already clear in the paper.

3 comments
AuthorsOpenReview ↗

Response to reviewer Erkk

Thank you again for your careful assessment. We have revised the paper with several results that strengthen the finite-group analysis and explain the behavior of the experimental sampler. The additions are marked in blue.

New Theorem 3 gives matching lower bounds for estimation error, bias, and rollout variance. In particular, the sharp worst-case mean-squared-error and variance rate is K−min⁡1,2−α.K^{-\min\\{1,2-\alpha\\}}. This establishes standard GRPO, α=1\alpha=1, as the strongest normalization power in this family retaining the worst-case 1/K1/K rate. At this exponent, averaging NN independent prompt groups gives the corresponding 1/(NK)1/(NK) rollout-variance rate.

The revised binary theorem (Theorem 4; Theorem 3 in the submission) also gives nonasymptotic attenuation bounds. For each fixed α>0\alpha>0, group size of order 1/σ21/\sigma^2 is necessary and sufficient to preserve a prescribed fraction of the population-normalized update. Proposition 3 gives the corresponding inverse-variance scaling for accuracy and sampling cost under a confidence-based doubling rule.

We now analyze the experimental sampler itself in Proposition 8 and Table 1. For standard GRPO and p∈[0.01,0.99]p\in[0.01,0.99], its response-level expected-update multiplier lies in [0.9139,1.2259][0.9139,1.2259], compared with approximately 0.17190.1719 to 0.80800.8080 for four fixed responses. The adaptive multiplier is closer to one at every probability in this range. Table 1 also gives the expected rollout counts, making the sampling cost of preserving these contributions explicit.

We have also added an initial-model reward profile using 128 responses for each of 320 GSM8K training prompts (new Table 3, Appendix E). Exact averages over reward orderings show that doubling produces mixed-reward groups for 96.1%96.1\% of prompts using 13.41 rollouts per prompt, versus 81.6%81.6\% with uniform groups of 14 and 67.3%67.3\% when the doubling size distribution is assigned independently of rewards at the same expected cost. A uniform-size control matched to that expected cost reaches 80.9%80.9\%. At α=3/2\alpha=3/2, relative squared prompt-weight error after common rescaling is 0.25%0.25\% for doubling and 2.48%2.48\% for uniform groups of 14, using reference weights estimated from the full reward records. The table also quantifies the corresponding increase in the conditional-variance factor. This provides a controlled empirical comparison of allocation at the initial policy.

Finally, Proposition 8 gives an exact second-moment identity, and Proposition 9 uses it to derive the allocation minimizing leading large-budget gradient variance and mean-squared error for a fixed set of prompts. These additions connect the finite-group theory to both the behavior of the sampler and the allocation of a fixed rollout budget.

Thank you for adding Table 3. At the initial policy, adaptive sampling produced both correct and incorrect responses on 96.1% of prompts versus 80.9% under uniform sampling at the same expected rollout budget. This addresses my concern about allocating rollouts to obtain within-prompt reward contrast. Whether adaptive allocation improves final accuracy after training with matched total rollout counts remains open. I still support acceptance based on the finite-group analysis.

AuthorsOpenReview ↗

Adaptive sampling during training: rollout-budget comparison

You noted that whether adaptive allocation improves final accuracy after training with matched total rollout counts remained open. We have now completed training comparisons addressing this question. Adaptive sampling is applied throughout training, and we evaluate the resulting trained policies on the held-out test set. Adaptive sampling achieves essentially the same test reward at 2.5 epochs and higher test reward at 4.5 and 6 epochs, while the fixed-group baselines receive slightly more generated training rollouts.

All comparisons use Qwen2.5-0.5B-Instruct on GSM8K, normalization exponent α=1\alpha=1 (standard GRPO), sequence-mean loss aggregation, the same data processing order, AdamW optimizer, cosine learning-rate schedule, and evaluation procedure. Test reward is evaluated using four responses per prompt. All runs use training seed 20260811.

We compare three fixed-group training runs with the same adaptive training run. Adaptive sampling starts with K=4K=4 and doubles the group size, reusing all previously generated responses, until the group contains both reward outcomes or reaches Kmax⁡=64K_{\max}=64. We choose the fixed group sizes by rounding up the adaptive run’s realized average group size at each training horizon:

Training horizonAdaptive training rolloutsAdaptive mean group sizeFixed group size
2.5 epochs259,97613.886114
4.5 epochs515,46015.310116
6 epochs724,89616.167017

The test rewards of the trained policies at these endpoints are:

Training horizonAdaptiveFixed groupDifference (percentage points)
2.5 epochs0.4435180.443897−0.04
4.5 epochs0.4727070.468726+0.40
6 epochs0.4634190.461524+0.19

These training results complement our earlier sampling diagnostic with the initial policy held fixed. They show that adaptive allocation achieves comparable or modestly better test reward than larger uniform groups when the total training rollout budget is controlled.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.