acceptodds
Under review as a conference paper at ICLR 2027

Within-Class Score Variation in Standardized Group-Relative Policy Gradients

Abstract

Success rates determine how often a binary-reward group contains a learning signal, but they do not determine the variability of its policy-gradient update. For the on-policy, unclipped standardized estimator used in Group Relative Policy Optimization (GRPO), we derive an exact finite-group covariance decomposition into success-count variation and score variation within each reward class. A four-outcome policy shows that the latter can change while success probability and the expected update remain fixed. We also derive the estimator's influence function: estimating the reward standard deviation contributes to the leading covariance, even though that estimate is consistent. In an audit of 3,072 responses from two Qwen checkpoints, within-class variation accounts for median covariance fractions of 97.8% and 82.4% at group size eight, measured in fixed last-layer LoRA coordinates and conditional on mixed empirical response pools. Validation with 4,096 independent responses separates this algebraic identity from moment-estimation accuracy: 32-response calibration pools underpredict aggregate covariance by about 16% for the smaller model. A separate nine-run sampling-allocation study gives mean held-out accuracy of 57.67% for covariance-informed allocation and 57.11% for uniform allocation; a descriptive paired interval for the difference includes zero. Success rates therefore omit a substantial source of update variation in the measured score space, but using this additional information for sampling has yet to yield a reliable accuracy gain.

Primary area:
reinforcement learning
Keywords:
GRPO, policy gradients, variance analysis, reward normalization, finite-sample statistics, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GBk3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the covariance of the on-policy, unclipped standardized policy-gradient estimator underlying GRPO. For binary rewards, it derives an exact finite-group decomposition into a success-count component of rank at most one and within-reward-class score covariance components. The analysis shows that success probability alone does not determine gradient variability, even when the expected gradient is held fixed. The paper also derives an influence function that accounts for the contribution of the estimated reward standard deviation to the leading-order covariance. Experiments on two Qwen checkpoints quantify these components in fixed LoRA coordinates and evaluate covariance estimates using independently sampled responses. Finally, the authors investigate covariance-informed sampling allocation. However, using this information for sampling does not yield a reliable held-out accuracy improvement over uniform allocation in the reported experiments.

Critical strengths

The paper provides an explicit finite-group covariance decomposition for the standardized GRPO estimator under binary rewards, separating success-count variation from within-class score variation. This clarifies why success rates alone do not fully characterize gradient variability. The authors complement the analysis with experiments on two Qwen checkpoints, showing that within-class score variation can account for a substantial fraction of the covariance in the examined LoRA coordinates. They also use independently sampled responses to assess the accuracy of the covariance estimates.

Critical weaknesses

First, the motivation and significance of the decomposition seem to be insufficiently established. The introduction emphasizes that success rates do not fully determine gradient covariance, but does not sufficiently explain why this observation is consequential for GRPO training. The presentation focuses heavily on defining and quantifying the components before establishing why readers should care about this particular decomposition.

Second, as also disclosed in the abstract of this paper, the downstream experiments provide limited evidence of practical utility. Covariance-informed allocation improves mean held-out accuracy over uniform allocation by only 0.56 percentage points, with a reported interval of [-1.78, 3.00]. The experiment uses one model, 32 training prompts, and three sampling/order seeds, which limits the strength of the conclusions.

Questions for authors

Could the authors explain more about the intended scientific significance of the covariance decomposition? Beyond establishing that success rates alone do not determine gradient variability, what specific phenomenon in GRPO training does the decomposition help explain, or what concrete training decision should it inform? Please clarify the substantive insight gained from separating these covariance components in, e.g., the training process.

2 comments

Thank you for asking us to state more concretely why the decomposition matters for GRPO training. Its immediate use is to distinguish prompts that have the same success probability but different score geometry. A success-rate-only allocator must treat those prompts as equally variable, whereas the full finite-group expression can assign different rollout counts through the within-class score covariances and mean-score separation. This leads to a specific decision problem: under a fixed rollout budget, do the additional terms change the integer allocation and reduce the held-out covariance of the group-relative update?

We tested this question on frozen, independently split response pools. Each model uses 32 prompts, 512 calibration responses, an independent 512-response validation pool, a total budget of 256, and bounds of 4 to 12 rollouts per prompt. For Qwen2.5-1.5B, held-out equal-prompt covariance is 0.06031 with uniform allocation, 0.05904 with a success-only rule, 0.05883 and 0.05893 with the VIP-Dr.GRPO and VIP-RLOO objectives, and 0.05585 with the full finite-group rule. The full rule reduces the diagnostic by 7.40% from uniform, compared with 2.46% for the strongest success-probability-based VIP objective. For 7B, all non-uniform objectives reach the same boundary allocation and obtain the same 27.00% reduction. In that case, the additional covariance information has no decision value because the constrained allocations coincide.

The result supports a fixed-policy use of the decomposition: it can reveal when success-rate-based allocation omits consequential update variation. It does not show that a local covariance reduction improves final training accuracy. The earlier nine-run accuracy difference remains statistically uncertain, and we will continue to report it as a negative downstream result rather than evidence of a reliable gain.

Thank you for the additional experiment. The 1.5B result shows that the additional covariance information can change rollout allocation and reduce held-out update variability beyond what success-probability-based rules achieve. I also appreciate the contrasting 7B result, where the rules yield the same allocation. This addresses my question about a concrete use of the decomposition under a fixed policy, so the response partially resolves my concern.

Reviewer_weEvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the variance of the policy-gradient update depends not only on success-count variation but also on score-function variation within the correct and incorrect groups. Experiments on two Qwen checkpoints show that this within-class variation accounts for most of the update covariance, but using it to allocate samples has not yet produced a significant accuracy gain.

Critical strengths

The paper reports its negative training result honestly and accounts for the computational cost. It also releases the responses, Gram matrices, and training trajectories, which support reproducibility.

Critical weaknesses

The empirical evaluation is narrow. All experiments use only Qwen2.5 models, and the setting is also small-scale (e.g., the audit uses 48 prompts, training uses 32 prompts with 256 updates and three seeds). In addition, all analyses are on GSM8K, so it is unclear whether the findings generalize to other model families or to other tasks, such as harder math or code generation. Furthermore, this method isn't particularly novel.

Questions for authors

The authors might be better off adding this reference [1,2].

[1] Li, Shuozhe, et al. "Learning robust reasoning through guided adversarial self-play." arXiv preprint arXiv:2602.00173 (2026). [2] Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards, Hieu Trung Nguyen Hieu Trung Nguyen,Bao Nguyen, Wenao Ma, Yuzhi Zhao, Ruifeng She, Viet Anh Nguyen, ICLR 2026

2 comments

Thank you for pointing out the limited empirical breadth and for suggesting these references. Two Qwen checkpoints on GSM8K cannot establish generality across model families or tasks, and we will make that limitation more prominent rather than extrapolate the measured covariance fractions.

Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards is the closest allocation work, so we added a matched frozen-policy comparison using its Dr.GRPO and RLOO objectives. The experiment uses 32 prompts, independent 512-response calibration and validation pools, a budget of 256, and bounds of 4 to 12 rollouts per prompt. On the 1.5B checkpoint, the strongest VIP objective reduces held-out equal-prompt covariance by 2.46% from uniform, while the full finite-group covariance rule reduces it by 7.40%. On 7B, all non-uniform rules coincide and reduce the diagnostic by 27.00%. These numbers support only a limited conclusion: within-class score geometry can matter when it changes the allocation, but it adds no decision value when the constrained optima coincide.

We will also discuss Learning Robust Reasoning through Guided Adversarial Self-Play in the broader RLVR context while keeping the distinction clear: it studies robustness to corrupted conditioning rather than rollout allocation from gradient covariance. The new comparison is a frozen-policy diagnostic, not an online VIP reproduction, and it does not remove the cross-family or cross-task limitation you identified.

Thanks for the additional comment.

I still think that relying on such a narrow family of models leaves the method's overall efficiency and generality insufficiently supported. So, I'll keep my score.

Reviewer_3UqSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors investigate the variance structure of standardized group-relative policy gradients under binary rewards. They derive an exact finite-group covariance decomposition that separates fluctuations in the number of successful responses from within-class variations in likelihood-score vectors and construct a four-outcome example demonstrating that policies with identical success rates and expected gradient updates can exhibit different gradient covariance structures. They also derive an influence-function characterization of the standardized estimator, showing that fluctuations in the estimated reward standard deviation contribute to the leading-order covariance. Empirically, the authors analyze response-level gradients from two Qwen2.5 checkpoints using fixed last-layer LoRA coordinates. The results suggest that within-class score variation accounts for a substantial fraction of the measured gradient covariance.

Critical strengths
  1. The paper is well-written and easy to follow;

  2. The exact finite-group covariance decomposition is clearly formulated and accompanied by a proof. The four-outcome construction effectively demonstrates that identical success rates and expected updates do not necessarily imply identical gradient covariance. The influence-function analysis also clarifies the contribution of random reward normalization to the leading-order covariance.

  3. The authors distinguish exact covariance identities evaluated on empirical response distributions from the accuracy of covariance predictions on independently generated responses.

Critical weaknesses
  1. Although the proposed covariance decomposition takes a new and GRPO-specific form, the underlying mathematical techniques are relatively straightforward. In particular, Theorem 1 follows directly from conditioning on the number of successful responses, applying the law of total covariance, and evaluating elementary binomial moments. The influence-function result similarly relies on a standard application of the multivariate delta method.

  2. The underlying group-standard-deviation identity and its expected scalar gain have already been investigated by [1], Theorem 1 extends this identity to an explicit class-conditional covariance decomposition. Given that this result follows naturally from the established identity and standard conditional covariance arguments, its conceptual novelty may be weaker than suggested in the paper.

  3. [2] has already investigated gradient-variance-aware rollout allocation in RLVR, including variance-based budget optimization and practical allocation procedures. The paper similarly formulates a variance-minimization objective and derives an allocation rule proportional to the square root of the estimated gradient variance, while the authors did not discuss or compare with [2].

[1] Bay, Y. Y., & Yearick, K. A. (2026). GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity. arXiv preprint arXiv:2607.00152.

[2] Nguyen, T. H., Nguyen, B., Ma, W., Zhao, Y., She, R., & Nguyen, V. A. (2026, April). Adaptive rollout allocation for online reinforcement learning with verifiable rewards. In International Conference on Learning Representations (Vol. 2026, pp. 136100-136132).

Questions for authors

Please see Weaknesses.

2 comments

Thank you for drawing our attention to the relationship with the group-standard-deviation identity and VIP. We accept that conditioning and the delta method are standard tools and will narrow our novelty claim. Bay and Yearick characterize the scalar effect of reward normalization. We instead condition on the reward count to expose the class-conditional score covariances and mean-score separation that the success probability does not identify. The contribution is this GRPO-specific finite-group covariance expression and its empirical measurement, not a new total-covariance technique.

Following your suggestion, we implemented the VIP Dr.GRPO and RLOO allocation objectives on the same frozen response pools used by our method. For each model, the experiment contains 32 prompts, 512 calibration responses, an independent 512-response validation pool, a total budget of 256 rollouts, and integer bounds of 4 to 12 rollouts per prompt. Every allocation is solved by exact integer dynamic programming.

For Qwen2.5-1.5B, held-out equal-prompt covariance is 0.06031 with uniform allocation, 0.05904 with our success-only asymptotic rule, 0.05883 with the VIP-Dr.GRPO objective, 0.05893 with the VIP-RLOO objective, and 0.05585 with the full finite-group covariance rule. The respective reductions from uniform are 2.10%, 2.46%, 2.29%, and 7.40%. In this controlled case, the full rule therefore adds 4.94 percentage points of reduction beyond the strongest success-probability-based VIP objective. For 7B, all four non-uniform objectives reach the same boundary allocation and reduce the diagnostic by 27.00%. This negative result shows that the additional geometry does not change every allocation problem.

These results also define the relationship between the methods. VIP is an online allocation system based on predicted success probabilities; our decomposition identifies covariance information that success probability alone cannot recover. We do not present our method as a replacement for VIP. The new experiment is a frozen-policy allocator diagnostic using calibration probabilities: it omits VIP's GP predictor, online updates, and training loop, and is not a full VIP reproduction. We also retain the negative nine-run result because the current evidence does not establish a reliable downstream accuracy gain.

Thank the authors for their responses. The authors acknowledge that conditioning and the delta method are standard tool, and considering the similarity of this paper and VIP, I think the submission's contributions are incremental. I thus will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.