- Rating
- 1
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the covariance of the on-policy, unclipped standardized policy-gradient estimator underlying GRPO. For binary rewards, it derives an exact finite-group decomposition into a success-count component of rank at most one and within-reward-class score covariance components. The analysis shows that success probability alone does not determine gradient variability, even when the expected gradient is held fixed. The paper also derives an influence function that accounts for the contribution of the estimated reward standard deviation to the leading-order covariance. Experiments on two Qwen checkpoints quantify these components in fixed LoRA coordinates and evaluate covariance estimates using independently sampled responses. Finally, the authors investigate covariance-informed sampling allocation. However, using this information for sampling does not yield a reliable held-out accuracy improvement over uniform allocation in the reported experiments.
Critical strengths
The paper provides an explicit finite-group covariance decomposition for the standardized GRPO estimator under binary rewards, separating success-count variation from within-class score variation. This clarifies why success rates alone do not fully characterize gradient variability. The authors complement the analysis with experiments on two Qwen checkpoints, showing that within-class score variation can account for a substantial fraction of the covariance in the examined LoRA coordinates. They also use independently sampled responses to assess the accuracy of the covariance estimates.
Critical weaknesses
First, the motivation and significance of the decomposition seem to be insufficiently established. The introduction emphasizes that success rates do not fully determine gradient covariance, but does not sufficiently explain why this observation is consequential for GRPO training. The presentation focuses heavily on defining and quantifying the components before establishing why readers should care about this particular decomposition.
Second, as also disclosed in the abstract of this paper, the downstream experiments provide limited evidence of practical utility. Covariance-informed allocation improves mean held-out accuracy over uniform allocation by only 0.56 percentage points, with a reported interval of [-1.78, 3.00]. The experiment uses one model, 32 training prompts, and three sampling/order seeds, which limits the strength of the conclusions.
Questions for authors
Could the authors explain more about the intended scientific significance of the covariance decomposition? Beyond establishing that success rates alone do not determine gradient variability, what specific phenomenon in GRPO training does the decomposition help explain, or what concrete training decision should it inform? Please clarify the substantive insight gained from separating these covariance components in, e.g., the training process.
2 comments
Thank you for the additional experiment. The 1.5B result shows that the additional covariance information can change rollout allocation and reduce held-out update variability beyond what success-probability-based rules achieve. I also appreciate the contrasting 7B result, where the rules yield the same allocation. This addresses my question about a concrete use of the decomposition under a fixed policy, so the response partially resolves my concern.
Thank you for asking us to state more concretely why the decomposition matters for GRPO training. Its immediate use is to distinguish prompts that have the same success probability but different score geometry. A success-rate-only allocator must treat those prompts as equally variable, whereas the full finite-group expression can assign different rollout counts through the within-class score covariances and mean-score separation. This leads to a specific decision problem: under a fixed rollout budget, do the additional terms change the integer allocation and reduce the held-out covariance of the group-relative update?
We tested this question on frozen, independently split response pools. Each model uses 32 prompts, 512 calibration responses, an independent 512-response validation pool, a total budget of 256, and bounds of 4 to 12 rollouts per prompt. For Qwen2.5-1.5B, held-out equal-prompt covariance is 0.06031 with uniform allocation, 0.05904 with a success-only rule, 0.05883 and 0.05893 with the VIP-Dr.GRPO and VIP-RLOO objectives, and 0.05585 with the full finite-group rule. The full rule reduces the diagnostic by 7.40% from uniform, compared with 2.46% for the strongest success-probability-based VIP objective. For 7B, all non-uniform objectives reach the same boundary allocation and obtain the same 27.00% reduction. In that case, the additional covariance information has no decision value because the constrained allocations coincide.
The result supports a fixed-policy use of the decomposition: it can reveal when success-rate-based allocation omits consequential update variation. It does not show that a local covariance reduction improves final training accuracy. The earlier nine-run accuracy difference remains statistically uncertain, and we will continue to report it as a negative downstream result rather than evidence of a reliable gain.