acceptodds
Under review as a conference paper at ICLR 2027

CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning

Abstract

Group Relative Policy Optimization (GRPO) is widely used to train reasoning language models, where it computes advantages by centering and normalizing rewards across rollouts of the same prompt. With multiple rewards, GRPO sums the reward components and normalizes the total reward by its within-group standard deviation. The corresponding variance equals the sum of all pairwise reward covariances.For a fixed centered reward, larger aggregate covariance produces smaller advantages, and vice versa, allowing update magnitudes to adapt to reward dependence. However, correlated rewards with large scales can dominate this normalization and suppress signals from smaller-scale rewards. We propose Correlation-Normalized GRPO (CorrGRPO), which normalizes pairwise covariances into Pearson correlation coefficients. CorrGRPO keeps the centered total reward unchanged while balancing the influence of differently scaled rewards on the correlation-based normalization. This allows advantage magnitudes to adapt to reward correlations without the normalization being dominated by large-scale reward components. We compare CorrGRPO with GRPO and other variants on code generation, tool calling, and agent security, using models ranging from 0.5B to 8B parameters. These tasks all involve multiple rewards that can improve together or present tradeoffs. Results show improvements across all three domains and an outward expansion of the empirical Pareto frontier between competing reward objectives.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, LLM, Multi-Reward Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VPsoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CorrGRPO modifies GRPO for multi-reward learning by replacing the summed reward's standard-deviation denominator with a correlation-based normalization, while retaining the centered summed reward in the numerator. The paper shows that this amounts to rescaling each group's GRPO advantages rather than changing their relative values within a group. Experiments on coding, tool calling, and agent security generally show improved task-level performance, though some individual metrics exhibit tradeoffs.

Critical strengths
  • The covariance analysis shows how reward scales affect GRPO's normalization and motivates a simple correlation-based alternative.
  • The paper clearly identifies CorrGRPO's effect as rescaling advantages between groups while preserving their relative values within each group.
  • Improvements over GRPO across coding, tool calling, and agent security suggest that the small change can be useful in different multi-reward settings.
Critical weaknesses
  • The motivation illustrates scale imbalance with a reward reaching 8, but both rewards in the agent-security task are binary in [0,1][0,1]. The paper does not show whether their within-group variances differ enough for the proposed scale-based mechanism.
  • There appears to be a material mismatch between the mathematical definition and the implementation. The paper defines the CorrGRPO denominator as Q+ε\sqrt{Q}+\varepsilon, whereas Listing 1 uses max⁡(Q,0)+ε\sqrt{\max(Q,0)+\varepsilon}. These are not equivalent when QQ is small.
  • CorrGRPO preserves the centered summed-reward numerator, so for a fixed rollout group it is exactly a positive scalar rescaling of GRPO advantages. The paper's reward-balancing claim is therefore stronger than the mechanism establishes.
  • The gains of CorrGRPO may reflect correlation-conditioned group scaling rather than a benefit of its treatment of reward dependence. The paper lacks controls that match the distribution of update scales while decoupling each group's scaling factor from its correlations.
  • CorrGRPO relies on sample correlations computed within each rollout group, yet several experiments use only four rollouts per prompt. The paper reports no sensitivity analyses over rollout-group size and number of reward components.
  • The baseline coverage is uneven across domains: GDPO is evaluated only for coding, while RDPO (the most directly relevant correlation-aware multi-reward baseline) is discussed but not evaluated.
Questions for authors
  • Can the authors compare CorrGRPO against controls that isolate correlation-conditioned scaling from generic group-level rescaling?
  • What happens if one reward component is rescaled and the actual summed-reward numerator is recomputed?
  • How stable are the results across training seeds and rollout group sizes, especially when only four rollouts are evaluated?
Reviewer_anCvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CorrGRPO (Correlation-Normalized GRPO) to improve advantage estimation in multi-reward reinforcement learning. The authors observe that standard GRPO sums multiple rewards and normalizes the result using the within-group standard deviation of the total reward. Since the variance in this denominator equals the sum of all pairwise reward covariances, it can be dominated by reward components with large scales of variation, thereby diminishing important dependencies among smaller-scale rewards.

CorrGRPO retains the centered total reward as the numerator but replaces the covariance terms in the denominator with the sum of Pearson correlation coefficients. This removes the influence of reward scale while allowing the advantage magnitude to adapt dynamically to correlations among rewards. The experiments cover three multi-reward settings—code generation, tool calling, and the utility–security trade-off for agents—using models ranging from 0.5B to 8B parameters. The results show that, compared with standard GRPO and related baselines, CorrGRPO generally improves code correctness and efficiency, tool-calling accuracy, and the overall balance between task utility and resistance to prompt-injection attacks.

Critical strengths
  1. The method is clearly motivated, and the improvement is simple yet effective. The paper reinterprets GRPO in multi-reward settings from a covariance perspective, pointing out that reward components with larger scales may dominate the normalization denominator, thereby weakening the role of correlations among small-scale rewards. CorrGRPO replaces covariance with the Pearson correlation coefficient while retaining the centered total reward and the original reward weights; the modification is concise and has a clear theoretical motivation.

  2. The experiments have broad coverage, and the results are relatively consistent. The paper conducts experiments in three different multi-reward scenarios: code generation, tool calling, and the trade-off between agent utility and safety, covering multiple models and parameter scales from 0.5B to 8B. CorrGRPO generally outperforms GRPO and related baselines across different tasks and models, indicating that the method has a certain degree of generality rather than being designed only for a specific dataset or task.

Critical weaknesses

None

Questions for authors

I have no question about the paper it self. But I want to discuss: I think length reward/punish can also be regarded as a type of reward and incorporated into the multi-reward system. Have the authors conducted any related experiments on this case? (Even if you cannot answer this question, I will not lower the paper score; this is just an extension topic.)

2 comments
Reviewer_anCvOpenReview ↗

Additional Question

My new question is here: If the problem stems from normalizing by the standard deviation of the total reward, why not simply remove this normalization? What additional benefit does correlation-based normalization provide over a no-std baseline such as Dr. GRPO?

Even if it is difficult for the authors to run experiments within the remaining time to answer this question, I will still maintain a positive score. Relax.

Thank you for your question. DR.GRPO (GRPO without standard normalization) serves as an important baseline. We've already used it to train the qwen2.5-coder-7b-instruct model on the LeetCode dataset, following the training settings outlined in the paper. Our comparisons of the results show that CorrGRPO consistently outperforms these baselines.

MethodLeetCodeHumanEvalMBPPLCB v6Avg. Pass@1
GRPO15.79%73.78%78.40%21.14%47.28%
Dr.GRPO (no-std)18.42%78.05%77.80%22.86%49.28%
CorrGRPO24.12%78.66%78.60%24.57%51.49%

We find that Dr.GRPO surpasses GRPO by 2% in pass@1, and CorrGRPO exceeds Dr.GRPO by more than 2% in the same metric.

We hypothesize that:

  1. GRPO's rescaling mechanism is overwhelmed/broken by the large-scale rewards. Additionally, std norm introduces bias, resulting in the positive effects (rescaling) being outweighed by the negative effects (bias). This could explain why Dr.GRPO is better than GRPO.
  2. Correlation-based normalization (CorrGRPO) mitigates this issue, allowing it to outperform Dr.GRPO.

To further validate our hypothesis, we need more time to expand the experiments to additional settings. We hope our response adequately addresses your concerns.

Reviewer_bvUBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CorrGRPO modifies advantage normalization for multi-reward language-model reinforcement learning. Starting from the covariance decomposition of total reward variance, it replaces GRPO's total-reward standard-deviation denominator with the square root of the sum of reward correlations, while retaining the centered weighted total reward as the numerator. This preserves each fixed group's reward-surrogate gradient direction but changes the relative weighting of groups and the balance with KL regularization. Experiments span code generation, tool calling, and agent utility/security with 0.5B–8B models, alongside transfer evaluations and integrations with other optimization methods. The reported aggregate results support a promising, inexpensive practical modification, although gains are not universal across individual metrics.

Critical strengths
  1. The modification is simple and its mathematical scope is presented transparently. Equation (2), the implementation, and Appendices L.2–L.4 explain the handling of zero-variance components, compatibility with clipping, and the distinction between denominator invariance and full-advantage invariance. The authors also correctly distinguish group-dependent rescaling from a single global learning-rate change. This makes the method straightforward to understand and implement without a critic or matrix inversion. [pp. 4, 14, 36–38]

  2. The empirical coverage is a substantive strength. CorrGRPO has the highest four-benchmark average coding Pass@1 at all four model scales, while aggregate tool-call scores improve for all three backbones. Transfer evaluations, training curves, timing measurements, and integration experiments provide additional context. The paper also reports mixed results, including a decrease when combined with GDPO. These broad findings are encouraging and should not be dismissed solely because some individual tool-call gains are small. [pp. 5–9, 25–27, 34]

Critical weaknesses
  1. Positive denominators do not establish moderate advantage magnitudes. With four rollouts, let u=(0,1,0,1), R1=0.25u, and R2=0.75(1-u). Their correlation sum is zero, but the centered total reward is ±0.25; Listing 1's stabilizer therefore produces advantages of ±250. This illustrates a relevant possible instability for heterogeneous, conflicting rewards, not an observed failure of the reported runs. The missing information is how often small correlation sums and extreme advantages occur in the actual training distributions. The mathematical and implementation conventions for epsilon should also be made consistent. [pp. 14, 38–39]

  2. The evidence supports a promising practical method more strongly than a settled explanation or universal superiority. Independent training-repeat uncertainty is not reported, and the experiments do not isolate off-diagonal correlations from removing standard-deviation normalization and changing reward/KL scaling. The latter limits attribution of the gains to the proposed dependence mechanism; it does not invalidate the aggregate performance evidence. Claims should reflect both the broad positive results and these remaining uncertainties. [pp. 5–8, 31–32, 34–38]

Questions for authors
  1. In the existing runs, what are the distributions of the correlation sum Q, advantage magnitudes, and group scaling factors, especially their tails and the frequency of stabilizer-dominated updates? Please clarify the epsilon convention; these diagnostics would distinguish a rare boundary case from a practically consequential issue. [pp. 14, 38–39]

  2. Which principal gains have been checked across independent training runs, and how were the reported checkpoints selected? Any available uncertainty estimates would help calibrate the preliminary empirical evidence. [pp. 5–8, 31–32]

  3. Can the mechanism claim be narrowed to the demonstrated effect, or supported by one representative matched comparison with centered total rewards or a diagonal-only denominator? This is an attribution question, not a request for a new benchmark suite. [pp. 34–38]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.