- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
CorrGRPO modifies GRPO for multi-reward learning by replacing the summed reward's standard-deviation denominator with a correlation-based normalization, while retaining the centered summed reward in the numerator. The paper shows that this amounts to rescaling each group's GRPO advantages rather than changing their relative values within a group. Experiments on coding, tool calling, and agent security generally show improved task-level performance, though some individual metrics exhibit tradeoffs.
Critical strengths
- The covariance analysis shows how reward scales affect GRPO's normalization and motivates a simple correlation-based alternative.
- The paper clearly identifies CorrGRPO's effect as rescaling advantages between groups while preserving their relative values within each group.
- Improvements over GRPO across coding, tool calling, and agent security suggest that the small change can be useful in different multi-reward settings.
Critical weaknesses
- The motivation illustrates scale imbalance with a reward reaching 8, but both rewards in the agent-security task are binary in . The paper does not show whether their within-group variances differ enough for the proposed scale-based mechanism.
- There appears to be a material mismatch between the mathematical definition and the implementation. The paper defines the CorrGRPO denominator as , whereas Listing 1 uses . These are not equivalent when is small.
- CorrGRPO preserves the centered summed-reward numerator, so for a fixed rollout group it is exactly a positive scalar rescaling of GRPO advantages. The paper's reward-balancing claim is therefore stronger than the mechanism establishes.
- The gains of CorrGRPO may reflect correlation-conditioned group scaling rather than a benefit of its treatment of reward dependence. The paper lacks controls that match the distribution of update scales while decoupling each group's scaling factor from its correlations.
- CorrGRPO relies on sample correlations computed within each rollout group, yet several experiments use only four rollouts per prompt. The paper reports no sensitivity analyses over rollout-group size and number of reward components.
- The baseline coverage is uneven across domains: GDPO is evaluated only for coding, while RDPO (the most directly relevant correlation-aware multi-reward baseline) is discussed but not evaluated.
Questions for authors
- Can the authors compare CorrGRPO against controls that isolate correlation-conditioned scaling from generic group-level rescaling?
- What happens if one reward component is rescaled and the actual summed-reward numerator is recomputed?
- How stable are the results across training seeds and rollout group sizes, especially when only four rollouts are evaluated?
Additional Question
My new question is here: If the problem stems from normalizing by the standard deviation of the total reward, why not simply remove this normalization? What additional benefit does correlation-based normalization provide over a no-std baseline such as Dr. GRPO?