- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
When training language models with reinforcement learning, algorithms often generate a small group of responses for each prompt and compare their scores (e.g., GRPO, which rewards each response by how much it beats the group average). Most work assume that using a small group size only adds random noise to the training process. This paper proves that the group size actually changes the underlying goal the model is trying to achieve.
The authors show that, for a broad class of update rules that includes the core of GRPO, a group of responses can only produce update rules shaped like polynomials of degree (at most). Because of this limit, common training tricks behave in surprising ways. First, dividing rewards by the group's standard deviation works cleanly when answers are simply graded as pass or fail. However, if a task uses three or more reward levels—like fail, partial credit, and full credit—the updates begin to swirl (intuitively, due to groups with mixed outcomes having higher variance, which unevenly discounts their updates)), and no longer optimize any single objective function. Second, seemingly harmless choices, like subtracting the group average or reusing a response's own count, quietly shift the model's final behavior. Experiments on Qwen2.5 models confirm these exact mathematical predictions.
Critical strengths
- The paper covers an important topic in a (what seems to me) novel and interesting theoretical direction. The connection of the theoretical results with the empirical predictions is quite helpful. The authors work through various and varied consequences of their theory and approach, connecting it to concrete training choices practitioners make (reward standardization, group-mean centering, self-inclusion, and group size) and to recent methods such as MaxRL. In that sense, there many be several communities interested in the results herein.
Critical weaknesses
-
The paper is not always easy to read. It is quite dense and full of mathematical jargon that may take those not well-acquained with this field much time to parse.
-
The theoretical framework (Eq. 1) relies on assumptions that may exclude several mechanisms standard in practical RLVR and RLHF pipelines (e.g., off-policy reuse across multiple mini-batch epochs with clipped importance ratios, KL regularization toward a reference policy, per-token length normalization (which replaces the sequence-level score with a length-weighted score and breaks outcome-conditional independence), and learned critics). Because the empirical evaluations (Section 7, Appendix E.2) disable all of these components, the clean experimental record here confirm that the theory holds for pure on-policy, single-step, sequence-level updates, but might be limited in application to these other domains. Standard production recipes coincide with this idealized oracle only on the very first gradient step of each batch (where the importance ratio is ) in the absence of KL penalties and length normalization. It remains unclear how strongly the exact polynomial hierarchy and count-rule objective shifts persist once these practical mechanisms are reintroduced, although in principle further ablations could show this.
-
Theorem 3 establishes a nice statement showing that ordinary within-group variance normalization admits a scalar simplex potential if and only if rewards take at most two distinct levels. However, as noted in Section 9, non-integrability of the population vector field does not automatically imply divergence, limit cycles, or optimization failure; the exploratory trajectory in Appendix E.7 shows the policy still ascending smoothly toward the highest-reward outcome despite nonzero curl. To help readers understand the practical impact of crossing the three-reward-level boundary, the paper could clarify whether lacking a single objective function ever changes where the model ultimately converges -- or if the policy still reliably climbs to the highest-reward answer anyway, with the 'swirl' only shifting the final outcome when combined with regularizers like a KL penalty.
Questions for authors
Related to the above, does losing the objective under three or more reward levels ever change where training ends, or only the path it takes to get there? In principle, the update sequence could differ, but if the standardized weights always favor higher-reward answers, as they appear to in the E4 setting, a sufficiently flexible model would still end at the highest-reward answer, the same endpoint as plain expected-reward maximization?