acceptodds
Under review as a conference paper at ICLR 2027

Finite Rollout Groups as Score-Conditioned Policy-Gradient Oracles

Abstract

A finite rollout group determines which population policy-gradient fields are available, not only how accurately they are estimated. We study iid on-policy groups over finite outcomes with outcome-measurable, permutation-equivariant, stop-gradient coefficients and no direct policy input. Conditioning leaves G − 1 peers: the conditional fields are exactly polynomials of degree at most G−1. A common-baseline quotient separates field representation from objective existence. Ordinary reward standardization has an exact boundary: for every finite G ≥ 2 and every positive strictly increasing variance denominator, a scalar simplex potential exists if and only if rewards have at most two distinct levels. Count-based rules yield exact self-inclusion and centering transformations. Under affine reuse of two fixed coefficient rules across linear reward weights, with positive response, we further prove Θ(G−2) minimax loss at the best surrogate optimum over a fixed strongly concave target class. Controlled Qwen2.5 studies at 1.5B and 7B identify count-rule objectives: all 65 objective-map endpoints are nearest their assigned optimum among three analytic candidates. Separate tests recover a rollout threshold, self-inclusion fixed points, and the predicted zero-to-nonzero circulation transition from two to three reward levels under ordinary standardization.

Primary area:
learning theory
Keywords:
reinforcement learning, policy gradients, group relative policy optimization, reward normalization, finite-sample theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_agnCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

When training language models with reinforcement learning, algorithms often generate a small group of responses for each prompt and compare their scores (e.g., GRPO, which rewards each response by how much it beats the group average). Most work assume that using a small group size only adds random noise to the training process. This paper proves that the group size actually changes the underlying goal the model is trying to achieve.

The authors show that, for a broad class of update rules that includes the core of GRPO, a group of GG responses can only produce update rules shaped like polynomials of degree G−1G-1 (at most). Because of this limit, common training tricks behave in surprising ways. First, dividing rewards by the group's standard deviation works cleanly when answers are simply graded as pass or fail. However, if a task uses three or more reward levels—like fail, partial credit, and full credit—the updates begin to swirl (intuitively, due to groups with mixed outcomes having higher variance, which unevenly discounts their updates)), and no longer optimize any single objective function. Second, seemingly harmless choices, like subtracting the group average or reusing a response's own count, quietly shift the model's final behavior. Experiments on Qwen2.5 models confirm these exact mathematical predictions.

Critical strengths
  • The paper covers an important topic in a (what seems to me) novel and interesting theoretical direction. The connection of the theoretical results with the empirical predictions is quite helpful. The authors work through various and varied consequences of their theory and approach, connecting it to concrete training choices practitioners make (reward standardization, group-mean centering, self-inclusion, and group size) and to recent methods such as MaxRL. In that sense, there many be several communities interested in the results herein.
Critical weaknesses
  • The paper is not always easy to read. It is quite dense and full of mathematical jargon that may take those not well-acquained with this field much time to parse.

  • The theoretical framework (Eq. 1) relies on assumptions that may exclude several mechanisms standard in practical RLVR and RLHF pipelines (e.g., off-policy reuse across multiple mini-batch epochs with clipped importance ratios, KL regularization toward a reference policy, per-token length normalization (which replaces the sequence-level score sis_i with a length-weighted score and breaks outcome-conditional independence), and learned critics). Because the empirical evaluations (Section 7, Appendix E.2) disable all of these components, the clean experimental record here confirm that the theory holds for pure on-policy, single-step, sequence-level updates, but might be limited in application to these other domains. Standard production recipes coincide with this idealized oracle only on the very first gradient step of each batch (where the importance ratio is 11) in the absence of KL penalties and length normalization. It remains unclear how strongly the exact polynomial hierarchy and count-rule objective shifts persist once these practical mechanisms are reintroduced, although in principle further ablations could show this.

  • Theorem 3 establishes a nice statement showing that ordinary within-group variance normalization admits a scalar simplex potential if and only if rewards take at most two distinct levels. However, as noted in Section 9, non-integrability of the population vector field does not automatically imply divergence, limit cycles, or optimization failure; the exploratory trajectory in Appendix E.7 shows the policy still ascending smoothly toward the highest-reward outcome despite nonzero curl. To help readers understand the practical impact of crossing the three-reward-level boundary, the paper could clarify whether lacking a single objective function ever changes where the model ultimately converges -- or if the policy still reliably climbs to the highest-reward answer anyway, with the 'swirl' only shifting the final outcome when combined with regularizers like a KL penalty.

Questions for authors

Related to the above, does losing the objective under three or more reward levels ever change where training ends, or only the path it takes to get there? In principle, the update sequence could differ, but if the standardized weights always favor higher-reward answers, as they appear to in the E4 setting, a sufficiently flexible model would still end at the highest-reward answer, the same endpoint as plain expected-reward maximization?

Reviewer_Xh3KOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper characterizes the expected policy-gradient updates that finite rollout groups can represent and studies when these updates correspond to a scalar objective. It also analyzes how self-inclusion and group normalization change the induced updates, with controlled experiments illustrating the theoretical results.

Critical strengths

The distinction between representing an update field and establishing the existence of an objective is useful.

The controlled experiments provide concrete examples of how group-level computations can change the learned distribution.

Critical weaknesses

The presentation is difficult to follow. The paper moves between polynomial representation, objective existence, self-inclusion, and a restricted minimax result without a sufficiently clear central narrative. The dense notation makes it difficult to distinguish the main insight from supporting technical results or understand which conclusions should matter to practitioners.

The significance of exact finite-group characterization is insufficiently established. Exact unbiasedness or exact agreement with a prescribed gradient is not generally necessary for effective optimization. Controlled bias or asymptotically accurate updates may be sufficient. The paper needs a clearer connection between its structural results and optimization error, convergence, or the tradeoff between bias and variance. Similarly, the absence of an exact scalar potential does not itself imply undesirable learning behavior, as the authors acknowledge.

The experiments mainly confirm the analytic constructions. They show that different rules approach their predicted targets, but provide limited evidence about when the resulting discrepancies harm performance or when correcting them justifies the additional computation.

Questions for authors

For a fixed target objective, how do the relevant bias and optimization error change as the group size increases?

Under a fixed total rollout budget, when do exact or leave-one-out constructions outperform simpler biased alternatives?

What concrete learning consequence follows from the lack of a scalar potential in the settings considered?

Reviewer_PJvAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

I found the paper very difficult to read and follow.

Critical strengths

I could not assess the strengths with sufficient confidence.

Critical weaknesses

The presentation is excessively compressed and places too much burden on the reader to reconstruct the explanations and connections between results.

Questions for authors

Please read through the manuscript from the perspective of someone unfamiliar with the work and reconsider its readability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.