acceptodds
Under review as a conference paper at ICLR 2027

WHICH VERIFIER ERRORS REACH THE GRADIENT? A MOMENT DECOMPOSITION FOR AUDITING GROUP- RELATIVE REINFORCEMENT LEARNING

Abstract

Group-relative RL with verifiable rewards updates the policy with score-weighted contrasts of verifier rewards, yet verifiers are audited with the intraclass correlation (ICC) or effective sample size (ESS), statistics designed for a group mean. We characterize exactly which properties of verifier error reach the update. Casting the verifier-induced perturbation as a U-statistic, Hoeffding’s variance formula decomposes its second moment into three channels: the within-group error variance, which a mean-square-within (MSW) audit estimates; a coupling between squared error and score norm; and a squared reward-hacking gradient, whose bias accumulates across a batch instead of averaging out. The ICC and ESS capture none of them. The decomposition covers length-normalized updates and shared verifier randomness, and yields an unbiased audit statistic. On two Qwen2.5 policies the MSW channel carries 0.90–1.00 of the per-group perturbation, so variance-based audits are a sound default; yet for a format-sensitive checker the systematic channel reaches 0.26 [0.15,0.37] of a 64-prompt batch update (post hoc). In a pre-specified three-seed audit with LLM judges, ESS ranked three verifiers in the reverse order of their gradient perturbation, while MSW ranked them correctly. These results give practitioners a principled, inexpensive audit target.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, RLVR, GRPO, verifier noise, correlated errors, policy gradients, reward modeling, robustness, statistical diagnostics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9gxKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how verifier errors affect a mean-centered, group-relative policy-gradient update. It applies a classical variance decomposition to separate the expected squared gradient perturbation into within-group error variance, coupling between errors and score-vector magnitudes, and a component associated with systematic gradient bias. It also examines how bias behaves under batch averaging. Experiments on Qwen2.5 policies find that the MSW component usually dominates per-group perturbation, while one post hoc condition suggests a substantial systematic component at larger batch sizes. A three-seed audit shows that ESS can rank verifiers opposite to their measured perturbations, although the MSW baseline ranks them correctly. The paper does not establish a connection between the additional audit components and improved training outcomes

Critical strengths

The finding that MSW accounts for most per-group perturbation provides evidence for a simpler auditing default. The authors also clearly report failed predictions and the null learning result, helping readers understand the limits of their conclusions.

Critical weaknesses

The authors explicitly acknowledge that the main decomposition, estimator, and several supporting observations are classical. The contribution therefore depends substantially on the usefulness of their audit interpretation and empirical findings. Yet MSW explains 90–100% of per-group perturbation in the main sequence-level audits, and the MSW baseline correctly ranks the verifiers in the prospective comparison where ESS fails (Sections 5–6). The comparison establishes a limitation of ESS but does not establish the value of the additional gradient-based measurements. Appendix O also does not validate the finer prediction through learning outcomes. A diagnostic paper need not improve a training benchmark, but I would expect stronger evidence that the additional components enable a useful diagnosis or decision that the simpler audit misses.

In addition, the gap between identifying mathematically distinct error components and demonstrating that distinguishing them provides enough additional scientific or practical value, is my concern. The paper makes progress on the first; its evidence for the second is presently too limited for me to recommend acceptance.

Questions for authors
  1. Clarification: Which completed result most clearly supports an auditing conclusion or decision that an MSW-based audit would miss? How should this result be weighed against the null learning result in Appendix O?

  2. Robustness evidence: Does the batch-bias finding remain when variability across independent sketches is included, or when checked using exact gradients on a feasible subset? How much would this change its confidence interval?

  3. Applicability: Is there direct evidence that the additional components remain informative for the actual update used in training, including reward-standard-deviation normalization?

1 comment

Q1. Which completed result supports a decision an MSW audit would miss? A grader and its permuted twin share the error rate, confusion matrix, MSW, ICC and Kish ESS in every group. Yet GRPO ends them at 0.042 and 0.495 accuracy (Figure 1). The twin reshuffles the grader's verdicts at random among completions with the same gold label in a group.

The drift audit, a gradient-level check on a gold-labelled sample, predicted this direction before any training run (Table 2). It predicts which error rates training will raise, and in some rows accuracy. It also needs no feature named in advance. We selected two phrase graders from twelve style features fixed in advance, and a length-stratified rate does not see them. For both, the audit's false-positive prediction was confirmed on five seeds each (App. V.4, V.6). Post hoc, a length-stratified error rate flags the main pair too. The training reward does not show these drifts. Under the short grader it rises from 0.64 to 0.98, while the share of correct completions falls from 0.48 to 0.05. Under the phrase graders it reaches 0.99, while that share goes from 0.47 to 0.48 and from 0.46 to 0.42, against 0.47 to 0.61 under gold (Figure 9).

How does this weigh against the null learning result (App. M, formerly Appendix O)? The two results concern different parts of the perturbation. The toy varied the noise covariance inside the variance part. In its largest cell the reward gap was 0.0034, against 0.0386 extrapolated post hoc. A post hoc reading is that the shape of that covariance may matter little for learning, which fits MSW as the per-group summary of the variance part. The twin changes the systematic part instead. It keeps every per-group statistic of labels and verdicts that ignores the order of the completions. Under the linear operator it removes most of the feature-tied part of the systematic bias, 0.88 for false positives at the pooled class share, an indicative value (App. Q.2). Between the twin and its grader, accuracy moves from 0.495 to 0.042. Read together, the two results match our recommendation: MSW for the variance part, and the direction of the systematic part.

Q2. Does the batch-bias finding survive both checks? We ran both, and it remains. We recomputed π64\pi_{64} for rule_strict from exact gradients of all 1.54 billion parameters, without a sketch. It is 0.257 [0.152, 0.365] by the prompt bootstrap, against the published 0.261 [0.154, 0.369]. Eight independent sketches of the same vectors give 0.256 to 0.260. Their standard deviation is 0.0013, against 0.054 from prompt resampling, so the interval barely moves (App. O, App. P). The cell stays post hoc and one raw cell. At the trained batch of 16 prompts, π16\pi_{16} by eq. (4) is 0.009 [0.000, 0.036] in the training-time audit (§5).

Q3. Do the additional components stay informative under the actual update? Under the trained update, what we find informative is the direction of the systematic part. We compute the drift directions under z-score and token averaging, and training tests them under the full update with Adam and clipping (Table 2). The sizes of the channel shares are measured under the trained update and are descriptive. Post hoc, MSW carries 0.82 to 0.99 of the per-group perturbation under the z-score and 0.33 to 0.96 with token averaging (§5, App. O). By Proposition 4, the bias and variance split, the batch identity and the unbiasedness of T^\hat T extend to the z-score with token averaging, and AdamW with clipping lies outside its scope. The closed-form channels and the first-order dynamics are derived for the linear operator (§3, §4). On E18's three verifiers, MSW ranks them as T^\hat T does. On the sample of §7, MSW and the gradient-level quantities order 8 pairs of the 19 verifiers differently with interval support, each with a length grader (App. T.3). These quantities are T^\hat T and the second moments of 16- and 64-prompt batches.

A cheaper baseline that reads length. Where length is suspected in advance, a length-stratified error rate predicts the drift audit's directions on length-tied graders and, through length, the 7B judge's false-positive direction against both twins (App. V). For the judge, the effect goes beyond length. We fixed a rule for the share that length explains after the judge's arm had finished and before either twin was trained. By that rule, length carries 0.009 and 0.007 of the judge's 0.063, and the rest stays above 0 at seed level (App. V.3).

Thank you for the review.

Reviewer_uFfaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks How and which properties of the verifier error effect the gradient update? It introduces a framework for characterizing verifier-induced gradient perturbations through their second moment by decomposing the perturbation into three channels. The authors also examine why intraclass correlation (ICC) and effective sample size (ESS) may not adequately characterize the optimization consequences of verifier errors.

Assume that

r~i=ri+ϵi,\tilde r_i = r_i + \epsilon_i,

where ϵi\epsilon_i denotes the verifier error for sample ii.

Let si=∇θlog⁡πθ(yi∣x)s_i = \nabla_\theta \log \pi_\theta(y_i \mid x) denote the score vector for response yiy_i given prompt xx. Define the group-centering matrix as

P=I−1G11⊤.P = I - \frac{1}{G}\mathbf{1}\mathbf{1}^{\top}.

Under the paper's linear, sequence-level update formulation, the reference-reward gradient contribution and the verifier-based gradient contribution are

\qquad \tilde g = \frac{1}{G}SP\tilde r,$$ where $S$ collects the response score vectors and $r$ and $\tilde r$ are the corresponding reward vectors. The resulting gradient perturbation is therefore $$\Delta g = \tilde g-g = \frac{1}{G}SP\epsilon.$$ They say that the optimizer responds to the **within-group variances in verifier error, weighted by the policy's score vectors**. The authors decompose the second moment of $\Delta g$ into three channels based on this interaction: 1. **Within-group error variation:** how verifier errors vary among responses in the same group, and how this variation interacts with the covariance structure of their score vectors. $$T_{\mathrm{MSW}}=(G-1)\operatorname{Var}(\epsilon)\operatorname{tr}(\operatorname{Cov}(s)).$$ 2. **Error–score-norm coupling:** how the magnitude of centered verifier errors is associated with the squared norms of the score vectors. Even when error magnitude alone appears uninformative, its association with the scale of the policy's gradients can affect the resulting perturbation. $$T_{\mathrm{coupling}}=\frac{G-1}{G}\operatorname{Cov}((\epsilon-\bar{\epsilon})^2,\\|\bar{s}\\|^2).$$ 3. **Systematic error–score alignment:** how verifier errors covary with policy score vectors, producing a systematic component of the gradient perturbation rather than merely increasing its variability. $$T_{\mathrm{hacking}}=\frac{(G-1)(G^2-2G+2)}{G}\\|\operatorname{Cov}(\epsilon,s)\\|^2.$$ These channels provide a more optimization-oriented characterization of reward hacking. ICC and ESS summarize aspects of within-group dependence, but they do not, by themselves, capture every interaction between verifier errors and the policy's score vectors. Two verifiers with similar ICC and ESS values can induce different gradient perturbations if their errors differ in how they align with those vectors. The paper provides insights as to *which statistical properties of its errors reach the gradient, and through what mechanisms?* However, this decomposition does not, establish whether and how those perturbations effect the downstream training outcomes.
Critical strengths

The paper provides interesting insights about the decomposition of the second moment of verifier errors and its effect on policy gradient.

Critical weaknesses
  1. The paper assumes that ϵi\epsilon_i, which is the verifier error is known, which in general cases is not true. Human engineers try to ensure that the environment or verifier is reliable and if the errors are already known they would fix it. Hence, though this paper provides an interesting analysis, it might not be useful in practice.

  2. It is not well studied if the three terms in the decomposition are mutually exclusive and exhaustive.

  3. The paper does not show how this decomposition or verifier error analysis would effect the training performance.

Questions for authors

My main concern / question is #1 in the weakness.

2 comments
Reviewer_uFfaOpenReview ↗

Reminder for Rebuttal

Dear Authors,

Today is the last day of rebuttal. I am happy to engage in discussion if you answer my questions. Just a friendly reminder to put in the rebuttal :)

W1, your main concern. Does the audit assume that the verifier's errors are known? Only on a labelled audit sample. The audit needs gold labels on a reserved sample of prompts, as any verifier audit does, and not the verifier's errors on the training completions. Many of those errors appear once the policy learns to produce them, and the training reward hides them. For example, under the short grader the reward rises from 0.64 to 0.98, while the share of correct completions falls from 0.48 to 0.05 (Figure 9).

We flagged this grader before training with the drift audit, a gradient-level check run on 600 labelled prompts. Post hoc, we resampled subsets of this one sample. For 0.90 of the subsets to predict the length graders' directions, 50 to 100 prompts were enough. The judge's directions took 300 to 400 or more, and the short grader's accuracy more than 400 (App. Q.6).

The audit tells which mistakes training will amplify. Before training, it predicts the sign of the drift of each error rate. A grader whose mistakes are tied to a feature the policy can learn, such as brevity or a phrase, may look harmless in its error rates and still steer training. Under two phrase graders the reward reaches 0.99. Yet the share of correct completions goes from 0.47 to 0.48 under the first and from 0.46 to 0.42 under the second, against 0.47 to 0.61 under gold (App. V.4). For both kinds of grader, the audit had predicted a higher false-positive rate than under the grader's permuted twin, and training confirmed it. The twin is a control that keeps the grader's verdicts but reshuffles them at random among completions with the same gold label in a group.

We also ran a competent judge in place of gold. MSW carries most of the perturbation of a Qwen2.5-7B-Instruct judge that reads the reference, 0.855 [0.765, 0.976] under the linear operator (App. T). For two permuted twins this judge moved T^\hat T by more than 20% (App. T.3), so the audit keeps gold labels on the small reserved sample above.

W2. Are the three terms mutually exclusive and exhaustive? They are exhaustive. By Theorem 2, the three terms sum to G2E∥Δg∥2G^2\mathbb E\|\Delta g\|^2 as an identity under the linear update. The split of the second moment into squared bias and variance is orthogonal. The MSW term and the coupling are distinct functionals, and as error types the terms overlap: one format checker feeds all three channels, and the coupling can be negative (§3, App. O).

W3. Does the paper show how verifier errors affect training performance? Yes. The revision adds a training study: we train Qwen2.5-0.5B-Instruct by GRPO. A grader and its twin with the same error rate and confusion matrix in every group end at 0.042 and 0.495 accuracy, against 0.535 under gold. This is the direction predicted before any training run, and on SVAMP the pair separates again, at 0.306 and 0.604 (§7, Table 2, App. R.2). Our summary gives the full predictive record.

Thank you for putting the practical question so clearly.

Reviewer_MLQDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that the ICC and ESS, which describe the precision of a group mean, are the wrong statistics for auditing verifiers in group-relative RL, since the update discards the mean and consumes score-weighted centered errors. Casting the verifier-induced perturbation as a U-statistic, they use Hoeffding's formula to split its second moment into three channels: an MSW term, an error–score-norm coupling term, and a squared reward-hacking gradient whose bias does not average out over a batch. On Qwen2.5 policies (GSM8K, G=8G=8), the MSW channel carries 0.90–1.00 of the per-group perturbation, and with LLM judges, ESS ranks three verifiers in reverse order while MSW ranks them correctly.

Critical strengths
  • A sound reframing. The observation that the update "sees" only centered, score-weighted errors is simple and correct, and the batch-level treatment of the hacking bias (Eq. 4) is its most insightful consequence.
  • Exemplary transparency. Every empirical claim is labelled pre-specified, exploratory or post hoc, failed predictions are reported prominently, and the classical ingredients are clearly credited.
Critical weaknesses
  • Limited novelty. Theorem 2 is Hoeffding's decomposition, Corollaries 6–7 mirror the Moulton factor, and T^\hat T is the CR0 sandwich meat, so the contribution is mostly an interpretation.
  • The evidence supports the status quo. The MSW channel dominates everywhere, so variance-based audits suffice. The other channels matter in one post hoc cell only, extrapolated beyond the trained batch, and the ESS reversal follows from ESS being scale-free.
  • Weak predictive record and no training link. Most pre-specified predictions failed, including the z-score theory. Moreover, the training-outcome experiment (E19) was not run, and the identities hold only for the linear Dr. GRPO update.
  • Narrow setting and presentation. The setting is two small Qwen2.5 models, and the two judges disagree with gold more often than a constant "incorrect" classifier. Finally, Appendices P.1–P.11 are only pointers to the supplementary material.
Questions for authors
  1. Which concrete verifier decision would the three-channel audit change compared with an MSW audit alone?
  2. Could a small E19 relate the hacking share, at the trained batch size and under the full GRPO operator, to final accuracy?
  3. Could you propagate the variance of the shared count sketch, given the negative estimates of a nonnegative quantity?
  4. Do the conclusions hold with a competent, reference-based judge?
2 comments

Q1. Which verifier decision would the three-channel audit change over an MSW audit alone? The decision is whether to train with a verifier whose mistakes are tied to a feature the policy can learn. An MSW audit rates the grader short_lenient and its permuted twin alike, yet GRPO ends them at 0.042 and 0.495 accuracy (Figure 1). The twin reshuffles the grader's verdicts at random among completions with the same gold label in a group. Under the grader, the policy moves to short wrong answers that the grader accepts.

The decomposition shows where this decision sits. The third channel measures the size of the systematic part, the part of the perturbation that does not average out in a batch. The decision rests on the direction of that part, which we read before training as the initial drift of error rates and length, using the drift audit, a gradient-level check on a gold-labelled sample (§4). For two phrase graders that a length-stratified error rate does not see, the drift audit's false-positive prediction is confirmed on five seeds each (our summary). For this length grader, a length-stratified error rate predicts the same directions, post hoc.

Do variance-based audits suffice? Per group and without token averaging they carry most of the perturbation, and per decision the twin pair shows what they miss. Post hoc, MSW carries 0.82 to 0.99 of the per-group perturbation under the z-score. Post hoc, with token averaging added, the operator we train with, it carries 0.33 to 0.96 (§5, App. O). The grader and its twin share every order-free per-group statistic of labels and verdicts, that is, every such statistic that ignores the order of the completions, and they still end far apart. By Corollary 8, the feature-tied part of the bias is invisible to every audit that reads only labels and verdicts at one policy. Such an audit identifies at most the accuracy and the law of the errors given the gold labels.

Is the predictive record weak? In training, the plans' pre-specified tests scored 66 predictions fixed before their runs. Of these, 49 came out as predicted, 42 of them also at seed level, and none was reversed at its planned level. Of the drift audit's own 60, 47 are confirmed (Table 11).

Error-rate directions partly follow from how a grader is built, so accuracy and gold reward are the sharper test: 13 of 21 came out as predicted (Table 11). The submission's pretrained-audit predictions and z-score simulations stay in the paper with their verdicts (App. N.5, App. L), and the training study is the test the revision adds.

Q2. Could a small E19 relate the hacking share to final accuracy? We ran it. The experiment you call E19 is E21 in App. Q. It uses the trained batch of 16 prompts and the full GRPO update: z-scored advantages, token-mean loss, AdamW and clipping. Its planned rank correlation of the hacking share with final accuracy needs more than two base verifiers with trained twin pairs, so it is left for a larger study (App. Q.5). In this run, the drift audit's predicted direction foretold the collapse.

Post hoc, we read σ16\sigma_{16} at step 0, the share of a 16-prompt batch's second moment that is shared across prompts (Table 14). It was 0.037 for the grader that collapsed and 0.048 for rule_strict, while the drift audit's predicted direction separated the two. The shares are descriptive, and here the direction, not the share, foretold the collapse.

Is the contribution mostly an interpretation? New here are the permuted twin with its blind-spot result, the drift audit and their test in training. Theorem 2 is an instance of Hoeffding's formula, and we credit it, the Moulton factor, the Kish design effect and the CR0 sandwich (§8). The twin is a matched control. It keeps every order-free per-group statistic of labels and verdicts, and its drift differs from its grader's by a closed form (Proposition 7). Under the linear operator it also removes most of the bias tied to the learnable feature, 0.88 for false positives at the pooled class share, an indicative value (App. Q.2). As a result, the twin turns the blind spot of error-matrix audits into a controlled training experiment (§7). The batch-level reading of the hacking bias, eq. (4), which you named the paper's most insightful consequence, stays from the submission.

Do the other channels matter only in one post hoc cell beyond the trained batch? That cell is post hoc, and it holds under exact gradients. Recomputed for rule_strict with exact gradients and no sketch, π64\pi_{64} is 0.257 [0.152, 0.365] against the published 0.261, and eight fresh sketches give 0.256 to 0.260, so the interval barely moves (App. P). We obtained the 64-prompt value post hoc, from one raw cell beyond the trained batch, and we therefore call the batch-level shares descriptive. At the trained batch of 16 prompts, the submission's training-time audit gives π16\pi_{16} by eq. (4) as 0.009 [0.000, 0.036] (App. O). The step-0 audit of the training experiment reports a different quantity, σ16\sigma_{16}, the share of a 16-prompt batch's second moment that is shared across prompts (Table 14). For rule_strict it is 0.048 [0.032, 0.067]. At the trained batch, the direction of the systematic part is what training tests (Table 2).

Does the ESS reversal follow from scale-freeness? Yes, and we state it as a classical corollary (Corollary 12). It matters in practice because audits in use report ESS (Xin, 2026).

Which identities extend beyond the linear update? By Proposition 4, the bias and variance split, the batch identity and the unbiasedness of T^\hat T hold for every advantage map and score map that depends on the group alone. This includes the z-score and token averaging. The closed-form channels, the bias as the gradient of the expected error, the twin's drift formula and the first-order dynamics are derived for the linear update (§3, §4). We therefore use the dynamics for one purpose, the signs of the initial drifts, and training tests these signs under the full update (Table 2).

Q3. Could you propagate the variance of the count sketch? We did, and we also recomputed without a sketch, both post hoc. For rule_strict in the pretrained audit, the sketch adds a standard deviation of 9.4×10−59.4\times10^{-5} to S/Θˉ\mathcal S/\bar\Theta, against 1.5×10−31.5\times10^{-3} from resampling prompts (App. O). With exact gradients, π64\pi_{64} is 0.257 against the published 0.261. Eight fresh sketches give 0.256 to 0.260, and none of their 16 estimates of S\mathcal S is negative (App. P). The three negative estimates all came from the training-time audit at its step 0, where about 6.1 would be expected if every true share were 0 (App. O, App. N.9). The revision drops the sentence that attributed them to the sketch.

Q4. Do the conclusions hold with a competent, reference-based judge? MSW carries most of a competent judge's own perturbation. A Qwen2.5-7B-Instruct judge given the reference agrees with gold on 0.990 of the training audit sample, and its MSW share is 0.855 [0.765, 0.976] under the linear operator (App. T). The gradient-level quantities, T^\hat T and the second moments of 16- and 64-prompt batches, order 8 pairs of the 19 verifiers differently from MSW with interval support, each pair with a length grader (App. T.3). For the length graders' directions, post hoc, subsets of 50 to 100 of the 600 gold-labelled prompts gave the full-sample direction in 0.90 of resamples (App. Q.6). For two permuted twins, which reshuffle a grader's verdicts within each gold class of a group, this judge moved T^\hat T by more than 20%, so we keep gold labels for the audit.

What is the setting, and are the appendices in the PDF? The appendices now stand in the PDF. Appendices P.1 to P.11 of the submission are App. N.1 to App. N.10, with P.1 and P.2 merged into App. N.1. Each gives its design, its results and the verdict of every prediction it fixed, and the supplement holds every experiment's extended record. Training covers one 0.5B policy on two data sets of one kind in 114 runs, and SmolLM2-1.7B-Instruct is audited without training (App. N.10). In the multi-seed audit, the two judges disagree with gold more often than a constant classifier, 0.64 and 0.55 against 0.33 (Figure 4). We therefore use them as sources of structured error, and App. T adds the competent judge above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.