- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how verifier errors affect a mean-centered, group-relative policy-gradient update. It applies a classical variance decomposition to separate the expected squared gradient perturbation into within-group error variance, coupling between errors and score-vector magnitudes, and a component associated with systematic gradient bias. It also examines how bias behaves under batch averaging. Experiments on Qwen2.5 policies find that the MSW component usually dominates per-group perturbation, while one post hoc condition suggests a substantial systematic component at larger batch sizes. A three-seed audit shows that ESS can rank verifiers opposite to their measured perturbations, although the MSW baseline ranks them correctly. The paper does not establish a connection between the additional audit components and improved training outcomes
Critical strengths
The finding that MSW accounts for most per-group perturbation provides evidence for a simpler auditing default. The authors also clearly report failed predictions and the null learning result, helping readers understand the limits of their conclusions.
Critical weaknesses
The authors explicitly acknowledge that the main decomposition, estimator, and several supporting observations are classical. The contribution therefore depends substantially on the usefulness of their audit interpretation and empirical findings. Yet MSW explains 90–100% of per-group perturbation in the main sequence-level audits, and the MSW baseline correctly ranks the verifiers in the prospective comparison where ESS fails (Sections 5–6). The comparison establishes a limitation of ESS but does not establish the value of the additional gradient-based measurements. Appendix O also does not validate the finer prediction through learning outcomes. A diagnostic paper need not improve a training benchmark, but I would expect stronger evidence that the additional components enable a useful diagnosis or decision that the simpler audit misses.
In addition, the gap between identifying mathematically distinct error components and demonstrating that distinguishing them provides enough additional scientific or practical value, is my concern. The paper makes progress on the first; its evidence for the second is presently too limited for me to recommend acceptance.
Questions for authors
-
Clarification: Which completed result most clearly supports an auditing conclusion or decision that an MSW-based audit would miss? How should this result be weighed against the null learning result in Appendix O?
-
Robustness evidence: Does the batch-bias finding remain when variability across independent sketches is included, or when checked using exact gradients on a feasible subset? How much would this change its confidence interval?
-
Applicability: Is there direct evidence that the additional components remain informative for the actual update used in training, including reward-standard-deviation normalization?
Q1. Which completed result supports a decision an MSW audit would miss? A grader and its permuted twin share the error rate, confusion matrix, MSW, ICC and Kish ESS in every group. Yet GRPO ends them at 0.042 and 0.495 accuracy (Figure 1). The twin reshuffles the grader's verdicts at random among completions with the same gold label in a group.
The drift audit, a gradient-level check on a gold-labelled sample, predicted this direction before any training run (Table 2). It predicts which error rates training will raise, and in some rows accuracy. It also needs no feature named in advance. We selected two phrase graders from twelve style features fixed in advance, and a length-stratified rate does not see them. For both, the audit's false-positive prediction was confirmed on five seeds each (App. V.4, V.6). Post hoc, a length-stratified error rate flags the main pair too. The training reward does not show these drifts. Under the short grader it rises from 0.64 to 0.98, while the share of correct completions falls from 0.48 to 0.05. Under the phrase graders it reaches 0.99, while that share goes from 0.47 to 0.48 and from 0.46 to 0.42, against 0.47 to 0.61 under gold (Figure 9).
How does this weigh against the null learning result (App. M, formerly Appendix O)? The two results concern different parts of the perturbation. The toy varied the noise covariance inside the variance part. In its largest cell the reward gap was 0.0034, against 0.0386 extrapolated post hoc. A post hoc reading is that the shape of that covariance may matter little for learning, which fits MSW as the per-group summary of the variance part. The twin changes the systematic part instead. It keeps every per-group statistic of labels and verdicts that ignores the order of the completions. Under the linear operator it removes most of the feature-tied part of the systematic bias, 0.88 for false positives at the pooled class share, an indicative value (App. Q.2). Between the twin and its grader, accuracy moves from 0.495 to 0.042. Read together, the two results match our recommendation: MSW for the variance part, and the direction of the systematic part.
Q2. Does the batch-bias finding survive both checks? We ran both, and it remains. We recomputed π64 for rule_strict from exact gradients of all 1.54 billion parameters, without a sketch. It is 0.257 [0.152, 0.365] by the prompt bootstrap, against the published 0.261 [0.154, 0.369]. Eight independent sketches of the same vectors give 0.256 to 0.260. Their standard deviation is 0.0013, against 0.054 from prompt resampling, so the interval barely moves (App. O, App. P). The cell stays post hoc and one raw cell. At the trained batch of 16 prompts, π16 by eq. (4) is 0.009 [0.000, 0.036] in the training-time audit (§5).
Q3. Do the additional components stay informative under the actual update? Under the trained update, what we find informative is the direction of the systematic part. We compute the drift directions under z-score and token averaging, and training tests them under the full update with Adam and clipping (Table 2). The sizes of the channel shares are measured under the trained update and are descriptive. Post hoc, MSW carries 0.82 to 0.99 of the per-group perturbation under the z-score and 0.33 to 0.96 with token averaging (§5, App. O). By Proposition 4, the bias and variance split, the batch identity and the unbiasedness of T^ extend to the z-score with token averaging, and AdamW with clipping lies outside its scope. The closed-form channels and the first-order dynamics are derived for the linear operator (§3, §4). On E18's three verifiers, MSW ranks them as T^ does. On the sample of §7, MSW and the gradient-level quantities order 8 pairs of the 19 verifiers differently with interval support, each with a length grader (App. T.3). These quantities are T^ and the second moments of 16- and 64-prompt batches.
A cheaper baseline that reads length. Where length is suspected in advance, a length-stratified error rate predicts the drift audit's directions on length-tied graders and, through length, the 7B judge's false-positive direction against both twins (App. V). For the judge, the effect goes beyond length. We fixed a rule for the share that length explains after the judge's arm had finished and before either twin was trained. By that rule, length carries 0.009 and 0.007 of the judge's 0.063, and the rest stays above 0 at seed level (App. V.3).
Thank you for the review.