- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates why majority-vote self-generated rewards can track verifiable RL despite noisy pseudo-labels. Its central claim is that batch-level gradient structure matters beyond prompt-level label accuracy: incoherent errors can cancel across prompts, while coherent errors create systematic misdirection. Gradient measurements across five models indicate near-alignment between the expected majority-vote and gold updates after correcting for sampling noise. Controlled reward-corruption experiments support the distinction between error rate and error structure. The analysis is local, however, and the training experiments do not establish equivalent trajectories or final performance.
Critical strengths
-
Direct comparison of reward-induced updates. Reweighting the same per-response gradient sketches allows majority-vote and gold rewards to be compared on identical samples. Together with measurements across five models from two families, this provides useful evidence beyond correlations between reward accuracy and downstream performance.
-
Controlled evidence for the importance of error structure. The matched-error-rate intervention is particularly informative: on Qwen3-1.7B, independent flips and shared-rule flips both corrupt 20.4% of response rewards, but their corrected gradient cosines are approximately 1.01 and 0.29, respectively (Table 3). This supports the claim that reward error rate alone does not determine update quality.
Critical weaknesses
-
The proposed explanation remains conditional, with limited validation of when its key condition holds. Proposition 2 characterizes cancellation when error coherence is sufficiently small; it does not establish why errors generated by majority vote should satisfy this condition or when they cease to do so. The gradient measurements on DeepScaleR and the artificial corruption controls support the mechanism in the tested settings, but predictions concerning natural shared misconceptions or repeated prompts remain untested. Consequently, the evidence is stronger as a description of observed alignment than as a criterion for when majority-vote rewards can replace verification. This limits the explanatory contribution.
-
The connection between estimated gradient alignment and actual training remains insufficiently established. The primary diagnostic concerns expected raw-gradient directions, whereas training uses finite batches and Adam. The appendix usefully checks advantage normalization and inclusion of gold-silent groups, but the latter increases finite-batch misdirection from 0.43 to 0.75 despite a similar corrected cosine (Appendix A). Only short runs on Qwen3-4B connect these measurements to training outcomes; on full MATH-500 at step 20, maj@64 is 77.2% for MV versus 84.5% for gold (Section 5). The authors acknowledge the local scope, but the diagnostic does not yet adequately explain this separation. Stronger validation against actual optimizer updates and training outcomes would materially strengthen the contribution.
Questions for authors
-
Clarification: What conclusions about practical substitutability are justified by a corrected cosine near one, given the larger finite-batch misdirection when gold-silent groups are included? Please distinguish evidence for expected-direction alignment from evidence for comparable realized updates.
-
Additional evidence relevant to this assessment: Does near-alignment persist for Adam-preconditioned updates at the available checkpoints, including all prompt groups and the training advantage normalization? Can these measurements help account for the observed MV–gold performance gap?
-
Additional evidence relevant to this assessment: Can the proposed coherence criterion predict changes in MV performance when prompt diversity or repetition is varied, beyond the artificial reward-corruption setting? Such evidence would help establish when the explanation applies.