acceptodds
Under review as a conference paper at ICLR 2027

WRONG VOTES, RIGHT GRADIENTS: WHY MAJORITY-VOTE REWARDS TRACK VERIFIABLE RL

Abstract

Reinforcement learning with self-generated labels, such as majority vote over a model's own samples, often recovers most of the gain of training with verifiable rewards. This is puzzling from the usual prompt-level view, in which majority vote only returns a model's modal answer and cannot add information about correctness. We argue that the relevant unit is the batch-level update. Because parameters are shared, a prompt's own gradient accounts for less than 1% of the change in that prompt's output at practical batch sizes, so a wrong label matters only to the extent that label errors share a direction across prompts. We decompose the majority-vote update into shrinkage and misdirection relative to the gold update, and show that misdirection vanishes with batch size whenever errors are incoherent across prompts, however often the labels are wrong. Measuring per-response gradients on five models, we find that majority-vote errors are as incoherent as a random-reward null, that the majority-vote update agrees with the gold update as closely as two independent gold updates agree with each other, and that its noise-corrected cosine with the gold update is 1 within error. Randomly flipping up to 30% of gold labels leaves the update direction intact, while the same fraction flipped by a shared rule destroys it, and rewards that favor a kind of response on every prompt, such as a learned reward model, fall measurably short. The equivalence holds at every checkpoint we examine but is local: gold-reward training still ends ahead on held-out majority accuracy. Error coherence, not error rate, determines when self-generated labels can stand in for verification.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-play, majority voting, LLM self-improvement, reinforcement learning, reinforcement learning with verifiable rewards (RLVR)

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_STZfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why majority-vote self-generated rewards can track verifiable RL despite noisy pseudo-labels. Its central claim is that batch-level gradient structure matters beyond prompt-level label accuracy: incoherent errors can cancel across prompts, while coherent errors create systematic misdirection. Gradient measurements across five models indicate near-alignment between the expected majority-vote and gold updates after correcting for sampling noise. Controlled reward-corruption experiments support the distinction between error rate and error structure. The analysis is local, however, and the training experiments do not establish equivalent trajectories or final performance.

Critical strengths
  1. Direct comparison of reward-induced updates. Reweighting the same per-response gradient sketches allows majority-vote and gold rewards to be compared on identical samples. Together with measurements across five models from two families, this provides useful evidence beyond correlations between reward accuracy and downstream performance.

  2. Controlled evidence for the importance of error structure. The matched-error-rate intervention is particularly informative: on Qwen3-1.7B, independent flips and shared-rule flips both corrupt 20.4% of response rewards, but their corrected gradient cosines are approximately 1.01 and 0.29, respectively (Table 3). This supports the claim that reward error rate alone does not determine update quality.

Critical weaknesses
  1. The proposed explanation remains conditional, with limited validation of when its key condition holds. Proposition 2 characterizes cancellation when error coherence is sufficiently small; it does not establish why errors generated by majority vote should satisfy this condition or when they cease to do so. The gradient measurements on DeepScaleR and the artificial corruption controls support the mechanism in the tested settings, but predictions concerning natural shared misconceptions or repeated prompts remain untested. Consequently, the evidence is stronger as a description of observed alignment than as a criterion for when majority-vote rewards can replace verification. This limits the explanatory contribution.

  2. The connection between estimated gradient alignment and actual training remains insufficiently established. The primary diagnostic concerns expected raw-gradient directions, whereas training uses finite batches and Adam. The appendix usefully checks advantage normalization and inclusion of gold-silent groups, but the latter increases finite-batch misdirection from 0.43 to 0.75 despite a similar corrected cosine (Appendix A). Only short runs on Qwen3-4B connect these measurements to training outcomes; on full MATH-500 at step 20, maj@64 is 77.2% for MV versus 84.5% for gold (Section 5). The authors acknowledge the local scope, but the diagnostic does not yet adequately explain this separation. Stronger validation against actual optimizer updates and training outcomes would materially strengthen the contribution.

Questions for authors
  1. Clarification: What conclusions about practical substitutability are justified by a corrected cosine near one, given the larger finite-batch misdirection when gold-silent groups are included? Please distinguish evidence for expected-direction alignment from evidence for comparable realized updates.

  2. Additional evidence relevant to this assessment: Does near-alignment persist for Adam-preconditioned updates at the available checkpoints, including all prompt groups and the training advantage normalization? Can these measurements help account for the observed MV–gold performance gap?

  3. Additional evidence relevant to this assessment: Can the proposed coherence criterion predict changes in MV performance when prompt diversity or repetition is varied, beyond the artificial reward-corruption setting? Such evidence would help establish when the explanation applies.

Reviewer_1DEzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explains majority-vote RL through shared batch gradients. It separates reward errors into shrinkage and misdirection, measures gradient alignment on five models, and contrasts independent label flips with structured errors. Short training runs suggest that majority-vote updates can resemble gold updates locally while producing weaker held-out majority accuracy.

Critical strengths
  1. A well-controlled mechanistic comparison. Scoring identical responses with different rewards reduces sampling confounds, and random versus length-structured flips directly separate error frequency from error structure.

  2. Useful evidence across models and checkpoints. Multiple model families, finite-batch growth measurements, and matched checkpoint comparisons support a substantive empirical contribution. The paper also honestly reports the gold–MV training gap.

Critical weaknesses
  1. The main estimator does not generally identify the claimed expected-gradient alignment. Section 2 forms (X/\sqrt{YZ}) from averaged split-half cosines. However, averaging normalized cosine similarities does not generally recover the cosine between expected raw gradients without additional concentration assumptions. These assumptions are not sufficiently justified or validated. Since the near-perfect expected-gradient alignment is a central empirical claim, this is an important methodological concern. A direct estimate using cross-half raw inner products would help resolve it.

  2. The connection between local gradient alignment and training outcomes remains unclear. Despite the reported near-perfect expected-gradient alignment at several checkpoints, gold training substantially outperforms majority-vote training on held-out majority accuracy (0.845 versus 0.772 on MATH-500 at step 20). The authors acknowledge that local alignment does not guarantee similar training trajectories, and Section 4.4 finds some accumulation of errors across steps. However, the analysis does not quantitatively explain how accumulated errors, gradient magnitudes, or adaptive optimization lead to the observed performance gap. This limits the practical explanatory power of the proposed mechanism.

  3. The coherence analysis does not directly measure the quantity required by the theory. Proposition 2 depends on norm-weighted alignment of error gradients projected orthogonally to the gold batch gradient, whereas Tables 2–3 primarily report unweighted pairwise cosines of raw errors. These quantities are not equivalent. The finite-batch gradient-growth measurements in Table 11 provide supporting evidence, but a more direct comparison with the theoretical quantity would strengthen the quantitative explanation of error cancellation.

  4. The main prompt filter limits the interpretation of the finite-batch results. The principal analysis excludes groups with homogeneous gold rewards, although majority-vote rewards can still produce updates on these groups. The authors provide a useful all-prompt robustness check showing that corrected alignment remains high, but relative misdirection increases from 0.43 to 0.75. Thus, the filtering does not appear to overturn the directional conclusion, but its effect on the magnitude and interpretation of the updates deserves greater emphasis.

  5. Some checkpoint-level claims need qualification. The corrected cosine cannot be reliably estimated at step 10, so the claim of equivalence at every examined checkpoint is stronger than the evidence supports. In addition, the statement that the gold–MV performance gap is smaller than between-seed variation appears inconsistent with Table 8, where the maj@64 gap is 0.049 compared with within-reward spreads of 0.007 and 0.010. These issues do not invalidate the main findings but should be clarified.

Questions for authors
  1. Does the expected-gradient alignment remain near one when estimated using cross-half raw inner products from the existing gradient sketches, including all prompts? Alternatively, can the authors provide diagnostics validating the concentration assumptions behind the current estimator?

  2. Can the authors quantitatively explain why gold and majority-vote training produce substantially different final performance despite the reported local gradient alignment? In particular, what roles do accumulated errors across steps, gradient magnitudes, and adaptive optimization play?

  3. Can the authors directly estimate the norm-weighted projected error coherence required by Proposition 2 and compare it with the reported raw-error cosine statistics? How sensitive are the conclusions to including groups with homogeneous gold rewards?

Reviewer_koDJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks why reinforcement learning with self-generated labels (majority vote over a model's own samples) recovers most of the gain of verifiable-reward training, even though majority vote only returns the model's modal answer and adds no correctness information at the prompt level. It argues that the relevant unit is the batch-level update: because parameters are shared, a prompt's own gradient accounts for less than 1% of the change in that prompt's output at practical batch sizes, so a wrong label matters only insofar as label errors share a direction across prompts. The authors decompose the majority-vote (MV) update into a shrinkage component along the gold direction and a misdirection component, and prove that misdirection vanishes as the batch grows whenever errors are incoherent, regardless of the error rate. They test this account with per-response gradient sketches, which allow arbitrary rewards to be evaluated on identical samples, across five models at initialization and along two training runs of Qwen3-4B-Base. MV errors are found to be as incoherent as a random-reward null, MV and gold updates agree about as well as two independent gold updates do (attenuation-corrected cosine near 1), and the update direction survives randomly flipping up to 30% of gold labels but not shared-rule flips. The equivalence is local: gold training still ends ahead on held-out majority accuracy.

Critical strengths
  1. The prompt-level account of self-generated labels is replaced by a batch-level one, and error coherence rather than error rate is identified as the governing condition. The decomposition of the MV update into shrinkage and misdirection, together with the result that misdirection vanishes for incoherent errors once the batch exceeds a critical size, converts a loose empirical observation into a testable criterion for when self-generated labels can stand in for verification. This matters for my assessment because it reframes the question the paper asks, and the criterion is stated in terms of a quantity (coherence of errors across prompts) that is measurable independently of any particular training run.
  2. The empirical design is unusually direct and well controlled. Per-response gradient sketches let every reward be evaluated on identical samples; split-half cosines are corrected for attenuation and reported with bootstrap intervals; and the flip experiments hold the error rate fixed while varying only the structure of the errors (independent flips versus a shared rule), which isolates the mechanism the theory predicts. The measurement spans five models across two families, base and instruction-tuned, and is repeated along training, with the anomalous step-10 gold-gold ceiling reported candidly rather than hidden.
Critical weaknesses
  1. The central equivalence claim rests on an estimator whose target quantity is not established. The split-half disattenuated cosine X/YZX/\sqrt{YZ} is computed from averaged split-half cosines and interpreted as cos⁡(EGmv,EGgold)\cos(\mathbb{E}G_{\rm mv}, \mathbb{E}G_{\rm gold}) (p. 3, L125–140; Appendix B, p. 18, L959–963), but for independent half-batch updates this ratio identifies the cosine of the expected normalized updates, and the two coincide only under additional norm-concentration conditions that are neither stated nor checked. A counterexample satisfying the paper's own conditions (independence, Z≥0.3Z \ge 0.3) yields a corrected value of 1 while the cosine of the expected updates is about 0.714. This is consequential because every “corrected cosine near 1” result, which constitutes the paper’s main evidence that MV matches gold supervision, relies on this estimator, yet the paper reports neither an unnormalized inner-product estimate nor a large-batch cross-check.
  2. The scope of the claim outruns the evidence, and the reported numbers are not fully reconciled. The five-model evidence is measured only at initialization, the training evidence comes from a single model (Qwen3-4B-Base, two MV runs), and the abstract's "every checkpoint" is contradicted by Section 5's own report that the corrected cosine is unreportable at step 10 on both seeds. In addition, p. 8, L431 states gold's step-10 advantage is weaker than the seed spread while Table 8's maj@64 gap (0.049) exceeds both within-reward spreads (0.007 and 0.010), and the counts implied by Table 14 (68/134 at base, 78/134 at step 20) disagree with the tabulated 0.503 and 0.585.
Questions for authors
  1. Which population quantity does the disattenuated cosine identify? Section 2 (p. 3, L125–140) and Appendix B (p. 18, L959–963) interpret X/YZX/\sqrt{YZ}, computed from averaged split-half cosines, as cos⁡(EGmv,EGgold)\cos(\mathbb{E}G_{\rm mv}, \mathbb{E}G_{\rm gold}). For independent half-batch updates with non-zero expectation, however, writing Uk=Gk/∥Gk∥U_k = G_k/\|G_k\|, that ratio of averaged cosines generally corresponds to cos⁡(EUmv,EUgold)\cos(\mathbb{E}U_{\rm mv}, \mathbb{E}U_{\rm gold}). Could you state the additional norm-concentration conditions under which the two coincide, report the empirical relation between norms and directions, and cross-check against an unnormalized inner-product estimate built from independent prompts or against an independent large-batch result? A counterexample showing that independence and Z≥0.3Z \ge 0.3 are not by themselves sufficient: if each half-batch independently produces (Ggold,Gmv)=((1,0),(100,0))(G_{\rm gold}, G_{\rm mv}) = ((1,0),(100,0)) or ((0,1),(0,1))((0,1),(0,1)) with probability 1/2 each, then X=Y=Z=0.5X = Y = Z = 0.5 and the corrected value is 1, whereas the cosine of the expected vectors is about 0.714. This does not show that your data carry a bias of that size; it does show that the stated conditions do not pin down the quantity being reported.

  2. How far does the equivalence claim extend? The five-model evidence is measured at initialization, whereas the evidence from training comes from a single model (Qwen3-4B-Base, two runs); the abstract also states the equivalence "at every checkpoint", while Section 5 itself reports that the corrected cosine cannot be reported reliably at step 10 for either seed (p. 8, L397–402). What is the intended scope of this claim: across models, across checkpoints, or only across the reported runs? Does the evidence from five models support that scope, or does it establish only the initialization result?

  3. Two statements appear inconsistent with the reported tables. On p. 8, L431 gold's advantage is called weaker than the spread between seeds, but the maj@64 gap of 0.049 in Table 8 exceeds both within-reward spreads (0.007 and 0.010). The counts implied by Table 14 (68/134 at base, 78/134 at step 20) also differ slightly from the tabulated 0.503 and 0.585. Could you correct the first and reconcile the second? Relatedly, in Table 12 the 95% intervals of the raw split-half difference (MV vs. gold minus gold vs. gold) on Qwen3-1.7B-Base and Llama-3.1-8B-Instruct both exclude zero, which should be kept distinct from the corrected claim about the expected direction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.