acceptodds
Under review as a conference paper at ICLR 2027

An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used for post-training Large Language Models, but practical verifiers can make errors. We study how the rate and structure of reward noise affect RLVR in code generation, with a preliminary scientific-reasoning check. In multi-seed Qwen3 8B experiments on MBPP, mean validation reward over two fixed late evaluations is within 1 percentage point of the clean baseline at the tested resampled group-rollout noise rates through 20%, and within about 2 points at 30%. Confidence intervals allow larger losses; these point estimates do not establish a general tolerance threshold. We also examine full-program pass@k, four controlled noise structures, two model-based verifiers, and policy models from three families spanning 4B–9B parameters. We derive conditional advantage distributions for symmetric and asymmetric group noise, including retained format penalties, and show why clipping limits simple gradient-scaling arguments. The analysis identifies information preserved by whole-group corruption and limits on interpreting our asymmetric sweep as a precision–recall comparison. Overall, the results indicate that imperfect verification can support effective RLVR in the tested settings, while aggregate error rates alone do not characterize the learning signal.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-training, coding, MBPP, Qwen, Noisy rewards, unit tests rewards, LLM-as-a-Judge

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_as7TOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether noisy reward signals are sufficient for RL training. They define 4 noise models, where some portion of the (rollouts x unit_test_case) reward matrix for a given prompt is noised by flipping the binary test-case label with some probability p. They show that on MBPP, Qwen 3 8B and GLM reach about the same performance under (a) ground truth rewards and (b) group-flipped rewards where every test case for a particular group is flipped with probability p=0.1. Other noise models show slightly diminished performance, and a 4B model-based judge shows worst performance of all.

The core result with group-flipped rewards is further verified on Llama 3.1 8B.

Critical strengths

I appreciate the inclusion of several noise models, which highlights the distinction between verifier error rate and error structure -- even under the same error rate, different errors have differing amounts of prompt/rollout correlation. This distinction in itself is not super surprising, but I feel there could be something interesting here if pushed further. For example, perhaps systematically characterizing how much reward bias vs reward variance is tolerable.

Critical weaknesses

My primary concerns are that (1) the finding that noisy rewards have useful training signal is unsurprising to me, and (2) the reward noise considered does not reflect the sorts of noise one observes in practice, thus limiting the generality of the paper's claims.

First, the novelty of the finding. Neural reward models / LLM judges are obvious examples of imperfect verifiers, and it is well-documented that RL training with these imperfect signals can improve student performance - in some sense, this is already a clear existence proof for the paper's central claim. The authors also note a wide literature on noisy rewards for RL in their related work, differentiating their work as "focus[ing] on observed performance under specified noise structures." What new insights does studying this known phenomenon with these noise modes allow us to glean? I would be happy to discuss further.

Second, the practicality of the reward noise. In practice, errors from verifiers are often systematic: they persist across encounters of the same prompt and depend on the content of the rollout (e.g. a bad ground truth answer). All noise models considered in the paper are completely independent of both prompt and rollout content. The model-based verifiers, which do exhibit realistic error structure, show diminished performance relative to ground truth.

Presentation. As a further note, I found the writing to be confusing and hard to parse throughout. For example, when I first read the abstract, I saw mention of "two fixed late evaluations" (what evaluations?), "confidence intervals" that "allow larger losses" (what confidence intervals, and what does loss refer to here?), and "symmetric and asymmetric group noise," all terms which are not yet clearly defined. The prose also repeatedly hedges the main claims (e.g. L134, L164, L180). While it is good to be upfront about limitations, such presentation makes it hard for me to parse exactly what the authors are arguing for.

For these reasons, I tend towards rejection.

Questions for authors
  1. How does Llama 8B perform with the other noise models? As the authors noted, the group-flipped reward is the most benign noise. It would be great to be able to contextualize how much of the gain under the other noise modes on Qwen/GLM is due to spurious effects, and how much is from useful signal.

  2. How well does training work if we remove the format reward?

  3. How well does training work if the noise is not re-sampled for every encounter of a prompt? This lets us better test prompt-correlated errors.

2 comments

The finding that noisy rewards have useful training signal is unsurprising to me. ... What new insights does studying this known phenomenon with these noise modes allow us to glean?

We agree that learning from imperfect judges is known, and the paper does not present that as its finding. Because code has an executable reference, we can set the verifier error exactly and measure its cost: (i) a dose–response curve for whole-group flips: within 1 point of clean training through 20%, −2.0 points at 30%, −8.4 at 40% (Table 8); (ii) structure matters at a fixed 10% rate: group-level errors give 0.891–0.893 and sample-level errors 0.854–0.864 (Table 1); (iii) an analysis of why (Section 6.2, Appendix F): designs with the same expected advantage differ in its variance, which is a first step toward the bias-versus-variance characterisation the reviewer suggests; (iv) high recall is not enough: both LLM judges have recall above 0.90, yet one trains to 0.871 and the other to 0.704 (Figure 5). We will list these contributions in the introduction.

The reward noise considered does not reflect the sorts of noise one observes in practice. In practice, errors from verifiers are often systematic: they persist across encounters of the same prompt and depend on the content of the rollout. All noise models considered in the paper are completely independent of both prompt and rollout content.

For the four controlled modes, this is correct and intended, and the paper says so (L162–164). Content-dependent error is covered by the two LLM judges (Sections 3.2, 5.2, 6.3), which run at temperature 0, so their errors are tied to the generated code. The 30B judge ends 1.5 points below clean training (0.869 vs 0.884) and the 4B judge fails (0.704 peak); our response to Reviewer 4qgx gives a second 30B seed. For completeness, the group-rollout result covers p = 0.05–0.50 with 3–6 seeds per rate (Table 8).

Presentation: abstract uses undefined terms ("two fixed late evaluations", "confidence intervals that allow larger losses", "symmetric and asymmetric group noise"); prose repeatedly hedges the main claims (L134, L164, L180).

We agree that the abstract used terms before defining them. The revised abstract defines the evaluation metric, gives the interval as numbers and describes group noise in plain words, for example: "validation reward at the end of training (the mean of the last two evaluations shared by all runs, with no checkpoint selection) is within 1 percentage point of clean training for p up to 20%, about 2 points lower at 30%, and about 8 points lower at 40%." The sentences at L134, L164 and L180 are scope statements; we will state each claim once and move the caveats to the Limitations.

How does Llama 8B perform with the other noise models? ...

We have not run Llama 3.1 8B under the other three structures (stated at L314–315). The four structures are compared on two other families, Qwen3 8B and GLM4 9B, in Table 1. On spurious versus useful signal, the p=0.50 runs are the control: the test signal is uninformative there, and late reward is 0.355 ± 0.339, against 0.890 for clean training and 0.487 for the untrained model (Tables 1 and 8).

How well does training work if we remove the format reward?

We did not run this ablation; the confound is stated in Section 6.2, Appendix K and the Limitations. Two existing results bound the penalty's role (details in our response to Reviewer jSap, second weakness): at p=0.50, where only the format penalty is informative, late reward is 0.355 ± 0.339 against 0.890 for clean training (Table 8), and truncation explains at most about 12 of the roughly 40 points gained.

How well does training work if the noise is not re-sampled for every encounter of a prompt? This lets us better test prompt-correlated errors.

We do not have this experiment: all synthetic-noise runs resample the noise at each encounter (L162), and the Limitations say that robustness to persistent biases is not established. The closest existing evidence is the two temperature-0 LLM judges, whose errors are tied to content (see our response to the second weakness). We will name the missing condition in the Limitations.

Thanks for the reply. I am maintaining my current score, as I remain concerned about the novelty and practicality of this study.

  • Thanks for pointing to the p=0.5 runs -- Table 8 to my understanding is on top of the benign group-flipped reward. In absence of similar ablations on top of the other noise models or runs with Llama, we cannot be sure that the gains from other types of noise are not spurious.
  • Format rewards do not just help for truncation, and it is possible that the other 28 points gained also come from format improvements irrespective of the correctness reward.
  • "Content-dependent error is covered by the two LLM judges" - this is a known finding that imperfect LLM judges can improve performance.
Reviewer_4qgxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how noisy verifier rewards affect reinforcement learning (RLVR). It finds that models can tolerate moderate levels of reward noise with minimal performance degradation, and that the structure of the noise matters, with group-level corruption generally being less harmful than sample-level corruption.

Critical strengths

Provides a systematic comparison of different noise structures, supported by empirical experiments and theoretical analysis.

Critical weaknesses

The experiments primarily rely on a small coding dataset with limited random seeds, making it difficult to determine whether the findings generalize to other domains or realistic verifier errors.

Questions for authors
  1. Does the tolerance hold when verifier errors are persistent and correlated with response content, as with real LLM judges?
  2. How much of the measured improvement is genuine task learning versus learning output formatting?
3 comments

The experiments primarily rely on a small coding dataset with limited random seeds, making it difficult to determine whether the findings generalize to other domains or realistic verifier errors.

The main result is multi-seed: the Qwen3 8B sweep has 37 runs, with 3–6 seeds at each of eight noise rates and 4 clean baselines, and reports 95% intervals (Tables 5 and 8). The paper also covers GLM4 9B (3 seeds up to p=0.10), Llama 3.1 8B, Qwen3 4B, two LLM judges as realistic verifiers (Section 6.3) and a GPQA check (Table 2). We agree that MBPP is small and that GPQA is only a preliminary second domain; both are in the Limitations.

Does the tolerance hold when verifier errors are persistent and correlated with response content, as with real LLM judges?

This is the model-based verifier experiment (Sections 3.2, 5.2 and 6.3; Figure 5). Two LLM judges replace test execution and run at temperature 0 (Appendix D.1), so their errors are tied to the generated code and not redrawn at each encounter. The 30B judge ends at 0.869 against 0.884 for clean training; the 4B judge peaks at 0.704 and then declines. A second 30B seed in the supplementary records, which crashed after step 239, peaked at 0.856 and was at 0.832 at its last evaluation. So the stronger judge trains the task at a cost of roughly 1.5 to 5 points, and the weaker one does not.

How much of the measured improvement is genuine task learning versus learning output formatting?

The logged records bound the formatting share: truncated validation generations fall from 9.4% to about 0.1%, which explains at most about 12 of the roughly 40 points gained. At p=0.50, where only the format penalty is informative, late reward is 0.355 ± 0.339 against 0.890 for clean training (Table 8). Details are in our response to Reviewer jSap.

Thank you for your review, this answers most of my questions, but I will be keeping my score.

Thank you for engaging with our rebuttal and for confirming it resolved most of your questions. We would be grateful to know which concerns remain, so we can address them during the discussion period.

Reviewer_jSapOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how verifier errors affect RLVR, primarily through code generation experiments on MBPP. It compares four synthetic noise structures and two LLM judges. Moderate whole group corruption produces validation rewards close to clean training, while the learned judges yield substantially different outcomes. The analysis explains how group corruption preserves useful advantage information and why aggregate error rates alone cannot characterize the training signal.

Critical strengths
  1. The distinction between noise rate and noise structure is useful for understanding imperfect verification.

  2. The analysis carefully accounts for normalization, retained formatting penalties, and clipping.

  3. The expanded Qwen3 8B sweep includes multiple seeds and common checkpoint comparisons. The authors appropriately distinguish small observed differences from statistical equivalence.

Critical weaknesses
  1. The connection between synthetic corruption and actual judge failures remains unexplained.

Section 6.2 explains why whole group corruption retains useful advantage information. However, Figure 5 shows much worse performance with the 4B judge, and the paper does not measure how that judge changes the comparisons among responses within a group.

For example, does it frequently rank incorrect programs above correct ones, erase useful reward differences, or repeatedly reward particular mistakes? Precision and recall on each judge’s own policy outputs do not answer these questions.

  1. The reward mixes noisy correctness feedback with clean formatting feedback.

Appendix K reports early outputs that fail to finish within the token budget. Without separating formatting success from correctness among completed outputs, the experiments cannot determine how much improvement comes from each signal.

Questions for authors
  1. Can you compare the clean and judge derived advantages on shared response groups to explain why the 4B judge produces substantially worse training outcomes? In particular, how often does each judge reverse or erase useful reward comparisons?

  2. How much of the observed improvement comes from the unchanged formatting penalty? Can you report formatting success, truncation rates, and correctness conditional on producing a complete answer?

1 comment

The connection between synthetic corruption and actual judge failures remains unexplained. Section 6.2 explains why whole group corruption retains useful advantage information. However, Figure 5 shows much worse performance with the 4B judge, and the paper does not measure how that judge changes the comparisons among responses within a group. For example, does it frequently rank incorrect programs above correct ones, erase useful reward differences, or repeatedly reward particular mistakes?

The paper does not measure within-group comparisons under the judges, and per-response verdicts were not stored. The logged metrics answer some of the question. At step 0, both judges score the same untrained policy (Figures 5 and 6): the 4B judge has accuracy 0.58 and recall 0.99, the 30B judge 0.72 and 0.92. With recall that high, almost all of the 4B judge's errors are failing tests marked as passing, so it mostly erases reward differences instead of reversing them. We will add this comparison to Section 6.3.

The reward mixes noisy correctness feedback with clean formatting feedback. Appendix K reports early outputs that fail to finish within the token budget. Without separating formatting success from correctness among completed outputs, the experiments cannot determine how much improvement comes from each signal.

The paper states this (Section 6.2, Appendix K), but the submitted supplementary records log the truncation rate at every evaluation, which the text did not use. For Qwen3 8B, 9.4% of the untrained model's validation generations are truncated, against 0.07% late in clean training and at most 0.26% at any noise rate, so truncation explains at most about 12 of the roughly 40 points gained. At p=0.50, where only the format penalty is informative, truncation also disappears, yet reward ends at 0.355 ± 0.339 (Table 8). Code-block presence was not logged, so an exact split is not available. We will add the truncation rates to Appendix K.

Can you compare the clean and judge derived advantages on shared response groups to explain why the 4B judge produces substantially worse training outcomes? In particular, how often does each judge reverse or erase useful reward comparisons?

Not from the archived records, which hold aggregate metrics only. The step-0 comparison in our response to the first weakness shows that the 4B judge's errors are almost all failing tests marked as passing, so erased contrasts, not reversed ones, are its dominant failure.

How much of the observed improvement comes from the unchanged formatting penalty? Can you report formatting success, truncation rates, and correctness conditional on producing a complete answer?

Truncation rates: yes; please see our response to the second weakness (9.4% for the untrained model, at most 0.26% late in training). Formatting success and conditional correctness: no, because code-block presence was not logged separately. Late truncation is below 0.3% in every condition, so it does not explain the differences between clean and noisy runs.

Comments on the paper

1 comment

We thank the reviewers. We could not run additional training during the rebuttal period, so the responses use results already in the paper and in the submitted supplementary records. Two of these were logged but not reported in the text, and the revision will add them: truncation rates at every evaluation (Appendix K) and a comparison of the two LLM judges on the same untrained policy (Section 6.3). The revision will also rewrite the abstract and list the contributions in the introduction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.