- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether noisy reward signals are sufficient for RL training. They define 4 noise models, where some portion of the (rollouts x unit_test_case) reward matrix for a given prompt is noised by flipping the binary test-case label with some probability p. They show that on MBPP, Qwen 3 8B and GLM reach about the same performance under (a) ground truth rewards and (b) group-flipped rewards where every test case for a particular group is flipped with probability p=0.1. Other noise models show slightly diminished performance, and a 4B model-based judge shows worst performance of all.
The core result with group-flipped rewards is further verified on Llama 3.1 8B.
Critical strengths
I appreciate the inclusion of several noise models, which highlights the distinction between verifier error rate and error structure -- even under the same error rate, different errors have differing amounts of prompt/rollout correlation. This distinction in itself is not super surprising, but I feel there could be something interesting here if pushed further. For example, perhaps systematically characterizing how much reward bias vs reward variance is tolerable.
Critical weaknesses
My primary concerns are that (1) the finding that noisy rewards have useful training signal is unsurprising to me, and (2) the reward noise considered does not reflect the sorts of noise one observes in practice, thus limiting the generality of the paper's claims.
First, the novelty of the finding. Neural reward models / LLM judges are obvious examples of imperfect verifiers, and it is well-documented that RL training with these imperfect signals can improve student performance - in some sense, this is already a clear existence proof for the paper's central claim. The authors also note a wide literature on noisy rewards for RL in their related work, differentiating their work as "focus[ing] on observed performance under specified noise structures." What new insights does studying this known phenomenon with these noise modes allow us to glean? I would be happy to discuss further.
Second, the practicality of the reward noise. In practice, errors from verifiers are often systematic: they persist across encounters of the same prompt and depend on the content of the rollout (e.g. a bad ground truth answer). All noise models considered in the paper are completely independent of both prompt and rollout content. The model-based verifiers, which do exhibit realistic error structure, show diminished performance relative to ground truth.
Presentation. As a further note, I found the writing to be confusing and hard to parse throughout. For example, when I first read the abstract, I saw mention of "two fixed late evaluations" (what evaluations?), "confidence intervals" that "allow larger losses" (what confidence intervals, and what does loss refer to here?), and "symmetric and asymmetric group noise," all terms which are not yet clearly defined. The prose also repeatedly hedges the main claims (e.g. L134, L164, L180). While it is good to be upfront about limitations, such presentation makes it hard for me to parse exactly what the authors are arguing for.
For these reasons, I tend towards rejection.
Questions for authors
-
How does Llama 8B perform with the other noise models? As the authors noted, the group-flipped reward is the most benign noise. It would be great to be able to contextualize how much of the gain under the other noise modes on Qwen/GLM is due to spurious effects, and how much is from useful signal.
-
How well does training work if we remove the format reward?
-
How well does training work if the noise is not re-sampled for every encounter of a prompt? This lets us better test prompt-correlated errors.
2 comments
Thanks for the reply. I am maintaining my current score, as I remain concerned about the novelty and practicality of this study.
- Thanks for pointing to the p=0.5 runs -- Table 8 to my understanding is on top of the benign group-flipped reward. In absence of similar ablations on top of the other noise models or runs with Llama, we cannot be sure that the gains from other types of noise are not spurious.
- Format rewards do not just help for truncation, and it is possible that the other 28 points gained also come from format improvements irrespective of the correctness reward.
- "Content-dependent error is covered by the two LLM judges" - this is a known finding that imperfect LLM judges can improve performance.
We agree that learning from imperfect judges is known, and the paper does not present that as its finding. Because code has an executable reference, we can set the verifier error exactly and measure its cost: (i) a dose–response curve for whole-group flips: within 1 point of clean training through 20%, −2.0 points at 30%, −8.4 at 40% (Table 8); (ii) structure matters at a fixed 10% rate: group-level errors give 0.891–0.893 and sample-level errors 0.854–0.864 (Table 1); (iii) an analysis of why (Section 6.2, Appendix F): designs with the same expected advantage differ in its variance, which is a first step toward the bias-versus-variance characterisation the reviewer suggests; (iv) high recall is not enough: both LLM judges have recall above 0.90, yet one trains to 0.871 and the other to 0.704 (Figure 5). We will list these contributions in the introduction.
For the four controlled modes, this is correct and intended, and the paper says so (L162–164). Content-dependent error is covered by the two LLM judges (Sections 3.2, 5.2, 6.3), which run at temperature 0, so their errors are tied to the generated code. The 30B judge ends 1.5 points below clean training (0.869 vs 0.884) and the 4B judge fails (0.704 peak); our response to Reviewer 4qgx gives a second 30B seed. For completeness, the group-rollout result covers p = 0.05–0.50 with 3–6 seeds per rate (Table 8).
We agree that the abstract used terms before defining them. The revised abstract defines the evaluation metric, gives the interval as numbers and describes group noise in plain words, for example: "validation reward at the end of training (the mean of the last two evaluations shared by all runs, with no checkpoint selection) is within 1 percentage point of clean training for p up to 20%, about 2 points lower at 30%, and about 8 points lower at 40%." The sentences at L134, L164 and L180 are scope statements; we will state each claim once and move the caveats to the Limitations.
We have not run Llama 3.1 8B under the other three structures (stated at L314–315). The four structures are compared on two other families, Qwen3 8B and GLM4 9B, in Table 1. On spurious versus useful signal, the p=0.50 runs are the control: the test signal is uninformative there, and late reward is 0.355 ± 0.339, against 0.890 for clean training and 0.487 for the untrained model (Tables 1 and 8).
We did not run this ablation; the confound is stated in Section 6.2, Appendix K and the Limitations. Two existing results bound the penalty's role (details in our response to Reviewer jSap, second weakness): at p=0.50, where only the format penalty is informative, late reward is 0.355 ± 0.339 against 0.890 for clean training (Table 8), and truncation explains at most about 12 of the roughly 40 points gained.
We do not have this experiment: all synthetic-noise runs resample the noise at each encounter (L162), and the Limitations say that robustness to persistent biases is not established. The closest existing evidence is the two temperature-0 LLM judges, whose errors are tied to content (see our response to the second weakness). We will name the missing condition in the Limitations.