acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) replaces costly human labeling with automated verifiers. To reduce verifier hacking, many RLVR systems binarize rewards to 0,1, but imperfect verifiers inevitably introduce false negatives (FN, rejecting correct answers) and false positives (FP, accepting incorrect ones). We formalize verifier unreliability as a stochastic reward channel with asymmetric noise rates and —the FP rate and the FN rate, respectively. From this abstraction we derive two lightweight corrections: (i) a backward correction that yields an unbiased surrogate reward and thus an unbiased policy-gradient estimator in expectation, and (ii) a forward correction that reweights score-function terms so the expected update aligns with the clean gradient direction and requires only the FN rate. We implement both as lightweight hooks in a group relative policy optimization pipeline, both corrections improve RLVR reasoning under synthetic and real verifier noise, with the forward variant being more stable under heavier noise. We further use a lightweight appeals mechanism to estimate FN rates online and show that the correction can complement stronger verifier-side baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Reinforcement Learning, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_BVxjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the RLVR setting for training LLMs. Instead of studying improvements to the reliability of verifiers, it frames the verifier as fundamentally noisy, which allows to study current training methods' robustness. The paper proposes two methods with reasonable assumptions how to obtain a good training signal, with theoretical grounding. Empirical results on synthetic FP/FN rates support the claims.

Critical strengths
  1. A new perspective to RLVR -- instead of improving verifiers, our training methods can be made more robust against imperfect verifiers. A synergy of improved verifiers and improved training methods may be the right direction for RLVR.
  2. Theoretical results showing the efectiveness of the two methods under reasonable assumptions, and Table 2 showing what happens when assumptions break, i.e., the sensitivity of the method to the FP/FN rates. It illustrates the methods are effective both in theory and in simulated practice.
Critical weaknesses
  1. I was expecting to see results of the full pipeline: what happens when the verifier noise is unknown? How much possible it is to estimate ρ0\rho_0 & ρ1\rho_1? Can the proposed training method with these estimates perform well? The method is well motivated and supported by theory, although a full end-to-end check is missing. Perhaps some of these questions can already be answered based on previous work?
Questions for authors
  1. Figure 4: are there any variance / min-max values / confidence intervals for these results? Do they overlap? Are these the average result across all benchmarks?
  2. Did the authors identify any limitations of their method?
Reviewer_pZz6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how to improve RLVR training when verifiers produce incorrect rewards. It proposes two methods that correct noisy reward signals or policy-gradient updates, showing improved performance across mathematical reasoning benchmarks.

Critical strengths

Provides theoretically motivated correction methods supported by extensive experiments across multiple models, benchmarks, and noise settings.

Critical weaknesses

The theoretical guarantees apply to idealized policy gradients, but the practical implementation modifies GRPO's advantage computation, making it unclear how directly the guarantees translate to the actual training procedure.

Questions for authors
  1. How much of the improvement comes from correcting verifier noise versus changing GRPO's advantage normalization?

  2. How reliable is the estimated false-negative rate when the secondary LLM verifier also makes systematic errors?

Reviewer_wc2dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors model verifier errors in RLVR as a binary reward channel with false-positive rate ρ0\rho_0 and false-negative rate ρ1\rho_1, assumed instance-independent with ρ0+ρ1<1\rho_0 + \rho_1 < 1. From this channel they derive two corrections: a backward correction (PGBC), R^=(R~−ρ0)/(1−ρ0−ρ1)\hat R = (\tilde R - \rho_0)/(1 - \rho_0 - \rho_1), which is unbiased for the clean reward, and a forward correction (PGFC), which reweights the score function by wR~∈{ρ1−1,ρ1}w_{\tilde R} \in \{\rho_1 - 1, \rho_1\} so that the expected update is parallel to the clean gradient and requires only ρ1\rho_1. They also propose estimating ρ1\rho_1 online by appealing a random subset of rule-based negatives to a small LLM verifier (TinyV). Experiments on four backbones (1.5B–7B) trained on DeepScaleR show gains over uncorrected training under both synthetic and real verifier noise.

Critical strengths
  • Relevant problem. False negatives of rule-based checkers and false positives of LLM judges are well documented, and correcting for them at the policy-gradient level is a natural and lightweight complement to improving the verifier itself.
  • Practical estimation of ρ1\rho_1. The appeals mechanism, with a Horvitz–Thompson correction, is a cheap and sensible way to track the FN rate online, and Appeals+PGFC improves over Appeals in 22 of 24 cells of Table 1.
  • Reasonable experimental breadth. The paper covers four backbones, six benchmarks, a sweep over misspecified rates and a non-i.i.d. noise setting.
Critical weaknesses
  • Very close to prior published work, which is not cited. The contribution is very close to that of El Mansouri et al., Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients (arXiv:2510.18924), published at ICML 2026 (poster). That work uses the same asymmetric binary flip channel under the same condition 1−ρ+−ρ−>01 - \rho^+ - \rho^- > 0, and the same Natarajan-style unbiased reward r^=(r~−ρ+)/(1−ρ+−ρ−)\hat r = (\tilde r - \rho^+)/(1 - \rho^+ - \rho^-), which is exactly PGBC (Theorem 3.3). It also integrates this reward into GRPO and Dr. GRPO, addressing the same issue that group centering and standardization cancel an affine correction (Section 3.3, "Bridging"), and shows that the noisy Dr. GRPO gradient is the clean one scaled by 1−ρ+−ρ−1 - \rho^+ - \rho^-, which is the constant cc of Theorem 3.5. The present submission neither cites nor compares with this work. The distinct contributions, namely PGFC, the online appeals-based estimation of ρ1\rho_1 and the larger-scale experiments, should therefore be clearly separated from the shared ones, and the backward correction should be compared empirically with the GRPO variant of that paper.
  • PGFC is close to uncorrected REINFORCE. Since wR~=R~−(1−ρ1)w_{\tilde R} = \tilde R - (1 - \rho_1), PGFC is the noisy-reward score function with a constant baseline. Indeed, because E[R~∣R∗]=cR∗+ρ0\mathbb{E}[\tilde R \mid R^*] = c R^* + \rho_0 and E[G]=0\mathbb{E}[G] = 0, the uncorrected noisy REINFORCE gradient already equals c∇θJ(θ)c \nabla_\theta J(\theta), so Theorem 3.5 holds for any constant baseline. The gains of PGFC over "Noise" must therefore come from the implementation, i.e., replacing GRPO's group-normalized advantage with a fixed coefficient, rather than from a noise correction. A GRPO baseline without standardization, and REINFORCE with a constant baseline, are needed to isolate the effect.
  • Restrictive channel model. The theory assumes instance-independent rates and errors independent of the score, which is precisely what fails for LLM judges, whose errors are content-dependent. The non-i.i.d. test of Table 2 keeps an instance-independent FP rate and is synthetic.
  • Surprising baselines. Under the format-dependent noise of Table 2 (ρ0=0.2\rho_0 = 0.2), uncorrected GRPO ends below the base model (17.6 against 20.3 average pass@1), and so does LLM-as-Judge for Llama in Table 1 (8.7 against 12.6). This calls for an explanation, since the channel remains informative and noise of this kind mainly rescales the expected gradient.
Questions for authors
  1. How does the submission position itself with respect to El Mansouri et al. (ICML 2026), and how does PGBC compare empirically with their noise-corrected GRPO and Dr. GRPO?
  2. Since PGFC equals REINFORCE with a constant baseline, could you compare it with uncorrected Dr. GRPO, GRPO without standardization and REINFORCE with a baseline, under the same noise?
  3. Why does uncorrected GRPO fall below the base model in Table 2?
  4. How does the correction behave when the FP rate is content-dependent, e.g., with an LLM judge as the primary verifier?
Reviewer_PRsMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper models imperfect RLVR verifiers as asymmetric binary reward channels and proposes backward reward correction and forward policy-gradient weighting. Experiments across four backbones and six math benchmarks show improvements under synthetic and real verifier noise, complemented by noise-sensitivity analyses and stronger-verifier comparisons.

Critical strengths
  1. The paper addresses an important practical limitation of RLVR: imperfect verifiers can reject correct solutions or reward incorrect outputs, compromising policy learning.

  2. The proposed corrections are lightweight, and the appeals-based estimation mechanism provides a practical way to incorporate verifier reliability into training.

  3. The empirical evaluation covers multiple backbones, synthetic and real noise, rate misspecification, content-dependent errors, and stronger auxiliary verifiers.

Critical weaknesses
  1. The theoretical justification does not establish a distinctive benefit of PGFC: under the assumed constant noise channel, the uncorrected noisy-reward gradient is already proportional to the clean gradient, and PGFC merely subtracts an action-independent constant. Thus, directional alignment alone does not justify the claimed need for FN-rate estimation.
  2. The main GRPO comparisons confound noise correction with changes to advantage construction: the proposed implementations bypass group standardization and alter baselines and coefficient scales. Although Appendix G.5 includes direct-coefficient REINFORCE results, matched GRPO ablations with tuned constant baselines and reward–KL scales are still needed to isolate the benefit of channel calibration.
  3. The evaluation does not clearly establish an independent correctness standard more reliable than the brittle rule-based checking that motivates the method. An audit using expert judgments or a separately validated verifier would help determine whether the reported gains reflect improved mathematical correctness or greater compatibility with the evaluation parser.
Questions for authors

refer to the weakness please

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.