acceptodds
Under review as a conference paper at ICLR 2027

Judge Noise, Not Difficulty: Diagnosing Weak Open-Source Verifiers in RLVR

Abstract

Reinforcement learning from verifiable rewards (RLVR) increasingly replaces the gold verifier with a learned or zero-shot LLM judge, especially in verify-hard domains such as multi-hop question answering where exact-match signals are brittle. A common implicit assumption is that such judges are reliable enough: that their per-example verdicts reflect true answer correctness, so that the group statistics driving RLVR are estimated on signal rather than noise. We show this assumption fails badly for weak open-source judges that are used in practice. On a controlled multi-hop QA verification suite, three weak base judges (Qwen3-8B, Qwen2.5-7B, and Qwen3-4B) exhibit per-prompt verdict instability of 39–66%: given the same question-candidate pair, the same judge flips between correct and wrong across repeated samples. A strong commercial judge is stable (0% flips) on the same prompts, confirming that the instability is a property of weak verifiers rather than the task. This instability is not a parsing artifact: one judge parses a clean verdict 99% of the time yet still flips on 66% of prompts. Because variance-driven rollout allocation cannot distinguish judge noise from true difficulty, it mis-allocates sampling budget toward judge-unstable prompts. We further report a negative control: an RLVR policy trained against a same-source judge yields no robust gain even under that same judge (mean +2.6 points, standard deviation 2.0 points across repeated scoring, spanning zero), because single-run judge scores are too noisy to trust. These results argue that judge-noise diagnostics should be a standard prerequisite before adopting an open-source verifier as an RLVR reward.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, reinforcement learning from verifiable rewards, large language model judges, verifier reliability, reward noise

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7DQkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the noise that comes from using an LLM judge to verify responses, and how this noise affects RL training with rewards produced by that judge. The setting is multi-hop question answering on MuSiQue, where exact-match rewards are brittle and LLM judges are often used instead. The authors test three small open-source judges (Qwen3-8B, Qwen2.5-7B, and Qwen3-4B) and compare them against a strong commercial judge. For a fixed prompt and response, each judge is sampled 8 times, and the authors propose three metrics to quantify judgment noise: the average per-prompt variance of the verdicts, the fraction of prompts with at least one flip, and the fraction of prompts where the verdicts are close to a coin flip. They report that the small judges flip their verdicts on 39–66% of prompts, while the commercial judge does not flip on the prompts it was tested on. They also show that the flips happen on both correct and wrong candidates, and that they are not explained by failures to parse the verdict. The authors then connect this noise to GRPO, arguing that since GRPO relies on per-prompt reward variance, judge noise can be mistaken for prompt difficulty. Finally, they train a policy with GRPO using one of the small judges as the reward and find no robust improvement, which they present as a negative control.

Critical strengths
  1. LLMs are used in both training and evaluation as verifiers and paper demonstrates that a small set of open-source verifiers are too noisy to be relied on.

  2. The link between reward variance, which is the main signal GRPO relies on, and the noise from judging a response a single time is the most interesting part of the paper. It points out that judge noise can contaminate the training signal.

Critical weaknesses
  1. The findings are expected: LLM judges producing noisy verdicts is well known, especially for small judges like the ones studied here (up to 8B) so the significance of the study is limited.
  2. Limited novelty: The connection between GRPO’s variance signal and single-pass judge noise is interesting but also fairly obvious: if reward variance comes partly from judge noise, the signal is contaminated. The paper does not go much beyond stating this, so I don’t see a novel contribution.
Questions for authors
  1. Did the authors investigate the effect of explicitly asking the verifier to provide an explanation before producing a verdict? How would that change the flip rate?
  2. Did the authors study the effect of adding few-shot examples in the system prompt of the grader? how does this affect the flip rate?
Reviewer_T42kOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether LLM judges used as rewards in RLVR for verify-hard domains give stable verdicts. It proposes a simple diagnostic: for a fixed (question, candidate, gold reference) triple from MuSiQue, query the judge k=8k=8 times at τ=0.7\tau=0.7 and report the mean per-prompt Bernoulli variance, a has-flip rate (any disagreement), and an unstable-flip rate (p^i(1−p^i)≥0.1875\hat p_i(1-\hat p_i)\ge 0.1875) (Sec. 3.2, 3.6). Three open-source base checkpoints (Qwen3-8B, Qwen2.5-7B, Qwen3-4B) are compared with a commercial API judge on 29 prompts. The weak judges have unstable-flip rates of 39.1–65.8% while the API judge never flips (Table 1), Qwen2.5-7B is unstable despite parsing 98.8% of the time (Fig. 4), flip rates are similar on gold-correct and gold-wrong candidates (Table 2), and a single sample agrees with the 8-sample majority 75.0–87.2% of the time (Table 3).

From this, the paper argues that variance-driven rollout allocation would mistake judge noise for difficulty (Sec. 3.3, 4.5). It also reports a GRPO "negative control" on Qwen3-8B with a same-source judge reward, where gold accuracy moves from 8.2% to 8.6% while the same-source judge gain varies between −0.2 and +4.4 points over three scoring repeats (Table 4).

Critical strengths
  • The diagnostic is simple and cheap, and it separates parse failures from verdict instability. Fixing the (question, candidate) pair and resampling only the judge isolates judge stochasticity, and reporting the parse rate next to the flip rates shows that a judge can give well-formed verdicts that are still irreproducible (Qwen2.5-7B: 98.8% parsed, 65.8% unstable-flip). The single-pass vs. majority agreement (Table 3, Fig. 6) is a practical summary that others could adopt. This is the main usable contribution of the paper.
  • The training result is framed honestly. The paper presents it as a negative control rather than a positive result, repeats the same-source scoring three times, and states in Sec. 1 and 5 that it does not establish an end-to-end causal chain. The point that single-pass judge evaluations cannot certify small gains (Sec. 4.4–4.5) is correct, although it is a known caution.
Critical weaknesses
  • The central claim, that judge noise misleads variance-driven allocation and is "not difficulty", is never tested, and the measured quantity is not the one an allocator reads. The allocator in Sec. 3.3 reads reward variance across different rollouts of a prompt, while the paper measures variance across repeated judge calls on one fixed candidate. The paper does not decompose rollout-level reward variance into judge noise and true correctness, does not implement any allocation method (or cite one), and does not measure difficulty, e.g., policy pass rate. So I cannot follow Sec. 4.5 when it calls the 0.11–0.17 mean variance "a direct estimate of the size of that misdirected component": that depends on how judge noise compares with between-rollout variance. Table 2 stratifies by candidate correctness, not difficulty, so equal flip rates are still compatible with noise concentrated on ambiguous questions, and calling the noise "prompt-correlated" (Sec. 2, 5) sits awkwardly with the "generic noise" reading. Since the title, abstract and recommendation rest on this link, the main claim is unsupported.
  • The judge setup does not represent "judges actually used in practice", and judge accuracy is never reported. All open-source judges are base checkpoints sampled at τ=0.7\tau=0.7 (Sec. 3.6), and the Qwen3 judges parse only 60.1% and 57.2% of the time (Table 1). In my understanding, practitioners mostly use instruction-tuned or purpose-trained verifiers with greedy or low-temperature decoding, and the paper gives no evidence for its "same sampling regime an RLVR pipeline would use" claim. With the gold reference in the prompt, an 87.2% has-flip rate for Qwen2.5-7B points to a weak judge configuration rather than to weak open-source verifiers in general. Without majority-vote accuracy against gold, τ=0\tau=0 results, instruct variants, or a larger open model, I cannot tell instability from plain inaccuracy. The strong-judge control is also small: 0/29 flips still has a 95% upper bound of about 10%, and it is unclear whether these 29 prompts are a random subset of the weak judges' prompts.

Other, less critical points:

  • Prompts are included once they have at least 2 parsed samples (Sec. 4), so the Qwen3 judges are scored on fewer samples per prompt than Qwen2.5-7B. This changes the flip rates mechanically (with two samples, any disagreement counts as unstable), so the comparisons in Table 1 and Fig. 1 are not like-for-like. Table 3 likewise keeps only 193/803 and 157/815 prompts with all 8 samples parsed.
  • The training control is weak. The policy reaches only 8.2–8.6% gold accuracy, so the null gold gain may reflect a setup with almost no signal, there is one training seed, and the same-source judge is not identified. With three repeats, the standard error of the mean gain is about 1.15 points, so "spans zero" describes the individual repeats rather than uncertainty about the mean.
  • The Fig. 5 caption calls the single-run +2.2 gain an "upper-tail draw", but it is below the 3-repeat mean of +2.6.
  • nit: Sec. 3.6 says hyperparameters are "listed in Appendix B", but Appendix B defers to supplementary scripts.
Questions for authors
  1. (Clarification) Are the three open-source judges base checkpoints rather than the Instruct variants, and is Qwen3 thinking mode on or off for the diagnostic? What is each judge's majority-vote accuracy against the gold labels on the gold-correct and gold-wrong strata? If the judges are also inaccurate, the distinction between noise and difficulty needs restating.
  2. (Suggestion) Please repeat Table 1 with greedy decoding and with the instruction-tuned versions of the same models, ideally adding a larger open model (e.g., Qwen2.5-32B/72B-Instruct) or a purpose-trained verifier. If the instability largely disappears, the conclusion becomes "do not sample base models at τ=0.7\tau=0.7" rather than "weak open-source judges are unreliable".
  3. (Suggestion) To support the allocation claim, please decompose rollout-level reward variance on real GRPO prompts into judge noise and between-rollout variance (e.g., by re-judging each rollout kk times), and correlate judge variance with an independent difficulty measure such as policy pass@k under the gold matcher. Ideally, also show that a variance-driven allocator using the weak judge picks a different and worse set of prompts than one using gold rewards.
  4. (Clarification) How were the 29 doubao prompts chosen relative to the weak-judge prompts? How do the Table 1 numbers change on a common prompt set with a common number of parsed samples per prompt?
  5. (Clarification) For the training control, which model is the same-source judge, what are the GRPO hyperparameters and number of seeds, and is the single-run +2.2 gain one of the three repeats in Table 4?

My score could increase if the authors (i) report judge accuracy and show that the instability persists for instruction-tuned judges at low temperature (Q1, Q2), and (ii) show directly that judge noise distorts rollout-level reward variance and allocation (Q3). Without (ii), the title and main claim are not supported.

Reviewer_XbrCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors focus on the issue that instability in the judge model within the RLVR framework can lead to misleading sampling and inefficient allocation of training resources.This topic has research significance. However, the experimental evaluation is limited.There is overlap in the samples used for voting and a lack of samples involving strong judges. The metric used to assess stability may not capture the accuracy of the judgments, failing to substantiate the paper's core arguments.

Critical strengths

The authors investigate the instability of rewards when small models act as judges in RLVR. It presents a method to integrate these rewards and reduce the waste of computational resources arising from judge instability. The authors introduce a protocol for repeated scoring of fixed inputs. This mechanism permits quantification of a judge's inherent instability. Rather than relying on a single score, the study performs three re-evaluations for each model output. It thus provides more reliable statistical evidence for the training improvements.

Critical weaknesses
  • The paper adopts the average variance of judgments as a proxy for stability. However, stability does not correspond to accuracy or the usefulness of the reward for training. A judge may consistently produce the same assessment. For example,it might always label an output as "correct". Such a judge might fail to possess actual ability to check whether responses are correct. By contrast, a judge that has randomness could still provide meaningful training signals, provided it remains broadly accurate.

  • There is a severe imbalance in sample sizes. Strong judges have only 29 valid samples, compared to 803–944 for weak judges. The main text and Appendix A trace this discrepancy to differences in parsing success ratios. However, Appendix B states that the lower sample count for commercial stronger models arises from API request limitations.

  • The experimental setup is narrow. It encompasses only three base models and one dataset. The temperature parameter is fixed at 0.7. For this reason, it remains impossible to determine whether the observed instability stems from a lack of model capability.

  • In Table 3, the majority vote obtained from a subset of k scores is compared with the majority vote derived from the full set of 8 scores. This design incurs sample overlap. The observed rise in consistency may simply be an artifact of this overlap. This issue makes the results unreliable. It also fails to demonstrate that increased sampling brings about improved scoring accuracy.

Questions for authors

Please refer to the weakness part.

Comments on the paper

1 comment

Dear Reviewers,

The authors have replied to your initial comments. Please have a look and engage in the discussion.

Thanks!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.