- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies the noise that comes from using an LLM judge to verify responses, and how this noise affects RL training with rewards produced by that judge. The setting is multi-hop question answering on MuSiQue, where exact-match rewards are brittle and LLM judges are often used instead. The authors test three small open-source judges (Qwen3-8B, Qwen2.5-7B, and Qwen3-4B) and compare them against a strong commercial judge. For a fixed prompt and response, each judge is sampled 8 times, and the authors propose three metrics to quantify judgment noise: the average per-prompt variance of the verdicts, the fraction of prompts with at least one flip, and the fraction of prompts where the verdicts are close to a coin flip. They report that the small judges flip their verdicts on 39–66% of prompts, while the commercial judge does not flip on the prompts it was tested on. They also show that the flips happen on both correct and wrong candidates, and that they are not explained by failures to parse the verdict. The authors then connect this noise to GRPO, arguing that since GRPO relies on per-prompt reward variance, judge noise can be mistaken for prompt difficulty. Finally, they train a policy with GRPO using one of the small judges as the reward and find no robust improvement, which they present as a negative control.
Critical strengths
-
LLMs are used in both training and evaluation as verifiers and paper demonstrates that a small set of open-source verifiers are too noisy to be relied on.
-
The link between reward variance, which is the main signal GRPO relies on, and the noise from judging a response a single time is the most interesting part of the paper. It points out that judge noise can contaminate the training signal.
Critical weaknesses
- The findings are expected: LLM judges producing noisy verdicts is well known, especially for small judges like the ones studied here (up to 8B) so the significance of the study is limited.
- Limited novelty: The connection between GRPO’s variance signal and single-pass judge noise is interesting but also fairly obvious: if reward variance comes partly from judge noise, the signal is contaminated. The paper does not go much beyond stating this, so I don’t see a novel contribution.
Questions for authors
- Did the authors investigate the effect of explicitly asking the verifier to provide an explanation before producing a verdict? How would that change the flip rate?
- Did the authors study the effect of adding few-shot examples in the system prompt of the grader? how does this affect the flip rate?
Dear Reviewers,
The authors have replied to your initial comments. Please have a look and engage in the discussion.
Thanks!
Your AC