acceptodds
Under review as a conference paper at ICLR 2027

More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges

Abstract

Self-reward, self-play, and LLM-as-a-judge pipelines for label-free self-improvement train a language model against its own reference-free judgments, assuming that a model's verdict on a shown answer is a usable proxy for correctness. We show the premise fails structurally: conditioned on a candidate, a reference-free judge scores plausibility, not correctness. This verification asymmetry runs opposite to the usual one, because the verifier is itself reference-free, and it leaves false-positive basins of plausible-but-wrong answers a policy learns to exploit. We measure the failure with a hidden-anchor audit: a held-out exact-match check the judge never sees. On GSM8K with Qwen3 policies in a reasoning-suppressed regime, self-play drives the judge's pass rate from 0.72 to 0.94 while true accuracy stays at 0.20. Unlike the judge biases already documented, these errors are optimized rather than incidental, and a better or a broader judge does not remove them: they transfer across the Qwen, Llama, and Gemma families and to larger judges; a strict three-family ensemble still accepts most of them; recompute prompts leave the false-positive rate at 0.850; and training against the ensemble reward widens the basin. What does fix it is a single measurable bit: whether the judge commits its answer independently of the candidate, a bit detectable from the judge's own solve accuracy before any training. Committing first, with the candidate still in view, holds the false-positive rate on wrong answers at 0.044. Used as the training reward with the candidate withheld, the same independent verdict keeps false positives near zero in every setting we run; in our main setting, true accuracy rises under this reward in every seed while the plausibility reward leaves it flat.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, reward hacking, self-play, self-reward, RLAIF, reference-free evaluation, verification asymmetry, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_Hgc7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reward hacking when policies are optimized against reference-free LLM judges. Hidden correctness checks reveal cases where judge acceptance increases substantially without corresponding accuracy improvements. The paper evaluates transfer across judge families, ensemble rewards, and additional tasks, and uses randomized exposure to wrong candidates to measure their influence on judges’ own answers. The proposed mitigation obtains an independently committed answer before comparison with the candidate. The paper also introduces a judge–truth gap decomposition and theoretical statements about independent matching and monotone aggregation.

Critical strengths

The randomized exposure experiment provides informative evidence beyond a correlation between reward and accuracy. Section 5.2 compares sealed question-only answers with answers produced after exposure to a randomly selected wrong candidate. Measuring movement toward that specific candidate through external matching avoids using the judge’s own verdict as ground truth and provides direct evidence of candidate influence.

The empirical investigation includes meaningful controls and exposes limitations of the proposed remedy. Frozen-judge analysis, oracle-reward controls, cross-family re-scoring, ensemble training, and crossed inference-budget experiments strengthen the central finding. The analyses of answer-space collisions and sparse preference-pair coverage also help explain when independent matching is useful and what it costs.

Critical weaknesses

The interpretation of Corollary 1 is stronger than what Proposition 1 alone establishes. Proposition 1 bounds false positives for protocols that both produce a candidate-independent answer and accept only on exact match. A generic score-based judge exceeding that bound need not be anchored; it may simply violate the exact-match acceptance condition. For example, a constant-accept rule can exceed the bound without using the candidate. Thus, the proposed diagnostic tests a combination of assumptions rather than identifying anchoring alone. The randomized exposure experiment supplies separate evidence of anchoring, but the corollary and the broader dichotomy should distinguish these claims.

The headline mitigation effect needs more careful attribution to protocol and reasoning computation. Appendix C.8 usefully separates these factors: Table 10 reports protocol effects of approximately 0.08–0.10 in false-positive rate, compared with budget effects of approximately 0.73–0.74. At the short budget, commit-first still has a false-positive rate of 0.776. This supports a protocol benefit, but qualifies the framing that a single commitment choice explains the much larger headline reduction. For training-time prevention, Appendix C.3 also compares different token caps, 1,536 for the anchored judge and 2,048 for blind solving. These results should be presented with their distinct conditions clearly separated.

Questions for authors

Can Corollary 1 be restricted to an appropriate acceptance-protocol class, or restated as a test of the joint independent-answer and exact-match assumptions? How would the proposed diagnostic distinguish anchoring from a permissive acceptance rule?

Could the abstract and main conclusions distinguish the matched-budget protocol effect from the combined effect of protocol and reasoning computation? The crossed experiment already provides useful evidence for this distinction.

For the longer-budget anchored training control in Appendix C.3, can you report held-out accuracy and usable preference-pair coverage alongside false-positive rate? This would clarify whether the lower false-positive rate of independent matching also yields a better learning signal under comparable resources.

Reviewer_CsJbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes breaking candidate anchoring either by requiring the LLM judge to commit an answer before seeing/using the candidate (commit-first) or withholding the candidate entirely (blind-solve), which ensures the false-positive rate is bounded above by the judge's own solve error.

Critical strengths
  • Clear, solid math behind the verification gap and its error ceiling.
  • A well-designed randomized test that proves showing a candidate directly causes the judge to repeat that mistake.
  • Both mathematical proof and test results confirm that standard judge ensembles still fall for the same convincing errors.
  • Experiments are conducted across model scales (1.7B-14B) and three model families.
Critical weaknesses
  • The proposed fix relies heavily on tasks with objective, final answers that can be checked with rules such as exact match (math and coding with unit tests). As the authors acknowledge, extending this mechanism to open-ended, subjective text generation remains an open challenge.
  • The biggest gaps happen when models are artificially forced into reasoning-suppressed JSON. While cot and code benchmarks are checked, running the full self-play training loop with complete reasoning traces is explored much less thoroughly.
Questions for authors

none

Reviewer_oGiGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reference-free self-improvement where one model answers GSM8K questions, judges its own answers without a reference, and trains with DPO on accepted versus rejected answers. A gold EM check stays outside the loop and measures the gap between the judge's pass rate and true acc. With Qwen3 policies, self-play raises the pass rate from 0.716 to 0.938 while accuracy stays near 0.20. Judges from other families, larger judges and a three-family ensemble also accept the learned wrong answers. The authors trace this to "candidate anchoring" meaning showing a wrong candidate moves the judge's own answer onto it. The authors make the judge commit its own answer before it uses the candidate and accept only on EM. As a training reward, the fix keeps FPR near 0.007 while accuracy rises slightly. The core claim is that the judge's independence from the candidate, decides whether the reward can be hacked, but not its capability.

Critical strengths

Overall, I think this is a careful empirical paper about a problem that matters for anyone who trains on reference-free judge rewards. The measurement setup is simple and easy to reuse. The authors also ran several of the controls I would have asked for, and they report some of these controls even where the results weaken their own main claims (Table 10, Table 6). In terms of clarity, the paper is quite dense and uses many numbers, but the main text and appendix are mostly consistent. The paper gives a clear and quantified warning for label-free pipelines that use cheap judges.

Critical weaknesses
  1. The Corollary 1 check compares an FPR measured at 256 tokens with a solve accuracy measured at 2048 tokens. However Proposition 1 bounds the FPR of an EM judge by that judge's error rate under the same protocol. At 256 tokens with thinking off, the judge's solve accuracy is probably much lower than 0.93. The same Qwen3-4B gets EM 0.209 as a thinking-off JSON policy, and commit-first at 256 tokens still has FPR 0.776 (Table 10). So a 256 token judge with no anchoring at all could exceed 1 - 0.931, and the reported excess of 0.48-0.85 mixes anchoring with the budget gap. The same mismatch appears at Line 313-315: the 0.850 comes from 256 tokens, while "1 - solve-acc = 0.078 ... at this judging budget" comes from 1536 tokens.
Questions for authors
  1. In the prevention comparison how many DPO pairs and optimizer steps did each reward arm use, per iteration and seed? What is the paired bootstrap interval of the EM change from 0.209 to 0.230?

  2. What is each judge's answer accuracy at the [256 token, thinking-off setting, and on the 462 exposure questions?

Reviewer_3TPyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies self-play training with reference-free LLM judges as rewards, arguing that such judges assess plausibility rather than correctness, so the policy learns to produce plausible but incorrect answers. The authors introduce a hidden anchor, an exact-match correctness check never shown to the judge or used in training, and measure the gap between the judge's pass rate and actual accuracy.

Critical strengths
  • The concern that rising judge scores in reference-free self-improvement may not reflect improved accuracy is well motivated, and measuring this with a hidden anchor excluded from training is a clear design. The oracle control, which replaces only the reward with exact match and shows no rise in judge score, appropriately identifies the judge reward as the cause of the inflation. (Section 3, Table 1)
  • The randomized-exposure intervention is convincing. By comparing the probability of producing a given wrong answer with and without the candidate displayed, and reporting the asymmetry between questions switching onto and off the candidate, the authors show that the effect is capture rather than added noise. (Section 5.2, Tables 3, 11)
  • The results are replicated with judges from other families and larger scales, a three-judge ensemble, training-free best-of-N code selection, and a Gemma policy. The paper also crosses inference budget with judging protocol and runs a budget-matched reward, which is a positive effort to check comparisons made under confounded conditions. (Sections 5.1, 5.3–5.4)
Critical weaknesses
  • The main results depend heavily on settings that give the judge a very small inference budget. The headline comparison of 0.850 for the recompute prompt against 0.044 for commit-first compares a 256-token, thinking-off condition with a 1,536-token, thinking-on condition. (Table 4) In Table 10, the budget effect is 0.73–0.74 while the protocol effect is only 0.08–0.10, and commit-first itself has an FPR of 0.776 at 256 tokens. In addition, the budget-matched anchored reward rises only from 0.131 to 0.140 after self-play. (Table 6) Most of the false-positive basin is therefore better explained by a judge that cannot reason sufficiently, and the claim in the abstract and conclusion that a "single measurable bit" decides the outcome is overstated.
  • The experimental setting is distant from practical self-improvement. The policy's GSM8K accuracy is lowered to about 0.2 through a reasoning-suppressed JSON format, and the examples in Table 5 are simple calculation errors such as "11 × 7 = 65". Training uses 128 questions per iteration, two iterations, LoRA rank 8, and at most 96 steps. Under CoT the gap shrinks to 0.10–0.13, it is negative on TruthfulQA, and only three of five Gemma seeds enter the basin. (Sections 5.1, 5.4, Table 9) It needs to be verified whether reward hacking of comparable magnitude occurs at the judging budgets and policy capabilities used in practice.
  • The practicality and novelty of the proposed reward are not sufficiently established. The blind-solve reward resembles using the solutions of a thinking-enabled Qwen3-4B (solve accuracy about 0.92) as pseudo-labels to train a policy with about 0.2 accuracy. It is therefore closer to answer labeling by a stronger verifier than to an alternative self-play reward. The accuracy gain is small (0.209 to 0.230), and only 7.8% of questions yield preference pairs. In code, the de-anchored gap@16 remains 0.227 even with an 8B judge and worsens at 1.7B, and the approach does not apply to open-ended outputs. (Sections 5.2–5.3, Table 13)
Questions for authors
  • How large does the judge–truth gap become over several self-play iterations with an anchored reward given a sufficient reasoning budget (e.g., thinking on, 1,536 tokens or more)?
  • Does self-play produce a basin of comparable size with policies whose reasoning is not suppressed, or on harder tasks where policy accuracy is naturally low (e.g., MATH, AIME)?
  • How could a principle similar to commit-first be applied to open-ended tasks where exact match is unavailable? Please also analyze the source of the residual de-anchored code gap of 0.227.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.