- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies reward hacking when policies are optimized against reference-free LLM judges. Hidden correctness checks reveal cases where judge acceptance increases substantially without corresponding accuracy improvements. The paper evaluates transfer across judge families, ensemble rewards, and additional tasks, and uses randomized exposure to wrong candidates to measure their influence on judges’ own answers. The proposed mitigation obtains an independently committed answer before comparison with the candidate. The paper also introduces a judge–truth gap decomposition and theoretical statements about independent matching and monotone aggregation.
Critical strengths
The randomized exposure experiment provides informative evidence beyond a correlation between reward and accuracy. Section 5.2 compares sealed question-only answers with answers produced after exposure to a randomly selected wrong candidate. Measuring movement toward that specific candidate through external matching avoids using the judge’s own verdict as ground truth and provides direct evidence of candidate influence.
The empirical investigation includes meaningful controls and exposes limitations of the proposed remedy. Frozen-judge analysis, oracle-reward controls, cross-family re-scoring, ensemble training, and crossed inference-budget experiments strengthen the central finding. The analyses of answer-space collisions and sparse preference-pair coverage also help explain when independent matching is useful and what it costs.
Critical weaknesses
The interpretation of Corollary 1 is stronger than what Proposition 1 alone establishes. Proposition 1 bounds false positives for protocols that both produce a candidate-independent answer and accept only on exact match. A generic score-based judge exceeding that bound need not be anchored; it may simply violate the exact-match acceptance condition. For example, a constant-accept rule can exceed the bound without using the candidate. Thus, the proposed diagnostic tests a combination of assumptions rather than identifying anchoring alone. The randomized exposure experiment supplies separate evidence of anchoring, but the corollary and the broader dichotomy should distinguish these claims.
The headline mitigation effect needs more careful attribution to protocol and reasoning computation. Appendix C.8 usefully separates these factors: Table 10 reports protocol effects of approximately 0.08–0.10 in false-positive rate, compared with budget effects of approximately 0.73–0.74. At the short budget, commit-first still has a false-positive rate of 0.776. This supports a protocol benefit, but qualifies the framing that a single commitment choice explains the much larger headline reduction. For training-time prevention, Appendix C.3 also compares different token caps, 1,536 for the anchored judge and 2,048 for blind solving. These results should be presented with their distinct conditions clearly separated.
Questions for authors
Can Corollary 1 be restricted to an appropriate acceptance-protocol class, or restated as a test of the joint independent-answer and exact-match assumptions? How would the proposed diagnostic distinguish anchoring from a permissive acceptance rule?
Could the abstract and main conclusions distinguish the matched-budget protocol effect from the combined effect of protocol and reasoning computation? The crossed experiment already provides useful evidence for this distinction.
For the longer-budget anchored training control in Appendix C.3, can you report held-out accuracy and usable preference-pair coverage alongside false-positive rate? This would clarify whether the lower false-positive rate of independent matching also yields a better learning signal under comparable resources.