- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors address reward hacking of an LLM judge inside RLVR training and propose a method for monitoring and preventing it. For monitoring, they measure log-probabilities of the accepted and rejected judgments and the gap between both, defined as the verdict margin. Based on it, the authors define BVM as a metric for detecting hacking behaviour during RLVR training. They find that the BVM declines over the training if proxy rewards diverge from gold or reference rewards, i.e. the model learns to game the proxy reward, without increasing the gold reward. To prevent this behaviour, they propose margin-guided reward intervention (MRI), which attenuates low-confidence positive rewards of the LLM judge.
Critical strengths
Throughout the paper, the authors present extensive training and monitoring experiments to empirically back their theory. On the monitoring side, they show how BVM can distinguish model improvement from exploitation of the LLM judge. On the training side, they show that MRI can prevent this. The authors show this on Math as well as in other domains and across multiple models.
Critical weaknesses
I am not thoroughly convinced of what reward hacking is investigated in the paper. It seems like the judge is very easily gamable, yet it is unclear to me how the judge fails. What kind of solutions does a hacking model actually generate, and how does it trick the LLM to produce pass verdicts, even though the gold verifier returns a fail? Can you give examples of models gaming the LLM judge that are not passing the gold verifier?
I expect RLVR is deployed without reasoning, which limits the significance of the work, since RLVR is typically deployed during post-training to scale reasoning with test-time compute. Yet I expect the papers' conjectures might also hold for reasoning models.
Questions for authors
The authors show that the method can help to detect hacking, yet it is not directly clear how well. How many of the hacking attempts are detected and how precisely? I think a conditional accuracy for BVM monitoring hacking at train time would make a good addition, i.e. how much % of the hacking attempts are detected. The authors could add the percentage of hacking detected via their monitoring, including TPR, FPR, FNR, and FNR.
How is the judge deployed? Does it get the reference answer, or does it just take an educated guess? RLVR can be very fragile to the judge deployed and how it is prompted. Have you had experience with different prompting setups for the deployed judge? What is the error rate of the judge, and what do you do if the judge's answer can not be parsed?
What are the untuned baseline accuracies for table 1/2? RLVR can be very fragile and might depend strongly on the LLM judge setup. Are models improving or degrading during training?
Does the gold reward ensure correctness? In math, RLVR often deploys a second LLM judge if the symbolic parser fails. Parsing and verifying solutions purely symbolically can be difficult. The model has many ways of representing or formatting the same solutions. E.g. a verifier might miss a solution because the model encapsulates the solution in \boxed{}. For other tasks such as “Expand a(b + c)”, the gold answer may be ab + ac, but an imperfect parser might reject equally correct answers such as ba + ca. An LLM trained with LLM judge reward might learn another format accepted by the judge yet rejected by the symbolic verifier, which might produce a fake hacking gap. That said, can you ensure that the generations marked as hacking are actually hacking?
6 comments
Response to Reviewer GrTA (2 of 4)
W2: "I expect RLVR is deployed without reasoning, which limits the significance of the work, since RLVR is typically deployed during post-training to scale reasoning with test-time compute. Yet I expect the papers' conjectures might also hold for reasoning models."
A: With ample budget for policy reasoning, BVM still tracks proxy–gold reward divergence, and MRI improves accuracy over LLM-judge training. Our original policy prompt already asks for step-by-step solutions (Appendix A.4). We further evaluate Qwen2.5-7B and Qwen2.5-Math-7B (Yang et al., 2024) using CoT instructions and <think>...</think> formatting adapted from LUFFY (Yan et al., 2025). The additional CoT instructions elicit longer responses:
Table 1: Policy response lengths under CoT prompting. Mean response lengths in tokens for Qwen2.5-7B on the same AIME 2024 questions, evaluated every 10 training steps from 0 to 60.
| Reward scheme | Original policy prompt | LUFFY-derived CoT prompt |
|---|---|---|
| Rule-based | 1,288 | 1,800 |
| LLM judge | 2,275 | 3,650 |
| Hard MRI | 1,348 | 1,782 |
With explicit policy reasoning, rule-based training improves gold reward while BVM remains stable or increases. Under LLM-judge training, BVM declines as proxy reward rises and gold reward deteriorates:
Table 2: BVM changes under CoT prompting. Relative BVM changes (%) from training step 0 to step 60 on AIME 2024.
| Policy model | Rule-based: BVM change | LLM-judge: BVM change |
|---|---|---|
| Qwen2.5-7B | +24.9 | −33.4 |
| Qwen2.5-Math-7B | +1.9 | −57.7 |
Table 3: Accuracy under CoT prompting. AIME 2024 accuracy (%), with training steps in parentheses. Rule-based and Hard MRI results use step 120; LLM-judge results are the highest evaluation accuracies observed during training.
| Policy model | Rule-based | LLM judge | Hard MRI |
|---|---|---|---|
| Qwen2.5-7B | 13.54 (120) | 7.81 (10) | 13.65 (120) |
| Qwen2.5-Math-7B | 24.06 (120) | 14.58 (30) | 20.62 (120) |
We will include these analyses in the appendix.
Q1: "... How many of the hacking attempts are detected and how precisely? ... The authors could add the percentage of hacking detected via their monitoring, including TPR, FPR, FNR, and FNR."
A: Verdict margin detects 79.53%–98.41% of incorrect acceptances with 97.54%–99.92% precision, with FPR ranging from 2.07% to 19.66%. BVM is a distribution-level metric for monitoring reward hacking. To quantify the effectiveness of our method at the sample level, we use individual verdict margins to detect incorrect acceptances in AIME 2024 responses from four LLM-judge training runs. Among judge-accepted responses, those rejected by the rule-based verifier form the positive class, while correctly accepted responses form the negative class.
For each run, we select the threshold that maximizes F1 on step-0 accepted responses, then keep it fixed and flag responses with margins below it.
Table 1: Sample-level detection of incorrect acceptances. All metrics are percentages, pooled over all evaluated training steps after step 0. Step 0 is used only for threshold selection.
| Policy model | Precision ↑ | Recall (TPR) ↑ | FPR ↓ | FNR ↓ | F1 ↑ |
|---|---|---|---|---|---|
| Qwen2.5-7B | 99.72 | 98.41 | 19.66 | 1.59 | 99.06 |
| Qwen2.5-32B | 99.92 | 79.89 | 5.66 | 20.11 | 88.79 |
| Qwen3-4B | 97.54 | 79.53 | 5.97 | 20.47 | 87.62 |
| Qwen3-8B | 98.45 | 91.97 | 2.07 | 8.03 | 95.10 |
These results show that verdict margin can effectively identify samples suspected of reward hacking. We will include these sample-level detection results and the threshold-selection protocol in Appendix B.3.
References
Jianhao Yan, Yafu Li, Zican Hu, et al. Learning to Reason under Off-Policy Guidance. arXiv preprint arXiv:2504.14945, 2025.
An Yang, Beichen Zhang, Binyuan Hui, et al. Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement. arXiv preprint arXiv:2409.12122, 2024.
Response to Reviewer GrTA (3 of 4)
Q2: "How is the judge deployed? Does it get the reference answer, or does it just take an educated guess? ... Have you had experience with different prompting setups for the deployed judge? What is the error rate of the judge, and what do you do if the judge's answer can not be parsed?"
A: We use a frozen LLM judge during training. For mathematical reasoning, it receives the problem, reference answer, and candidate response, and returns a binary verdict. For open-ended QA, it receives the question, candidate response, and evaluation rubrics, and returns one Boolean verdict per criterion.
In this offline evaluation, all four prompts preserve the same direction of BVM change: a decrease under LLM-judge training and an increase in the rule-based control. To assess prompt sensitivity, we rescored 24,960 saved AIME 2024 responses from the Qwen2.5-7B-Base LLM-judge training run and rule-based control using Qwen2.5-7B-Instruct. We varied the scoring instructions under otherwise identical settings.
Table 1: BVM sensitivity to judge prompts. Relative BVM changes (%) from training step 0 to step 60, computed by rescoring the same saved AIME 2024 responses under each prompt.
| Judge prompt | LLM-judge training: BVM change | Rule-based control: BVM change |
|---|---|---|
| P0: Original scoring instructions | −53.4 | +40.8 |
| P1: P0 + evaluation guidelines | −44.6 | +40.5 |
| P2: P0 + four scoring examples | −38.6 | +37.9 |
| P3: P0 + guidelines + examples | −33.8 | +30.4 |
Regarding judge errors, we address both incorrect judgments and failures to produce a parseable judgment.
For incorrect judgments, under P0 in this evaluation, disagreement with the rule-based verifier among valid verdicts increases from 9.17% at step 0 to 98.33% at step 60 as reward hacking develops. In the rule-based control, it decreases from 8.88% to 0.83%.
For parsing failures, across all evaluated steps under P0, the overall failure rate is 0.39% (98/24,960): 0.71% for LLM-judge training and 0.08% for the rule-based control. During training, unparseable judge decisions receive the rejection reward. Invalid decisions are excluded from BVM computation.
Q3: "What are the untuned baseline accuracies for table 1/2? RLVR can be very fragile and might depend strongly on the LLM judge setup. Are models improving or degrading during training?"
A: MRI improves over both initialization and unmodified LLM-judge training in all reported comparisons. We report untuned baseline performance alongside the results from Tables 1–2.
Table 1: Untuned and trained performance in mathematical reasoning. Mean accuracy (%) across eight benchmarks.
| Policy model | Untuned | Rule-based | LLM judge | Hard MRI |
|---|---|---|---|---|
| Qwen2.5-7B | 16.00 | 26.84 | 12.28 | 29.10 |
| Qwen2.5-32B | 16.46 | 33.55 | 24.66 | 32.95 |
| Qwen3-4B | 11.46 | 34.31 | 25.93 | 33.14 |
| Qwen3-8B | 24.53 | 40.59 | 34.75 | 37.68 |
| GLM-4-9B | 18.53 | 25.98 | 17.26 | 29.17 |
Table 2: Untuned and trained performance on Medicine QA. Values are GPT-OSS-120B reference scores scaled by 100.
| Dataset | Untuned | GPT-OSS-120B | LLM judge | Soft MRI |
|---|---|---|---|---|
| RubricHub-Medicine | 13.03 | 44.01 | 31.05 | 38.66 |
| HealthBench | 9.31 | 34.35 | 28.54 | 35.21 |
Table 3: Untuned and trained performance on Science QA. Values are GPT-OSS-120B reference scores scaled by 100.
| Dataset | Untuned | GPT-OSS-120B | LLM judge | Soft MRI |
|---|---|---|---|---|
| RubricHub-Science | 16.20 | 41.18 | 28.89 | 37.01 |
| ResearchQA | 15.55 | 47.66 | 34.50 | 40.47 |
LLM-judge training also improves performance over the untuned baseline in some settings, but performance can deteriorate later despite high proxy rewards, as shown by the reward-hacking curves in the paper.
Response to Reviewer GrTA (1 of 4)
Thank you for recognizing the extensive experimental support for BVM and MRI across tasks and models. We address your concerns point by point below.
A: We identify two hacking patterns: solution code without final answers and the generation of unrelated questions. We analyzed 200 false positives (FPs) per model from randomly sampled responses in the Qwen2.5-7B and Qwen2.5-32B LLM-judge training runs.
1. Solution code without final answers. The policy provides Python code for the original problem without reporting its execution result or final answer. This pattern occurs in 185/200 (92.5%) FPs for 7B and 200/200 (100%) for 32B. Crucially, the judge can accept these responses even when the code computes an incorrect answer. For 49 responses with executable code that produced incorrect answers, we varied only the appended answer and re-evaluated them.
Table 1: Judge acceptance after answer modifications. Entries report accepted responses / total responses.
Answer: $AnswerFor example, the judge accepts code that computes 65 instead of the correct answer 55, but rejects the response once 65 is explicitly reported. Overall, revealing the actual incorrect result changes 46/49 verdicts from acceptance to rejection. This supports a failure to adequately verify the computational results of apparently plausible solution code.
2. Generating and answering unrelated questions. The policy continues beyond its original solution by generating and answering unrelated questions. This pattern occurs in 28/200 (14.0%) FPs for 7B and none of the sampled FPs for 32B. For example, one response gives 1000 instead of the correct answer 45, then generates and answers an unrelated question, 3+x=5. But the judge accepts the entire response.
Removing the unrelated questions and their solutions changes 4/28 verdicts from acceptance to rejection. The 23 responses that also contain the first pattern remain accepted. Thus, unrelated content can affect some verdicts, while removing it alone may not resolve false acceptance when other hacking patterns are present.
During LLM-judge training, the first pattern becomes more frequent, while the second appears mainly at intermediate 7B checkpoints. Neither pattern appears in rule-based training. We will include these analyses in the appendix.