acceptodds
Under review as a conference paper at ICLR 2027

Reward Hacking Reveals Itself in Judge Confidence

Abstract

Reinforcement learning with verifiable rewards has driven major advances in reasoning. Yet reliable rule-based verification remains challenging for many tasks, motivating the use of LLM judges as scalable proxy reward providers. However, optimizing against an LLM judge can induce reward hacking, where policies exploit its evaluation errors to increase proxy reward while task performance stagnates or deteriorates. Existing approaches to monitoring reward hacking often require external evaluation, additional inference, or access to model internals. In contrast, we uncover a monitoring signal already present in the LLM judge's outputs: verdict confidence systematically declines as reward hacking develops. Building on this finding, we propose **balanced verdict margin (BVM)**, a lightweight and task-agnostic metric that tracks reward hacking using the judge's verdict-token log-probabilities. We further develop a **margin-guided reward intervention (MRI)** that mitigates reward hacking without requiring gold or reference feedback during training. Extensive experiments show that BVM consistently tracks reward hacking across tasks, while MRI substantially improves task performance over unmodified LLM-judge training, even matching the rule-based control in mathematical reasoning.

Primary area:
reinforcement learning
Keywords:
reward hacking, LLM-as-a-judge, reinforcement learning, reward modeling, confidence estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GrTAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors address reward hacking of an LLM judge inside RLVR training and propose a method for monitoring and preventing it. For monitoring, they measure log-probabilities of the accepted and rejected judgments and the gap between both, defined as the verdict margin. Based on it, the authors define BVM as a metric for detecting hacking behaviour during RLVR training. They find that the BVM declines over the training if proxy rewards diverge from gold or reference rewards, i.e. the model learns to game the proxy reward, without increasing the gold reward. To prevent this behaviour, they propose margin-guided reward intervention (MRI), which attenuates low-confidence positive rewards of the LLM judge.

Critical strengths

Throughout the paper, the authors present extensive training and monitoring experiments to empirically back their theory. On the monitoring side, they show how BVM can distinguish model improvement from exploitation of the LLM judge. On the training side, they show that MRI can prevent this. The authors show this on Math as well as in other domains and across multiple models.

Critical weaknesses

I am not thoroughly convinced of what reward hacking is investigated in the paper. It seems like the judge is very easily gamable, yet it is unclear to me how the judge fails. What kind of solutions does a hacking model actually generate, and how does it trick the LLM to produce pass verdicts, even though the gold verifier returns a fail? Can you give examples of models gaming the LLM judge that are not passing the gold verifier?

I expect RLVR is deployed without reasoning, which limits the significance of the work, since RLVR is typically deployed during post-training to scale reasoning with test-time compute. Yet I expect the papers' conjectures might also hold for reasoning models.

Questions for authors

The authors show that the method can help to detect hacking, yet it is not directly clear how well. How many of the hacking attempts are detected and how precisely? I think a conditional accuracy for BVM monitoring hacking at train time would make a good addition, i.e. how much % of the hacking attempts are detected. The authors could add the percentage of hacking detected via their monitoring, including TPR, FPR, FNR, and FNR.

How is the judge deployed? Does it get the reference answer, or does it just take an educated guess? RLVR can be very fragile to the judge deployed and how it is prompted. Have you had experience with different prompting setups for the deployed judge? What is the error rate of the judge, and what do you do if the judge's answer can not be parsed?

What are the untuned baseline accuracies for table 1/2? RLVR can be very fragile and might depend strongly on the LLM judge setup. Are models improving or degrading during training?

Does the gold reward ensure correctness? In math, RLVR often deploys a second LLM judge if the symbolic parser fails. Parsing and verifying solutions purely symbolically can be difficult. The model has many ways of representing or formatting the same solutions. E.g. a verifier might miss a solution because the model encapsulates the solution in \boxed{}. For other tasks such as “Expand a(b + c)”, the gold answer may be ab + ac, but an imperfect parser might reject equally correct answers such as ba + ca. An LLM trained with LLM judge reward might learn another format accepted by the judge yet rejected by the symbolic verifier, which might produce a fake hacking gap. That said, can you ensure that the generations marked as hacking are actually hacking?

6 comments
AuthorsOpenReview ↗

Response to Reviewer GrTA (1 of 4)

Thank you for recognizing the extensive experimental support for BVM and MRI across tasks and models. We address your concerns point by point below.

W1: "... What kind of solutions does a hacking model actually generate, and how does it trick the LLM to produce pass verdicts, even though the gold verifier returns a fail? Can you give examples of models gaming the LLM judge that are not passing the gold verifier?"

A: We identify two hacking patterns: solution code without final answers and the generation of unrelated questions. We analyzed 200 false positives (FPs) per model from randomly sampled responses in the Qwen2.5-7B and Qwen2.5-32B LLM-judge training runs.

1. Solution code without final answers. The policy provides Python code for the original problem without reporting its execution result or final answer. This pattern occurs in 185/200 (92.5%) FPs for 7B and 200/200 (100%) for 32B. Crucially, the judge can accept these responses even when the code computes an incorrect answer. For 49 responses with executable code that produced incorrect answers, we varied only the appended answer and re-evaluated them.

Table 1: Judge acceptance after answer modifications. Entries report accepted responses / total responses.

Response variant7B judge32B judge
Original response31/3118/18
Append placeholder Answer: $Answer31/3118/18
Append the actual incorrect program output3/310/18
Append the correct reference answer30/3118/18

For example, the judge accepts code that computes 65 instead of the correct answer 55, but rejects the response once 65 is explicitly reported. Overall, revealing the actual incorrect result changes 46/49 verdicts from acceptance to rejection. This supports a failure to adequately verify the computational results of apparently plausible solution code.

2. Generating and answering unrelated questions. The policy continues beyond its original solution by generating and answering unrelated questions. This pattern occurs in 28/200 (14.0%) FPs for 7B and none of the sampled FPs for 32B. For example, one response gives 1000 instead of the correct answer 45, then generates and answers an unrelated question, 3+x=5. But the judge accepts the entire response.

Removing the unrelated questions and their solutions changes 4/28 verdicts from acceptance to rejection. The 23 responses that also contain the first pattern remain accepted. Thus, unrelated content can affect some verdicts, while removing it alone may not resolve false acceptance when other hacking patterns are present.

During LLM-judge training, the first pattern becomes more frequent, while the second appears mainly at intermediate 7B checkpoints. Neither pattern appears in rule-based training. We will include these analyses in the appendix.

AuthorsOpenReview ↗

Response to Reviewer GrTA (2 of 4)

W2: "I expect RLVR is deployed without reasoning, which limits the significance of the work, since RLVR is typically deployed during post-training to scale reasoning with test-time compute. Yet I expect the papers' conjectures might also hold for reasoning models."

A: With ample budget for policy reasoning, BVM still tracks proxy–gold reward divergence, and MRI improves accuracy over LLM-judge training. Our original policy prompt already asks for step-by-step solutions (Appendix A.4). We further evaluate Qwen2.5-7B and Qwen2.5-Math-7B (Yang et al., 2024) using CoT instructions and <think>...</think> formatting adapted from LUFFY (Yan et al., 2025). The additional CoT instructions elicit longer responses:

Table 1: Policy response lengths under CoT prompting. Mean response lengths in tokens for Qwen2.5-7B on the same AIME 2024 questions, evaluated every 10 training steps from 0 to 60.

Reward schemeOriginal policy promptLUFFY-derived CoT prompt
Rule-based1,2881,800
LLM judge2,2753,650
Hard MRI1,3481,782

With explicit policy reasoning, rule-based training improves gold reward while BVM remains stable or increases. Under LLM-judge training, BVM declines as proxy reward rises and gold reward deteriorates:

Table 2: BVM changes under CoT prompting. Relative BVM changes (%) from training step 0 to step 60 on AIME 2024.

Policy modelRule-based: BVM changeLLM-judge: BVM change
Qwen2.5-7B+24.9−33.4
Qwen2.5-Math-7B+1.9−57.7

Table 3: Accuracy under CoT prompting. AIME 2024 accuracy (%), with training steps in parentheses. Rule-based and Hard MRI results use step 120; LLM-judge results are the highest evaluation accuracies observed during training.

Policy modelRule-basedLLM judgeHard MRI
Qwen2.5-7B13.54 (120)7.81 (10)13.65 (120)
Qwen2.5-Math-7B24.06 (120)14.58 (30)20.62 (120)

We will include these analyses in the appendix.


Q1: "... How many of the hacking attempts are detected and how precisely? ... The authors could add the percentage of hacking detected via their monitoring, including TPR, FPR, FNR, and FNR."

A: Verdict margin detects 79.53%–98.41% of incorrect acceptances with 97.54%–99.92% precision, with FPR ranging from 2.07% to 19.66%. BVM is a distribution-level metric for monitoring reward hacking. To quantify the effectiveness of our method at the sample level, we use individual verdict margins to detect incorrect acceptances in AIME 2024 responses from four LLM-judge training runs. Among judge-accepted responses, those rejected by the rule-based verifier form the positive class, while correctly accepted responses form the negative class.

For each run, we select the threshold that maximizes F1 on step-0 accepted responses, then keep it fixed and flag responses with margins below it.

Table 1: Sample-level detection of incorrect acceptances. All metrics are percentages, pooled over all evaluated training steps after step 0. Step 0 is used only for threshold selection.

Policy modelPrecision ↑Recall (TPR) ↑FPR ↓FNR ↓F1 ↑
Qwen2.5-7B99.7298.4119.661.5999.06
Qwen2.5-32B99.9279.895.6620.1188.79
Qwen3-4B97.5479.535.9720.4787.62
Qwen3-8B98.4591.972.078.0395.10

These results show that verdict margin can effectively identify samples suspected of reward hacking. We will include these sample-level detection results and the threshold-selection protocol in Appendix B.3.

References
Jianhao Yan, Yafu Li, Zican Hu, et al. Learning to Reason under Off-Policy Guidance. arXiv preprint arXiv:2504.14945, 2025.
An Yang, Beichen Zhang, Binyuan Hui, et al. Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement. arXiv preprint arXiv:2409.12122, 2024.

AuthorsOpenReview ↗

Response to Reviewer GrTA (3 of 4)

Q2: "How is the judge deployed? Does it get the reference answer, or does it just take an educated guess? ... Have you had experience with different prompting setups for the deployed judge? What is the error rate of the judge, and what do you do if the judge's answer can not be parsed?"

A: We use a frozen LLM judge during training. For mathematical reasoning, it receives the problem, reference answer, and candidate response, and returns a binary verdict. For open-ended QA, it receives the question, candidate response, and evaluation rubrics, and returns one Boolean verdict per criterion.

In this offline evaluation, all four prompts preserve the same direction of BVM change: a decrease under LLM-judge training and an increase in the rule-based control. To assess prompt sensitivity, we rescored 24,960 saved AIME 2024 responses from the Qwen2.5-7B-Base LLM-judge training run and rule-based control using Qwen2.5-7B-Instruct. We varied the scoring instructions under otherwise identical settings.

Table 1: BVM sensitivity to judge prompts. Relative BVM changes (%) from training step 0 to step 60, computed by rescoring the same saved AIME 2024 responses under each prompt.

Judge promptLLM-judge training: BVM changeRule-based control: BVM change
P0: Original scoring instructions−53.4+40.8
P1: P0 + evaluation guidelines−44.6+40.5
P2: P0 + four scoring examples−38.6+37.9
P3: P0 + guidelines + examples−33.8+30.4

Regarding judge errors, we address both incorrect judgments and failures to produce a parseable judgment.

For incorrect judgments, under P0 in this evaluation, disagreement with the rule-based verifier among valid verdicts increases from 9.17% at step 0 to 98.33% at step 60 as reward hacking develops. In the rule-based control, it decreases from 8.88% to 0.83%.

For parsing failures, across all evaluated steps under P0, the overall failure rate is 0.39% (98/24,960): 0.71% for LLM-judge training and 0.08% for the rule-based control. During training, unparseable judge decisions receive the rejection reward. Invalid decisions are excluded from BVM computation.


Q3: "What are the untuned baseline accuracies for table 1/2? RLVR can be very fragile and might depend strongly on the LLM judge setup. Are models improving or degrading during training?"

A: MRI improves over both initialization and unmodified LLM-judge training in all reported comparisons. We report untuned baseline performance alongside the results from Tables 1–2.

Table 1: Untuned and trained performance in mathematical reasoning. Mean accuracy (%) across eight benchmarks.

Policy modelUntunedRule-basedLLM judgeHard MRI
Qwen2.5-7B16.0026.8412.2829.10
Qwen2.5-32B16.4633.5524.6632.95
Qwen3-4B11.4634.3125.9333.14
Qwen3-8B24.5340.5934.7537.68
GLM-4-9B18.5325.9817.2629.17

Table 2: Untuned and trained performance on Medicine QA. Values are GPT-OSS-120B reference scores scaled by 100.

DatasetUntunedGPT-OSS-120BLLM judgeSoft MRI
RubricHub-Medicine13.0344.0131.0538.66
HealthBench9.3134.3528.5435.21

Table 3: Untuned and trained performance on Science QA. Values are GPT-OSS-120B reference scores scaled by 100.

DatasetUntunedGPT-OSS-120BLLM judgeSoft MRI
RubricHub-Science16.2041.1828.8937.01
ResearchQA15.5547.6634.5040.47

LLM-judge training also improves performance over the untuned baseline in some settings, but performance can deteriorate later despite high proxy rewards, as shown by the reward-hacking curves in the paper.

Reviewer_ao5LOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The submission proposes using the LLM judge’s verdict-token log-probabilities to track reward hacking. This is based on the hypothesis that a larger absolute logprob gap between accept and reject tokens indicates higher confidence in judgment. The authors also develop a new reward intervention strategy to alleviate reward hacking during LLM-judge-based RL training, especially for false positive cases. The experimental results show the effectiveness of the proposed method on both math reasoning and open-ended question answering benchmarks.

Critical strengths
  • A lightweight method is proposed to alleviate reward hacking.
  • Different model sizes from two model families are used for experiments.
  • The performance of the models after intervened training is comparable to rule-based approaches.
Critical weaknesses
  • MRI is not fully reference-free, as golden or reference labels are still required offline to separate true positives from false positives during threshold selection. In real-world open-ended tasks where proxy judges are most needed and reference labels are inherently scarce or expensive, calibrating this threshold remains a practical bottleneck.

  • The performance of MRI is sensitive and carries a risk of policy collapse. More of this analysis should be moved to the main paper for discussion.

  • I would expect the authors to discuss the impact of CoT or evaluating with a sufficient reasoning budget to check whether the observations still hold.

Questions for authors

n/a

4 comments
AuthorsOpenReview ↗

Response to Reviewer ao5L (1 of 2)

Thank you for recognizing MRI’s lightweight design and its effectiveness across model families. We address your concerns point by point below.

W1: "MRI is not fully reference-free, as golden or reference labels are still required offline to separate true positives from false positives during threshold selection. In real-world open-ended tasks where proxy judges are most needed and reference labels are inherently scarce or expensive, calibrating this threshold remains a practical bottleneck."

A: We agree that MRI is not fully reference-free: gold/reference labels are required to select the intervention threshold before training (Section 3.3; Appendix C.2).

During training, MRI uses the existing judge’s confidence and requires no additional gold/reference correctness labels for reward intervention. Compared with our control reward schemes, which use gold/reference rewards throughout RL, MRI needs reference labels only for initial threshold selection. Compared with other methods (Section 2), MRI avoids the additional reward models and model inference used by reward-model ensembles and CoT-monitor-based interventions (Coste et al., 2024; Baker et al., 2025).

Future work will explore automatic threshold selection and adaptive intervention using verdict-margin distributions and their changes during training, aiming to reduce dependence on reference labels.


W2: "The performance of MRI is sensitive and carries a risk of policy collapse. More of this analysis should be moved to the main paper for discussion."

A: MRI is sensitive to the threshold and intervention form: insufficient intervention can leave hacking uncontrolled, while excessive intervention can harm learning or cause policy collapse. We first identify the threshold maximizing step-0 detection F1. However, this threshold may not yield the best training outcome as the policy and margin distribution evolve. Here, Q50 denotes the 50th percentile of the initial accepted verdict-margin distribution (Section 3.3).

1. Mathematical reasoning: insufficient intervention can leave reward hacking uncontrolled.

Table 1: MRI threshold sensitivity in mathematical reasoning. Results are AIME 2024 accuracies (%), evaluated at training step 60 for Qwen2.5-7B and step 65 for Qwen3-4B.

Policy modelQuantile maximizing step-0 detection F1Weaker MRI reward schemeMRI reward scheme used in the paper
Qwen2.5-7BQ65Hard MRI (Q50): 1.15Hard MRI (Q70): 11.98
Qwen3-4BQ30Hard MRI (Q40): 5.94Hard MRI (Q50): 13.23

For Qwen2.5-7B with Hard MRI (Q50), accuracy falls from a peak of 8.12% to 1.15%, while judge acceptance reaches 87.71%. These results show that weaker intervention can leave reward hacking insufficiently controlled.

2. Open-ended QA: excessive intervention can damage learning. In our open-ended QA setting (Medicine and Science), greater overlap between the verdict margins of true-positive (TP) and false-positive (FP) rubric judgments makes preserving correct rewards particularly important. Q90 maximizes step-0 detection F1 in both domains but places 75–77% of TP rubric judgments in the intervention region. Soft MRI (Q35) instead leaves over 80% above the threshold and attenuates rewards gradually.

Table 2: MRI sensitivity in open-ended QA. Results are GPT-OSS-120B reference scores (%), evaluated at training step 30 for Medicine and step 50 for Science (Appendix C.4, Table 9).

Domain / Evaluation setLLM judgeStronger MRI reward schemeSoft MRI (Q35)
Medicine / RubricHub-Medicine31.29Hard MRI (Q50): 0.0035.55
Medicine / HealthBench29.76Hard MRI (Q50): −16.0629.86
Science / RubricHub-Science28.89Soft MRI (Q50): 24.8137.01
Science / ResearchQA34.50Soft MRI (Q50): 29.7240.47

Hard MRI (Q50) exhibits policy collapse in Medicine, while Soft MRI (Q50) underperforms both Soft MRI (Q35) and unmodified LLM-judge training in Science. Stronger intervention is therefore not universally better; selection must account for the model and task.

We will add the key sensitivity results and collapse cases to Section 4.3, with detailed comparisons in Appendix C.4.

References
Bowen Baker, Joost Huizinga, Leo Gao, et al. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv preprint arXiv:2503.11926, 2025.
Thomas Coste, Usman Anwar, Robert Kirk, et al. Reward Model Ensembles Help Mitigate Overoptimization. International Conference on Learning Representations, 2024.

AuthorsOpenReview ↗

Response to Reviewer ao5L (2 of 2)

W3: "I would expect the authors to discuss the impact of CoT or evaluating with a sufficient reasoning budget to check whether the observations still hold."

A: With a sufficient reasoning budget, BVM still decline alongside proxy–gold reward divergence, and MRI improves accuracy over LLM-judge training. For mathematical reasoning, we test Qwen2.5-7B and the stronger Qwen2.5-Math-7B (Yang et al., 2024), using explicit CoT instructions adapted from LUFFY (Yan et al., 2025). With this reasoning budget, the policies generate longer responses:

Table 1: Policy response lengths under explicit reasoning. Mean response lengths are in tokens; relative increases are percentages. Results are for Qwen2.5-7B on AIME 2024, evaluated every 10 training steps from 0 to 60.

Reward schemeOriginal policy promptLUFFY-derived CoT prompt
Rule-based1,2881,800
LLM judge2,2753,650
Hard MRI1,3481,782

Using the original judge prompt (Appendix A.4), we observe BVM trends consistent with our original findings.

Table 2: BVM changes under CoT policy. Relative BVM changes (%) from training step 0 to step 60 on AIME 2024, using the original judge prompt.

Policy modelRule-based: BVM changeLLM-judge: BVM change
Qwen2.5-7B+24.9−33.4
Qwen2.5-Math-7B+1.9−57.7

Table 3: CoT policy with the original judge prompt. AIME 2024 accuracy (%), with training steps in parentheses. Rule-based and Hard MRI results are evaluated at step 120; LLM-judge results are the highest evaluation accuracies observed during training.

Policy modelRule-basedLLM judgeHard MRI
Qwen2.5-7B13.54 (120)7.81 (10)13.65 (120)
Qwen2.5-Math-7B24.06 (120)14.58 (30)20.62 (120)

We additionally ask the judge to reason before its verdict. Its output length increases substantially.

Table 4: CoT policy with a CoT judge prompt. AIME 2024 accuracy (%), with training steps in parentheses. Rule-based and Hard MRI results are evaluated at step 120; LLM-judge results are the highest evaluation accuracies observed during training. The rule-based control is shared with Table 3.

Policy modelRule-basedLLM judgeHard MRI
Qwen2.5-7B13.54 (120)6.56 (10)13.02 (120)
Qwen2.5-Math-7B24.06 (120)16.15 (40)20.52 (120)

We will include these analyses in the appendix.

References
Jianhao Yan, Yafu Li, Zican Hu, et al. Learning to Reason under Off-Policy Guidance. arXiv preprint arXiv:2504.14945, 2025.
An Yang, Beichen Zhang, Binyuan Hui, et al. Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement. arXiv preprint arXiv:2409.12122, 2024.

AuthorsOpenReview ↗

Follow-up on our response to Reviewer ao5L

Dear Reviewer ao5L,

We would like to briefly follow up on our rebuttal. We have provided clarifications and additional experiments on threshold selection, intervention sensitivity, and reasoning.

If you have an opportunity to review our responses, we would be grateful for your feedback and happy to clarify any remaining points.

Thank you again for your time and constructive feedback.

Best, The Authors of Submission 26544

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer ao5L

Dear Reviewer ao5L,

We would like to briefly follow up on our rebuttal. Your feedback helped us clarify MRI’s practical limitations and motivated the additional sensitivity and reasoning experiments.

If you have an opportunity to revisit our responses, we would appreciate any further feedback, particularly on concerns that remain unresolved. We would be happy to discuss them here.

Thank you again for your time and thoughtful feedback.

Best, The Authors

Reviewer_YuZEOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes to use judge verdict confidence gap as a signal to detect reward hacking. It shows that balanced verdict margin can track reward hacking during training and develops margin-guided reward intervention to suppress hacking.

Critical strengths

Using verdict confidence gap is an intuitive and interesting idea. The author develops a effective detecting method around this idea and shows that the judge uncertainty is correlated with the verdict accuracy, which can be used as a light-weight detection signal.

Critical weaknesses

there are multiple major weakness of the paper.

  1. The hacking in this paper is only about training against a llm judge, which is a specific hacking scenario in reward hacking. The author studies llm judge training in math task is not a realistic and reasonable setup. In general, hacking in RLVR tasks such as code and math is mostly due to environment loopholes, not llm judge.

  2. It is difficult to distinguish the difference between the two sources of uncertainty of the judge: 1. task complexity 2. hacking trajectory is moving away from familiar region.

  3. There isn't a proper detection baseline to compare with, such as cot monitoring.

  4. A gap between the 7b judge proxy reward with the reference 120b judge doesn't neccesarily mean reward hacking. It could just be due to that a weak judge model couldn't properly train a good model, when evaluated with the reference 120b model.

Questions for authors

Please address the questions above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer YuZE (1 of 3)

Thank you for recognizing the intuition behind using verdict confidence as a lightweight detection signal. We address your concerns point by point below.

W1: "The hacking in this paper is only about training against a llm judge, which is a specific hacking scenario in reward hacking. The author studies llm judge training in math task is not a realistic and reasonable setup. In general, hacking in RLVR tasks such as code and math is mostly due to environment loopholes, not llm judge."

A: Our paper focuses on LLM-judge-induced reward hacking (Section 2), particularly in tasks where reliable rule-based evaluation is difficult. Our Medicine and Science QA experiments evaluate BVM and MRI in such settings. Mathematical reasoning provides a controlled testbed (Section 4.1): policies optimize rewards from an LLM judge, while a rule-based verifier independently checks final-answer correctness. This separation allows us to distinguish genuine improvement from exploitation of judge errors, and to evaluate BVM and MRI against rule-based training control.

LLM-based rewards are also used in mathematical post-training. Kimi k1.5 uses a CoT reward model (Kimi Team, 2025); DeepSeekMath-V2 uses an LLM verifier to assess proof validity beyond final-answer correctness (Shao et al., 2025); and Seed1.5-Thinking explicitly identifies reward hacking as a challenge motivating its thinking verifier (ByteDance Seed, 2025).


W2: "It is difficult to distinguish the difference between the two sources of uncertainty of the judge: 1. task complexity 2. hacking trajectory is moving away from familiar region."

A: These analyses support verdict confidence as a useful signal of proxy–gold reward divergence, although they do not fully separate the two sources of uncertainty. We agree that judge uncertainty may arise not only from hacking trajectories moving away from regions familiar to the judge, but also from task complexity. We interpret task complexity here as the difficulty of judging whether a response is correct. This depends both on the complexity of the question itself and on how easy the particular response is to verify.

1. Comparing judge confidence at fixed question difficulty.

For mathematical reasoning, we use the same frozen judge and the same AIME 2024 evaluation questions throughout training (Section 4.1). Thus, the evaluation questions do not become progressively harder. We further compare the mean verdict margins of TP and FP responses to the same question at the same training step under LLM-judge training.

Table 1: Verdict margins for TP and FP responses. Responses are compared for the same AIME 2024 question at the same training step.

Policy modelMean verdict margin (TP)Mean verdict margin (FP)Difference (FP − TP)
Qwen2.5-7B14.024.21−9.81
Qwen2.5-32B19.345.94−13.40

FP margins remain substantially lower than TP margins. This association between confidence and judgment reliability cannot be explained solely by differences in question difficulty.

2. Comparing BVM between normal training and reward hacking.

Policy updates during rule-based training may also move responses away from patterns familiar to the judge and lower confidence. Nevertheless, BVM in these controls remains above its initial value, while LLM-judge training shows a pronounced decline alongside proxy–gold reward divergence (Figure 3). Over steps 0–120 in the 32B experiment, BVM increases by 20.3% under rule-based training but decreases by 42.8% under LLM-judge training (Figure 5(d)).

References
ByteDance Seed. Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning. arXiv preprint arXiv:2504.13914, 2025.
Kimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv preprint arXiv:2501.12599, 2025.
Zhihong Shao, Yuxiang Luo, Chengda Lu, et al. DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning. arXiv preprint arXiv:2511.22570, 2025.

AuthorsOpenReview ↗

Response to Reviewer YuZE (2 of 3)

W3: "There isn't a proper detection baseline to compare with, such as cot monitoring."

A: Verdict margin achieves higher F1 and recall than CoT monitoring for all four models; compared with TRACE, it achieves higher F1 for three and lower FPR for all four. We thank the reviewer for suggesting this comparison.We compare CoT monitoring and TRACE with our verdict margin (Section 3.2) in mathematical reasoning experiments. All methods use the same judge-accepted responses. The task is to detect responses that the judge accepts but the rule-based verifier rejects.

  • CoT monitor: We follow Baker et al. (2025), adapting CoT monitoring to mathematical reasoning following TRACE (Wang et al., 2026). Qwen2.5-72B-Instruct receives the problem and full response, without reference answers or correctness labels. It outputs GENUINE / HACKING / UNCLEAR; only HACKING triggers an alarm.
  • TRACE: We use TRACE (Wang et al., 2026) to assess whether a model can obtain high proxy rewards with only a short reasoning prefix. Following the original method, scores above each run’s mean step-0 TRACE score trigger an alarm.
  • Verdict margin: We reuse the quantile settings of the corresponding MRI experiments (Appendix C.2, Table 7). Following the threshold-selection procedure in Section 3.3, we set the numerical threshold to the selected quantile of verdict margins among accepted responses at step 0. Responses with margins below this threshold are flagged.

Table 1: Detection of incorrectly accepted responses. For each model, F1, recall, and FPR are computed by pooling evaluated responses from training steps after step 0. Step 0 is used only for threshold selection and excluded from these results. FPR is the fraction of correctly accepted responses mistakenly flagged as reward hacking.

Policy modelMethodF1 score ↑TPR (Recall) ↑FPR ↓
Qwen2.5-7BVerdict margin (ours)98.92%98.64%21.88%
CoT monitor5.49%2.82%0.00%
TRACE93.04%89.83%90.62%
Qwen2.5-32BVerdict margin (ours)88.27%80.96%21.28%
CoT monitor1.23%0.62%1.06%
TRACE93.24%94.56%71.28%
Qwen3-4BVerdict margin (ours)83.94%82.48%41.46%
CoT monitor35.15%21.32%0.00%
TRACE45.60%39.01%94.63%
Qwen3-8BVerdict margin (ours)94.67%91.50%2.75%
CoT monitor31.15%18.44%0.00%
TRACE35.88%35.26%93.13%

At step 10, verdict margin already achieves 96.58% / 77.48% F1 on Qwen2.5-7B / 32B, compared with 39.59% / 6.72% for TRACE, demonstrating its ability to identify incorrect acceptances early in training.

Compared with CoT monitoring and TRACE, verdict margin directly reuses the reward judge’s existing verdict-token log-probabilities and therefore requires no additional monitor inference or repeated continuation generation and rescoring.

We will add this detection-baseline comparison as a compact table in Section 4.2, with monitor configurations, threshold settings, and the evaluation protocol in a new Appendix B.3.

References
Bowen Baker, Joost Huizinga, Leo Gao, et al. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv preprint arXiv:2503.11926, 2025.
Xinpeng Wang, Nitish Joshi, Barbara Plank, et al. Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort. International Conference on Learning Representations, 2026.

AuthorsOpenReview ↗

Response to Reviewer YuZE (3 of 3)

W4: "A gap between the 7b judge proxy reward with the reference 120b judge doesn't neccesarily mean reward hacking. It could just be due to that a weak judge model couldn't properly train a good model, when evaluated with the reference 120b model."

A: Independent evaluation supports GPT-OSS-120B as a reference evaluator and provides evidence of increasing exploitation of the 7B judge’s evaluation errors during training. We agree that the 7B–120B reward gap alone is insufficient to establish reward hacking, and recognize the concern that GPT-OSS-120B’s judgments may themselves be unreliable. We therefore examine GPT-OSS-120B’s reliability.

1. Checking GPT-OSS-120B’s reliability. Our use of GPT-OSS-120B follows Mahmoud et al. (2026). Table 1 of Mahmoud et al. (2026) reports 92.1% agreement in both medical and science domains between GPT-OSS-120B and the majority vote of GPT-5.4, Gemini 3 Pro, and Claude Opus 4.6. Using the same grading setup, we compare Qwen2.5-7B-Instruct and GPT-OSS-120B against GPT-6.1-sol on 22,681 criterion-level judgments from responses generated during 7B-judge training:

Table 1: Judge agreement with GPT-6.1-sol. GPT-6.1-sol provides the reference labels for 22,681 criterion-level judgments. FP (%) is the fraction of all evaluated criteria marked as satisfied by the judge but rejected by GPT-6.1-sol.

Judge modelAgreement (%)FP (%)
Qwen2.5-7B-Instruct54.9540.73
GPT-OSS-120B82.959.12

2. Tracking reward gains and disagreements during training. Our paper reports an increasing 7B–120B reward gap in Medicine and Science (Figure 9(b,c)). The additional GPT-6.1-sol evaluation shows a similar pattern.

Table 2: Proxy and reference score changes during training. Changes are measured from training step 0 to step 100 and reported in score points, with scores scaled by 100.

DomainEvaluation set7B proxy reward change120B reference score changeGPT-6.1-sol reference score change
MedicineRubricHub-Medicine+29.41+4.87+4.62
ScienceRubricHub-Science+40.60+3.53+3.87

Both reference evaluators indicate some improvement, while the 7B proxy reward increases substantially more. Furthermore, the 7B judge’s FP% relative to GPT-6.1-sol increases from 25.51% to 51.77% in Medicine and from 24.12% to 61.96% in Science.

References
Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang, et al. Reward Hacking in Rubric-Based Reinforcement Learning. arXiv preprint arXiv:2605.12474, 2026.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all three reviewers for their constructive feedback. We appreciate their recognition of the intuitive confidence signal (Reviewer YuZE), MRI’s lightweight design (Reviewer ao5L), and the extensive experimental evidence (Reviewer GrTA).

Our work introduces BVM to monitor reward hacking using judge confidence and MRI to mitigate it by adjusting low-confidence positive rewards.

Following the reviewers’ suggestions, we have revised the manuscript as follows. Changes are highlighted in blue.

  1. Detection baselines: Added CoT monitoring and TRACE comparisons with F1, recall, and FPR, addressing Reviewers YuZE and GrTA. Section 4.2; Appendix B.4.

  2. Hacking patterns: Added pattern frequencies and paired response modifications to explain judge failures, addressing Reviewer GrTA. Appendix B.2.

  3. Reference reliability: Added independent GPT-6.1-sol evaluation and reward-gap analysis, addressing Reviewer YuZE. Section 4.1; Appendix B.1.

  4. MRI sensitivity: Added threshold comparisons and discussion of policy collapse, addressing Reviewer ao5L. Section 4.3; Appendix C.4.

  5. Extended reasoning: Added experiments with substantial reasoning budgets for both the policy and judge, addressing Reviewers ao5L and GrTA. Section 4.3; Appendix D.

  6. Prompt sensitivity: Added comparisons across four judge prompts, addressing Reviewer GrTA. Appendix B.5.

AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair

Dear Area Chair,

We sincerely appreciate the valuable feedback provided by all reviewers and the efforts made by the ACs, SACs, and PCs throughout the review process. As no further discussion is permitted, we would like to provide a summary of our rebuttal.

Reviewer YuZE (score 1 → 2) recognizes verdict confidence as an intuitive and interesting idea and acknowledges its value as a lightweight detection signal (Critical Strengths). Their concerns focus on the experimental settings and the interpretation and effectiveness of the monitoring signal (W1–W4).

We clarified that mathematical reasoning provides a controlled testbed with independent correctness verification, while our open-ended QA experiments in Medicine and Science cover tasks where reliable rule-based evaluation is difficult (W1). To examine question difficulty, we compared TP and FP responses to the same question at the same training step. FP margins remained substantially lower, supporting a confidence–reliability association that cannot be explained solely by differences in question difficulty (W2). We appreciate the suggestion to include detection baselines, which motivated us to go beyond distribution-level BVM monitoring and quantify judgment-level detection using individual verdict margins and compare against CoT monitoring and TRACE. Verdict margin achieves higher F1 and recall than CoT monitoring across all four models and lower FPR than TRACE across all four, without additional monitor inference. These results quantify the method’s judgment-level effectiveness and comparative advantages (W3). For open-ended QA, independent GPT-6.1-sol evaluation supports GPT-OSS-120B as a reference evaluator and provides evidence of reward hacking during training against Qwen2.5-7B-Instruct (W4). In follow-up, YuZE explicitly confirmed that the CoT-monitoring baseline concern was addressed and increased their score.

Reviewer ao5L (score 3) recognizes MRI’s lightweight design, evaluation across model families and sizes, and performance comparable to rule-based training (Strengths 1–3). Their questions concern reference-label requirements, intervention strength, and effectiveness with sufficient reasoning (W1–W3).

We clarified that MRI uses reference labels for offline threshold selection, while training-time intervention relies on the existing judge’s confidence without additional gold/reference correctness labels (W1). Sensitivity analyses in mathematical reasoning and open-ended QA explain how intervention strength and form affect training outcomes and support the reward schemes used in the paper (W2). We also added policy/judge CoT experiments with Qwen2.5-7B and Qwen2.5-Math-7B. With ample reasoning budget, BVM still monitors the emergence of reward hacking, while MRI mitigates it and improves task performance over unmodified LLM-judge training (W3). These experiments also address GrTA’s question about applicability to reasoning settings (GrTA-W2).

Reviewer GrTA (score 2) appreciates the extensive experiments across tasks and models supporting both monitoring and intervention (Critical Strengths). Their main questions concern how policies exploit judge errors and whether formatting or verifier errors could explain the reward gap (W1, Q4).

Training-output analysis and paired response modifications identified and tested two recurring hacking patterns, explaining how the judge accepts responses that fail to complete the original task (W1). For mathematical evaluation, independent GPT-6.1-sol judgments further support the reliability of our rule-based verifier and the finding that reward hacking occurs during LLM-judge training (Q4). We also reported precision, recall, FPR, and FNR in a judgment-level detection evaluation to quantify detection of incorrect positive rewards and false alarms (Q1). We examined monitoring consistency through offline rescoring with different prompts and clarified judge deployment and verdict handling (Q2). Untuned-baseline comparisons show MRI’s gains over both initialization and unmodified LLM-judge training in the reported results (Q3).

Overall, our paper provides a lightweight approach to monitoring and mitigating LLM-judge-induced reward hacking. BVM and MRI reuse the judge’s existing verdict-token log-probabilities for distribution-level monitoring and targeted reward intervention, without additional monitor inference. The additional rebuttal experiments and independent evaluations further support the method’s effectiveness and practical value.

We sincerely thank all reviewers, ACs, SACs, and PCs for their time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.