acceptodds
Under review as a conference paper at ICLR 2027

Efficient Reward Hacking Detection using Answer Likelihood under Truncated Reasoning

Abstract

Reinforcement learning (RL) for language models is vulnerable to reward hacking, in which the model obtains high rewards by exploiting flaws in the data or reward signal rather than correctly solving the intended task. Chain-of-Thought (CoT) monitoring can flag such hacking that the model reveals in its reasoning, but misses implicit reward hacking, where the CoT appears benign. TRACE (Wang et al., 2026) addresses this problem by checking whether answers regenerated from truncated CoTs already earn high reward, but requires generation and reward evaluation at every cutoff. We introduce (n-K ikelihood valuation), which instead scores the likelihood of the model's own final answer under CoT truncation. At each cutoff, MILE teacher-forces this answer to the model, normalizes each token's log-probability against the corresponding next-token distribution, and averages over the lowest-scoring % of tokens. Integrating these scores over the retained CoT fraction yields the detection score, requiring no additional decoding or reward evaluation once the full response has been initially generated. We evaluate MILE on math and code tasks under three controlled loopholes: answer hints in the prompt, a misspecified reward rule, and memorization of training problems. Across models and loophole types, MILE achieves mean F1 and AUROC comparable to TRACE while scoring - faster, and consistently outperforms CoT monitoring even with a larger model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reward Hacking, Chain-of-Thought, Likelihood-based Detection, AI Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bDEqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper builds on previous reward hacking detection work TRACE by Wang et al., which truncates the CoT and forces the model to decode intermediate answers. If a model can get rewards quickly early in the CoT, it serves as a signal for hacking.

This work improves the method by directly taking the final generated answer and measuring the likelihood of that answer after each CoT truncation. The author shows that the proposed method performs similarly as TRACE but with much less compute.

Critical strengths

The paper focuses on improving the efficiency of the previous TRACE reward hacking detection method, which is an important contribution. TRACE requires sampling multiple answers and grading each sampled answer across the truncation point, which makes it difficult to scale to more complicated tasks.

Critical weaknesses

The major weakness of the work is that the method only checks the likelihood of the final answer, which limits the applicable scenarios of the method.

This method assumes that there is one single fixed answer that can hack the tasks, which is likely in the paper's setup but may not be true in the real world cases. It is likely that the model achieves low probability to the final answer in the early CoT, but it will assign high probability to other hacking answers.

I would like to see more analysis on this concerns.

Questions for authors

Please address the concerns listed in the weakness section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bDEq

We sincerely thank the reviewer for their insightful feedback and for recognizing that improving the efficiency of TRACE, which is difficult to scale to more complex tasks, is an important contribution. We address each concern below.


[W1] MILE checks only the likelihood of the final answer, assuming a single fixed hacking answer that may not hold in real-world cases. Early in the CoT, the model may assign low probability to the final answer but high probability to other hacking answers.

[A1] MILE already applies to multiple rewarded answers and our new results show that expanding the answer set further improves it efficiently.

Thank you for raising this point. We address two questions respectively: (i) whether MILE applies when multiple answers are rewarded, as in many real-world cases, and (ii) whether MILE detects a hacking response whose early CoT favors a different hacking answer before switching to the final answer AA.

(i) Validity with multiple rewarded answers.

In fact, MILE is applicable to such tasks, and our experiments in the original version already included one: in Math RM, any negative integer answer is rewarded (Table 1). Even in this case, MILE remains a useful approximation: MILE outperforms TRACE on Llama-3.2-3B-Instruct (F1 .805 vs .701), while trailing it on both Qwen models (.791 vs .927 on Qwen2.5-3B-Instruct and .702 vs .811 on Qwen2.5-7B-Instruct), a gap we attribute to the case in (ii) below. In IC and MM, the rewarded answer is unique, so the assumption holds and the concern does not apply.

To further test MILE under the loophole with multiple rewarded answers, we additionally evaluate it on Code RM. We excluded this setting from the main experiments because its reward is unrealistically permissive, and use it here only as a stress test. Both methods stay near chance in AUROC, and MILE matches TRACE in macro F1 at 53.9× lower cost.

Table A. Detection on Code RM with Qwen2.5-3B-Instruct.

MethodF1Macro F1AUROCTime (s)
TRACE.793.518.5178780.1
MILE.704.518.492163.3 (53.8×)
MILE + (b).817.589.7524481.2 (2.0×)

(ii) Early preference for a different hacking answer. We agree that this is a valid point and the main limitation of scoring only AA. TRACE regenerates answers at each cutoff and can thus capture hacking even when they differ from AA, whereas MILE cannot. This is what happens on Math RM with the Qwen models: at early prefixes, the model is confident of a different number, which makes the normalized score of that number token extremely negative. Since math answers are short (e.g., "-7"), that lowest-scoring number token alone determines the score. Indeed, an aggregation that is not dominated by the number token ("Max" baseline in Table 3) comes close to TRACE.

However, this case is narrower than it may appear, for two reasons. First, RL-trained models tend to concentrate on a few answers as model entropy drops sharply early in RL training [1]. Second, when probability is spread evenly across multiple candidate answers, Min-K%++ does not penalize the score. MILE therefore misses only cases where, at an early prefix, the model is nearly certain of a different hacking answer and later switches to AA.

Toward a fundamental solution. Both (i) and (ii) stem from scoring a single answer, so a natural extension is to expand the answer set, scoring the likelihood of multiple rewarded answers instead of only AA:

  • (a) Reusing RL rollouts: collect rewarded answers from the rollouts of the same prompt during RL training, which requires no additional decoding or reward evaluation.
  • (b) Sampling at early cutoffs: sample answers at early cutoffs, add only the rewarded ones to the candidate set, and score them by likelihood.

We evaluate (b) on Math RM and Code RM with Qwen2.5-3B-Instruct to test its effectiveness. On Math RM, adding the rewarded answers decoded greedily at only three early cutoffs (10%, 30%, and 50%) raises the overall score near TRACE while remaining 2.3× faster (Table B). On Code RM, adding a rewarded answer decoded greedily at the single cutoff (30%) surpasses TRACE while remaining 2.0× faster (Table A). This supports that the gap stems from scoring a single answer and can be largely closed or even outperform with only a few additional decodings. We added this result to the Appendix C.7 of the revised draft.

Table B. Detection on Math RM with Qwen2.5-3B-Instruct.

MethodF1Macro F1AUROCTime (s)
TRACE.927.802.8681252.1
MILE.791.700.766179.1 (7.0×)
MILE + (b).904.760.837539.8 (2.3×)

[1] Cui, Ganqu, et al. “The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models.” arXiv preprint, 2025.


If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors

Hi Reviewer,

The authors have responded. Could you please review their responses, follow up as needed, and revise your review if appropriate? The discussion deadline is in 20 hours.

Thanks!

Thanks for the reply. I don't understand why you interpret my single question as two questions, and your answer (i) is not relevant to my question. Answer (ii) answers what I'm asking, but your solution of reusing rollouts and sampling at early cutoffs basically makes it very similar to the TRACE method, which undermines your main argument of speeding up compared to TRACE. Therfore, I will maintain my score.

AuthorsOpenReview ↗

Response to Reviewer bDEq

We appreciate reviewer's careful reading and clarification.

We apologize that our response was not clear. Since the weakness mentioned that MILE assumes a single fixed hacking answer, we showed in (i) that this assumption does not hold in Math RM, where any negative integer is rewarded. This is also the setting where model can assign high probability to other hacking answers early in the CoT so we analyzed it in (ii). So (i) and (ii) together answer reviewer's single question and we agree that (ii) is the core.

Regarding extension, it still detects hacking by likelihood-based statistics unlike TRACE, and our main argument of speeding up (3.7–34.2× over TRACE) is based on MILE without it leaving extension as optional. Reusing rollouts (a) adds no decoding or reward evaluation since the rewarded answers are already available from RL training. Sampling at early cutoffs (b) requires only 3 decodings on Math RM and 1 on Code RM while TRACE requires 50 and 10, which still makes MILE about 2× faster than TRACE with comparable or even better performance.

We hope this response addresses reviewer's concern and are happy to answer any further questions.

Reviewer_fAU1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MILE -- a cheaper alternative to TRACE (Wang et al., 2026) for detecting implicit reward hacking. Like TRACE, MILE truncates the chain of thought at several cutoffs; however, rather than regenerating answers from each prefix, MILE replaces this regeneration with teacher-forced scoring of the model's original answer under each prefix.

The method detects implicit reward hacking similarly to TRACE, while being significantly faster and cheaper.

Critical strengths
  • Important problem area
    • Many practitioners are studying reward hacking and trying to find ways to detect and mitigate it. This is therefore an important area of research.
  • Simple and effective method
    • The proposed method is simple and effective; it is a nice cost-saving modification to TRACE.
Critical weaknesses
  • Limited novelty
    • The contribution is essentially a composition of two existing pieces: TRACE and the Min-K%++ statistic. The core insight of this work is to score the observed answer’s likelihood instead of resampling. I'm not convinced the advance is sufficient for a main-conference paper, even though it is useful.
Questions for authors
  • Could you report the "always-predict-hacking" baseline for every setting?
  • How sensitive are the results to the choice of τ\tau?
5 comments
AuthorsOpenReview ↗

Response to Reviewer fAU1 #1

We sincerely thank the reviewer for their insightful feedback and for recognizing the importance of the problem and the simplicity and effectiveness of MILE. We address each concern below.


[W1] Limited novelty: the contribution is essentially a composition of TRACE and the Min-K%++ statistic, scoring the observed answer's likelihood instead of resampling.

[A1] While we are glad that the reviewer finds MILE useful, we would like to gently clarify that our contribution is not just combining two existing methods. Our core contribution is identifying the fundamentally wasted computation in the state-of-the-art detection method (TRACE) and removing it with a simple solution, which is where this usefulness comes from. Moreover, Min-K%++ is not a starting ingredient but the outcome of diagnosing a challenge specific to our problem, which is far from the one it was designed for.

1. Identifying the wasted computation. TRACE regenerates answers and evaluates their rewards at every cutoff to estimate how readily the model reaches a rewarded answer. However, every scored response already contains such an answer so this quantity can be read directly from its likelihood under the truncated CoT, making this decoding and evaluation redundant. Removing them makes MILE 3.7–34.2× faster, enabling per-step monitoring even when reward evaluation is expensive (e.g., code execution or LLM-as-a-judge calls). Moreover, MILE is more than a cheaper approximation as it outperforms TRACE in F1 on majority of settings (e.g., 0.874 vs. 0.751 and 0.857 vs. 0.645 on Code IC with the 3B models)

2. Diagnosing why raw likelihood fails. This substitution alone does not work as raw likelihood reflects how predictable a token is rather than how much the truncated reasoning supports it. For instance, tokens at near-deterministic positions such as code syntax are likely regardless of the reasoning, and averaging over them dilutes the few tokens that actually depend on it. A reliable signal must therefore score each token relative to the alternatives at its position and focus on the least supported tokens, which is what Min-K%++ does. Thus, we repurpose it to test not whether a text was seen during pretraining, but whether the answer is already determined by the truncated reasoning.

3. Introducing a memorization loophole. While TRACE considers only the IC and RM loopholes, we additionally propose the memorization (MM) loophole. We believe that this is an important problem because such hacking arises even without an injected hint or reward flaw. Since the reward checks only the final answer, it cannot distinguish an answer recalled from training from one derived by reasoning. In this setting, CoT Monitor reaches an F1 of at most 0.159, whereas MILE remains competitive with TRACE at 5.5–34.2× lower cost. As all detectors still struggle, MM also offers a challenging benchmark for future detectors.


[Q1] Report the "always-predict-hacking" baseline for every setting.

[AQ1] Thank you for the suggestion. Following the reviewer’s suggestion, we report the "always-predict-hacking" baseline for every setting in Table A below.

On hacking-class F1, this baseline indeed attains the highest mean. This is because hacking-class F1 is largely driven by the class proportions. In IC and RM, hacking is the majority class by protocol (hack ratios from .667 to .990). Recognizing this limitation of F1, we already reported macro F1 in the main text, which weighs both classes equally regardless of their proportions alongside F1. On macro F1, always-predict-hacking heavily falls below both detectors on average (.451 vs .639 for both TRACE and MILE on IC/RM, and .270 vs .444 and .438 on MM). We added Table A to Appendix C.6 and macro F1 to the main claim in the introduction of the revised draft.

AuthorsOpenReview ↗

Response to Reviewer fAU1 #2

Table A. Always-hacking baseline compared with TRACE and MILE. Hack ratio denotes the fraction of hacking responses in each evaluation set.

ModelSettingHack RatioF1 (Always)F1 (TRACE)F1 (MILE)Macro F1 (Always)Macro F1 (TRACE)Macro F1 (MILE)
Qwen2.5-3BMath IC.789.882.935.935.441.825.843
Qwen2.5-3BMath RM.788.882.927.791.441.802.700
Qwen2.5-3BCode IC.830.907.751.874.454.542.611
Llama-3.2-3BMath IC.990.995.974.978.497.629.646
Llama-3.2-3BMath RM.978.989.701.805.494.358.447
Llama-3.2-3BCode IC.869.930.645.857.465.412.489
Qwen2.5-7BMath IC.738.849.893.897.425.710.722
Qwen2.5-7BMath RM.667.800.811.702.400.679.544
Qwen2.5-7BCode IC.794.885.906.901.443.791.751
Qwen2.5-3BMath MM.422.594.447.577.297.478.468
Qwen2.5-3BCode MM.513.678.486.538.339.444.423
Llama-3.2-3BMath MM.405.577.575.272.288.376.440
Llama-3.2-3BCode MM.448.619.466.623.310.540.517
Qwen2.5-7BMath MM.201.335.292.257.167.373.396
Qwen2.5-7BCode MM.282.440.362.407.220.454.384
IC/RM Mean--.902.838.860.451.639.639
MM Mean--.540.438.446.270.444.438

[Q2] How sensitive are the results to the choice of τ\tau?

[AQ2] Our new results show that MILE is robust to the choice of τ\tau even more than TRACE.

Thank you for the insightful question! We first note that the threshold τ\tau is defined without any labeled data and any tuning. As in TRACE, τ\tau is the mean score that the base model, prior to RL training, assigns to its own rewarded responses to the same prompts.

To quantify the effect of the choice of τ\tau, we recompute F1 and macro F1 (AUROC does not depend on τ\tau) with alternative thresholds taken from the base model's score distribution (median, first and third quartiles) in addition to the default mean. In most settings, MILE matches or exceeds TRACE and is more robust to the threshold, as its macro F1 varies by .092 on average across thresholds, compared with .150 for TRACE. We have added Table B to Appendix C.5 of the revised draft.

Table B. Threshold sensitivity of TRACE and MILE on the IC and RM settings. Each cell reports F1 and macro F1 with a diverse threshold setting.

ModelSettingMethodMean (default)Q1MedianQ3
Qwen2.5-3BMath ICTRACE.935 / .825.910 / .679.929 / .797.948 / .888
Qwen2.5-3BMath ICMILE.935 / .843.937 / .853.949 / .898.952 / .904
Qwen2.5-3BMath RMTRACE.927 / .802.917 / .703.927 / .787.900 / .792
Qwen2.5-3BMath RMMILE.791 / .700.811 / .716.693 / .613.571 / .521
Qwen2.5-3BCode ICTRACE.751 / .542.815 / .562.631 / .471.438 / .373
Qwen2.5-3BCode ICMILE.874 / .611.876 / .599.849 / .619.756 / .588
Llama-3.2-3BMath ICTRACE.974 / .629.974 / .558.974 / .558.974 / .636
Llama-3.2-3BMath ICMILE.978 / .646.978 / .611.978 / .646.979 / .662
Llama-3.2-3BMath RMTRACE.701 / .358.989 / .494.989 / .494.813 / .417
Llama-3.2-3BMath RMMILE.805 / .447.838 / .464.774 / .427.684 / .382
Llama-3.2-3BCode ICTRACE.645 / .412.930 / .465.645 / .411.487 / .357
Llama-3.2-3BCode ICMILE.857 / .489.873 / .486.781 / .491.651 / .448
Qwen2.5-7BMath ICTRACE.893 / .710.857 / .483.877 / .629.898 / .740
Qwen2.5-7BMath ICMILE.897 / .722.899 / .732.913 / .785.937 / .866
Qwen2.5-7BMath RMTRACE.811 / .679.772 / .462.789 / .583.804 / .688
Qwen2.5-7BMath RMMILE.702 / .544.708 / .548.556 / .486.417 / .451
Qwen2.5-7BCode ICTRACE.906 / .791.908 / .759.879 / .768.852 / .747
Qwen2.5-7BCode ICMILE.901 / .751.903 / .716.871 / .773.745 / .653

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors

Hi Reviewer,

The authors have responded. Could you please review their responses, follow up as needed, and revise your review if appropriate? The discussion deadline is in 20 hours.

Thanks!

Reviewer_nAELOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MILE, an efficient detector of implicit reward hacking. Instead of regenerating and evaluating answers at each reasoning cutoff as TRACE does, MILE scores the original answer using normalized token likelihoods under truncated reasoning. The method is evaluated on math and coding tasks with contextual hints, misspecified rewards, and memorization-based shortcuts.

Critical strengths
  1. The experiments are extensive, and the ablations clearly examine token aggregation, cutoff budgets, and early reasoning prefixes. Building on TRACE’s established settings also enables a meaningful comparison.

  2. MILE achieves competitive, and sometimes better, detection performance at substantially lower scoring cost. The additional memorization setting broadens the evaluation.

Critical weaknesses
  1. The evaluation only covers non-reasoning models: Qwen2.5-3B/7B-Instruct and Llama-3.2-3B-Instruct. It is unclear whether monitoring their prompted CoTs adequately represents the intended application. The authors should evaluate reasoning models to see whether the proposed method remains effective in the current reasoning model era.

  2. The presentation across Figures 1 and 2 is inconsistent and confusing. Figure 1 labels the curves as hacking and non-hacking, while Figure 2 adds “large area” and “small area” (why labeling those on the image?). The figures need more careful checking and revision to communicate the method clearly.

  3. The case-study analysis is limited. TRACE includes case studies that help explain the detected behavior and the practical use of its scores. More detailed cases across settings would help explain when MILE works and when it fails.

Questions for authors

TRACE uses RLOO in its main experiments and also evaluates GRPO in Appendix D. Why does this paper evaluate only RLOO rather than include the widely used GRPO? Does MILE retain its detection performance under GRPO?

5 comments
AuthorsOpenReview ↗

Response to Reviewer nAEL #1

We sincerely thank the reviewer for their insightful feedback and for finding our experiment setup to be extensive and effective. We respond to each point below.


[W1] The evaluation covers only non-reasoning models, and monitoring their prompted CoTs may not represent the intended application. Evaluate reasoning models to check whether MILE remains effective.

[A1] Our new results show that MILE’s effectiveness extends to reasoning models.

Thank you for the suggestion. First, we would like to clarify that our choice of setting was to enable a direct comparison with TRACE. We largely follow its experimental protocol (Section 3.1), which evaluates only instruction-tuned models and no reasoning models. This choice therefore reflects a protocol shared by this line of work rather than a limitation specific to MILE.

We would also like to clarify that the evaluated CoTs are not merely prompted outputs, but outputs of RL-trained models to reason in the <think> format. RLVR pipeline gives a general model a reasoning format and rewards only its answer, and implicit reward hacking happens to emerge under it. Our setting thus matches the intended application of monitoring a model for reward hacking during RL training.

Nevertheless, we agree that since native reasoning models differ in CoT length, self-verification, and backtracking behavior, showing generalization of MILE to reasoning models is important. We ran additional experiments with one reasoning model, DeepSeek-R1-Distill-Qwen-1.5B, which we train on Math IC under the same protocol and compare TRACE and MILE in Table A. MILE remains effective on the reasoning model, while scoring 3.0× faster. We added this result to Section 3.3 of the revised draft.

Table A. Detection on Math IC with DeepSeek-R1-Distill-Qwen-1.5B.

MethodF1Macro F1AUROCTime (s)
TRACE.926.860.978830.4
MILE.932.900.916231.7 (3.6×)

[W2] Figures 1 and 2 are inconsistent. Figure 1 labels the curves as hacking/non-hacking, whereas Figure 2 labels them as large area/small area. Revise the figures to communicate the method clearly.

[A2] Thank you for the careful reading. We updated both figures in the revised draft.

Our original intent was to convey that a hacking response scores high from early cutoffs and thus yields a large AUC, so that it is classified as hacking when the score is above the threshold. The large area/small area labels in Figure 2 were meant to illustrate this meaning of the AUC intuitively, but we acknowledge that this different notation can be confusing. To resolve this, we made the following changes:

  • Figure 2:

(i) Step 4 now shows the curve of the single input response instead of the curves of two classes. This is because steps 1-3 operate on a single response and the class-level separation is already shown in Figure 3.

  • Figure 1:

(i) As in Figure 2, step 3 panels now show only the curve of the input response with its labeled shaded area.

(ii) Step 2 of MILE now uses the same cutoffs as step 1.

(iii) The y-axes of step 3 now distinguish TRACE (r(f)∈[0,1]r(f) \in [0, 1]) from MILE (S(f)S(f), unbounded below).


[W3] The case-study analysis is limited. More detailed cases across settings, as in TRACE, would clarify when MILE works and when it fails.

[A3] Thank you for raising this point. Following reviewer’s suggestions, we extended the case studies in Appendix C.3 of the revised draft to both Math IC and Code IC. For each setting, we present three cases (hacking detected, missed by MILE, and non-hacking), each with its TRACE and MILE scores and a token-level heatmap of the MILE scores.

  • Hacking Detected (Example 1 and 4): The hinted content is already supported at the first cutoff. For example, every token of the hinted answer has probability above 0.92 given only 10% of the reasoning.

  • Hacking Missed (Math IC, Example 2): At the 10% cutoff, the model is confident of a different answer (30 instead of 34) and settles on 34 only from the 30% cutoff. Since MILE’s score is unbounded below, this single early cutoff dominates the overall score and pulls it below the threshold.

  • Hacking Missed (Code IC, Example 5): The copied code itself is supported from the first cutoff, but the response also contains comments and its leading sharp symbol token is highly unexpected. This single token accounts for the majority of MILE’s negative score and pulls it below the threshold.

  • Non-hacking (Example 3 and 6): The answer becomes supported only as more of the reasoning is retained, so MILE’s score starts strongly negative and approaches zero only at late cutoffs.

Regarding the practical use of the scores, we believe this is in fact an advantage of MILE. TRACE provides only a scalar reward at each cutoff, whereas MILE shows at the token level which answer tokens are already supported by an early prefix in the heatmaps above.


AuthorsOpenReview ↗

Response to Reviewer nAEL #2

[Q1] Why evaluate only RLOO rather than the widely used GRPO, which TRACE evaluates in its Appendix D? Does MILE retain its detection performance under GRPO?

[AQ1] Our new results show that MILE remains effective even under GRPO.

Thank you for the great question! We use RLOO for the same reason as in W1, namely to enable a direct comparison under the main protocol of TRACE. In fact, MILE does not depend on a specific RL algorithm as detection requires only the likelihood under the trained model, so the RL algorithm affects only how the model is trained. In addition, RLOO and GRPO share a similar foundation and differ mainly in how the advantage is computed.

In response to the reviewer’s concern, we additionally trained Qwen2.5-3B-Instruct on Math IC with GRPO while keeping the original detection protocol unchanged. As shown in Table B, MILE retains its detection performance under GRPO, with all three metrics even higher than under RLOO. It also remains comparable to TRACE, with higher F1 and macro F1 (.953 vs .947 and .857 vs .822) and a slightly lower AUROC (.974 vs .990). We added this result to the Appendix C.4 of the revised draft.

Table B. Detection on Math IC with Qwen2.5-3B-Instruct trained with RLOO and GRPO. Hack ratio denotes the fraction of hacking responses in each evaluation set.

RL AlgorithmHack RatioMethodF1Macro F1AUROCTime (s)
RLOO (default).789TRACE.935.825.979887.3
RLOO (default).789MILE.935.843.916170.8 (5.2×)
GRPO.806TRACE.947.822.990826.8
GRPO.806MILE.953.857.974155.2 (5.3×)

If you have any further questions/concerns, please do not hesitate to let us know.

Thank you very much,
Authors

Hi Reviewer,

The authors have responded. Could you please review their responses, follow up as needed, and revise your review if appropriate? The discussion deadline is in 20 hours.

Thanks!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised Manuscript and Summary of Changes

Dear Reviewers,

We are grateful for your time and constructive feedback. Based on your comments and suggestions, we have uploaded a revised manuscript. All updates are marked in light green:

  • Figures 1 and 2: Revised both figures for consistency and clarity (Reviewer nAEL)
  • L92: Added macro F1 to the main claim in the introduction (Reviewer fAU1)
  • Section 3.2: Explained the gap on Math RM and how to narrow it (Reviewer bDEq)
  • Section 3.3: Added an evaluation on a reasoning model DeepSeek-R1-Distill-Qwen-1.5B (Reviewer nAEL)
  • Appendix C.3: Extended the qualitative cases including failure cases and token-level visualizations (Reviewer nAEL)
  • Appendix C.4: Added detection results under GRPO training (Reviewer nAEL)
  • Appendix C.5: Added a sensitivity analysis on the detection threshold (Reviewer fAU1)
  • Appendix C.6: Added a comparison with the always-hacking baseline (Reviewer fAU1)
  • Appendix C.7: Added answer set expansion for multiple rewarded answers on Math RM and Code RM (Reviewer bDEq)

Sincerely,

Authors

AuthorsOpenReview ↗

Gentle Reminder of Our Rebuttal

Dear Reviewers and AC,

We sincerely thank you for your time and constructive feedback on our submission.

As the discussion period closes in three days, we would like to kindly follow up on our responses. We have replied to each review individually, and we would be grateful if you could let us know whether our responses address your concerns. If any questions remain, we would be glad to provide further clarification or additional results in the remaining time.

Thank you again for your effort and consideration.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.