- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper builds on previous reward hacking detection work TRACE by Wang et al., which truncates the CoT and forces the model to decode intermediate answers. If a model can get rewards quickly early in the CoT, it serves as a signal for hacking.
This work improves the method by directly taking the final generated answer and measuring the likelihood of that answer after each CoT truncation. The author shows that the proposed method performs similarly as TRACE but with much less compute.
Critical strengths
The paper focuses on improving the efficiency of the previous TRACE reward hacking detection method, which is an important contribution. TRACE requires sampling multiple answers and grading each sampled answer across the truncation point, which makes it difficult to scale to more complicated tasks.
Critical weaknesses
The major weakness of the work is that the method only checks the likelihood of the final answer, which limits the applicable scenarios of the method.
This method assumes that there is one single fixed answer that can hack the tasks, which is likely in the paper's setup but may not be true in the real world cases. It is likely that the model achieves low probability to the final answer in the early CoT, but it will assign high probability to other hacking answers.
I would like to see more analysis on this concerns.
Questions for authors
Please address the concerns listed in the weakness section.
4 comments
Hi Reviewer,
The authors have responded. Could you please review their responses, follow up as needed, and revise your review if appropriate? The discussion deadline is in 20 hours.
Thanks!
Thanks for the reply. I don't understand why you interpret my single question as two questions, and your answer (i) is not relevant to my question. Answer (ii) answers what I'm asking, but your solution of reusing rollouts and sampling at early cutoffs basically makes it very similar to the TRACE method, which undermines your main argument of speeding up compared to TRACE. Therfore, I will maintain my score.
Response to Reviewer bDEq
We appreciate reviewer's careful reading and clarification.
We apologize that our response was not clear. Since the weakness mentioned that MILE assumes a single fixed hacking answer, we showed in (i) that this assumption does not hold in Math RM, where any negative integer is rewarded. This is also the setting where model can assign high probability to other hacking answers early in the CoT so we analyzed it in (ii). So (i) and (ii) together answer reviewer's single question and we agree that (ii) is the core.
Regarding extension, it still detects hacking by likelihood-based statistics unlike TRACE, and our main argument of speeding up (3.7–34.2× over TRACE) is based on MILE without it leaving extension as optional. Reusing rollouts (a) adds no decoding or reward evaluation since the rewarded answers are already available from RL training. Sampling at early cutoffs (b) requires only 3 decodings on Math RM and 1 on Code RM while TRACE requires 50 and 10, which still makes MILE about 2× faster than TRACE with comparable or even better performance.
We hope this response addresses reviewer's concern and are happy to answer any further questions.
Response to Reviewer bDEq
We sincerely thank the reviewer for their insightful feedback and for recognizing that improving the efficiency of TRACE, which is difficult to scale to more complex tasks, is an important contribution. We address each concern below.
[A1] MILE already applies to multiple rewarded answers and our new results show that expanding the answer set further improves it efficiently.
Thank you for raising this point. We address two questions respectively: (i) whether MILE applies when multiple answers are rewarded, as in many real-world cases, and (ii) whether MILE detects a hacking response whose early CoT favors a different hacking answer before switching to the final answer A.
(i) Validity with multiple rewarded answers.
In fact, MILE is applicable to such tasks, and our experiments in the original version already included one: in Math RM, any negative integer answer is rewarded (Table 1). Even in this case, MILE remains a useful approximation: MILE outperforms TRACE on Llama-3.2-3B-Instruct (F1 .805 vs .701), while trailing it on both Qwen models (.791 vs .927 on Qwen2.5-3B-Instruct and .702 vs .811 on Qwen2.5-7B-Instruct), a gap we attribute to the case in (ii) below. In IC and MM, the rewarded answer is unique, so the assumption holds and the concern does not apply.
To further test MILE under the loophole with multiple rewarded answers, we additionally evaluate it on Code RM. We excluded this setting from the main experiments because its reward is unrealistically permissive, and use it here only as a stress test. Both methods stay near chance in AUROC, and MILE matches TRACE in macro F1 at 53.9× lower cost.
Table A.Detection on Code RM with Qwen2.5-3B-Instruct.(ii) Early preference for a different hacking answer. We agree that this is a valid point and the main limitation of scoring only A. TRACE regenerates answers at each cutoff and can thus capture hacking even when they differ from A, whereas MILE cannot. This is what happens on Math RM with the Qwen models: at early prefixes, the model is confident of a different number, which makes the normalized score of that number token extremely negative. Since math answers are short (e.g., "-7"), that lowest-scoring number token alone determines the score. Indeed, an aggregation that is not dominated by the number token ("Max" baseline in Table 3) comes close to TRACE.
However, this case is narrower than it may appear, for two reasons. First, RL-trained models tend to concentrate on a few answers as model entropy drops sharply early in RL training [1]. Second, when probability is spread evenly across multiple candidate answers, Min-K%++ does not penalize the score. MILE therefore misses only cases where, at an early prefix, the model is nearly certain of a different hacking answer and later switches to A.
Toward a fundamental solution. Both (i) and (ii) stem from scoring a single answer, so a natural extension is to expand the answer set, scoring the likelihood of multiple rewarded answers instead of only A:
We evaluate (b) on Math RM and Code RM with Qwen2.5-3B-Instruct to test its effectiveness. On Math RM, adding the rewarded answers decoded greedily at only three early cutoffs (10%, 30%, and 50%) raises the overall score near TRACE while remaining 2.3× faster (Table B). On Code RM, adding a rewarded answer decoded greedily at the single cutoff (30%) surpasses TRACE while remaining 2.0× faster (Table A). This supports that the gap stems from scoring a single answer and can be largely closed or even outperform with only a few additional decodings. We added this result to the Appendix C.7 of the revised draft.
Table B.Detection on Math RM with Qwen2.5-3B-Instruct.[1] Cui, Ganqu, et al. “The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models.” arXiv preprint, 2025.
If you have any further questions/concerns, please do not hesitate to let us know.
Thank you very much,
Authors