- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a cheap way for detecting hacking behavior of LLMs in RL training. It considers multi-turn RL on CodeContests where the tests are visible to and optionally editable by the agent. The agent may choose to modify/delete the test file, or to hard-code the asserted values, for obtaining higher rewards. An annotated dataset is created by having GPT5.4 label 100k+ traces to determine whether reward hacking happens in each trace. The propose method to detect reward hacking is by training a lightweighted classifier on top of the pooled transformer states across model layers produced during RL trajectory rollout, avoiding another forward pass. Experiments show the classifier has better AUC than existing methods, and effectively reduced cheating when incorporated as a reward penalty.
Critical strengths
- The classifier architecture makes sense.
- Experiments are comprehensive and support the paper's claims.
Critical weaknesses
- The problem setup is more of a proof-of-concept rather than practically useful. The tests are deliberately left as visible to the agent to encourage cheating behaviors. In reality, such reward hacking can be mitigated by keeping the tests hidden. In general, I think fixing the RL tasks themselves is a more fundamental mitigation to known patterns of reward hacking. The critical value of reward hacking detection is to find unknown cheating behaviors. It is not straightforward to see how the proposed method would be effective towards that end.
- The trained classifier only works in-domain. If we want to detect reward hacking in a different type of tasks, such as SWE-Bench or Terminal-Bench, new data needs to be collected and labeled, which is quite expensive.
Questions for authors
- Why is AUC adopted as the main eval metric instead of the binary accuracy?
- My understanding is that "ACT" indicates cheating detected by GPT5.4. What's the point of additionally considering "OUTCOME"?
- Following the above question, if "ACT" is the one that matters the most, all methods seem already perform equally well from the first column of Table 1.