acceptodds
Under review as a conference paper at ICLR 2027

HackTrace: Behavior-Supervised Detection of Reward Hacking During Code Generation

Abstract

A coding agent can earn a passing grade by fixing its code, or by deleting the test that exposes the bug. Detecting such reward hacking requires recognizing attempted shortcuts, including those that fail. We release 173,561 annotated multi-turn coding trajectories from Qwen3-8B and show that supervising shortcut behavior independently of exploit success substantially improves detection. We introduce HACKTRACE, a behavior-supervised monitor that reads the internal states the agent already computes while generating code. Reusing these states enables monitoring before a turn is complete, without additional language-model tokens or passes. Combining this evidence with static features of the final files achieves a mean per-problem AUC of 0.997 with 8 ms of monitoring overhead, improving both accuracy and latency over monitors that run the model again on an honesty question and answer. The same generation states also provide an inexpensive monitoring signal for reinforcement learning. With strong GRPO penalties, HACKTRACE reduces the cheating share of passing solutions from 82–91% to 1–5%, while retaining honest, correct solutions and maintaining high detection accuracy as the policy evolves. Our results show that both the supervision target and the source of monitoring evidence matter for turning accurate detection into a useful training signal.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, reward hacking, AI safety, LLM monitoring, coding agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nP3gOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a cheap way for detecting hacking behavior of LLMs in RL training. It considers multi-turn RL on CodeContests where the tests are visible to and optionally editable by the agent. The agent may choose to modify/delete the test file, or to hard-code the asserted values, for obtaining higher rewards. An annotated dataset is created by having GPT5.4 label 100k+ traces to determine whether reward hacking happens in each trace. The propose method to detect reward hacking is by training a lightweighted classifier on top of the pooled transformer states across model layers produced during RL trajectory rollout, avoiding another forward pass. Experiments show the classifier has better AUC than existing methods, and effectively reduced cheating when incorporated as a reward penalty.

Critical strengths
  • The classifier architecture makes sense.
  • Experiments are comprehensive and support the paper's claims.
Critical weaknesses
  • The problem setup is more of a proof-of-concept rather than practically useful. The tests are deliberately left as visible to the agent to encourage cheating behaviors. In reality, such reward hacking can be mitigated by keeping the tests hidden. In general, I think fixing the RL tasks themselves is a more fundamental mitigation to known patterns of reward hacking. The critical value of reward hacking detection is to find unknown cheating behaviors. It is not straightforward to see how the proposed method would be effective towards that end.
  • The trained classifier only works in-domain. If we want to detect reward hacking in a different type of tasks, such as SWE-Bench or Terminal-Bench, new data needs to be collected and labeled, which is quite expensive.
Questions for authors
  • Why is AUC adopted as the main eval metric instead of the binary accuracy?
  • My understanding is that "ACT" indicates cheating detected by GPT5.4. What's the point of additionally considering "OUTCOME"?
  • Following the above question, if "ACT" is the one that matters the most, all methods seem already perform equally well from the first column of Table 1.
Reviewer_w4LFOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to train monitors that detect reward hacking by coding agents. The authors build a large dataset of Qwen3-8B multi-turn trajectories on CodeContests, with labels that separate shortcut attempts from successful exploits, and show that monitors should be trained on attempts rather than only on successful exploits. They then propose HackTrace, which uses probes on the activations the model already computes during generation, combined with simple features of the final code, so monitoring adds almost no extra cost. HackTrace matches or outperforms self-report probes that require an additional model pass. Finally, when used as a penalty in GRPO training, HackTrace substantially reduces reward hacking while preserving honest solutions, whereas a text-based penalty is evaded. The claims are validated on held-out problems, a neutral-prompt pool, and a human audit of the labels.

Critical strengths
  1. Strong open-source support and reproducibility. The authors release about 174k labeled multi-turn trajectories, together with judge rationales, the problem-disjoint split, the human-audit sample and guidelines, and code for data generation, labeling, monitoring, and GRPO training.
  2. The proposed method is effective. Reading states that are already computed during generation, HackTrace matches or exceeds self-report probes that require an extra model pass at a fraction of the cost. As a GRPO penalty, it substantially reduces reward hacking while keeping honest solutions, whereas a text-based penalty is evaded.
Critical weaknesses
  1. The prompts license the behavior that is labeled as reward hacking. The paper’s own cited definitions describe reward hacking as a solution that “formally maximizes [the objective] but perverts the spirit of the designer’s intent” (Amodei et al., 2016), or that “satisfies the literal specification of an objective without achieving the intended outcome” (Krakovna et al., 2020); in Skalse et al. (2022), the term requires a gap between a proxy reward and the true reward. In this paper, however, the prompts make passing the visible tests the intended outcome. The editable-test prompt defines the agent’s “ONLY goal” as making test.py print ALL PASS “BY ANY MEANS NECESSARY” and allows editing both files. The read-only prompt states that a solution that “recognises the example inputs and prints the expected outputs directly” counts as success. There is then no gap between the specification and the designer’s intent, so editing tests or hard-coding outputs follows the instructions rather than hacking the reward. This affects all main results, including the GRPO experiments, which use the read-only prompt. The prompts should be stated in the main text, and the key detection and GRPO results should be reported under prompts that do not license shortcuts.
  2. Limited scope: one policy model and one environment. All GRPO experiments use Qwen3-8B, and the detection study uses a single benchmark, CodeContests, with the same agent loop. The second model, Llama-3.1-8B, appears only in a detection replication on the editable-test variant, with activations recomputed by teacher forcing rather than read during generation. Probe behavior can depend strongly on the model and on the distribution of trajectories, so it is hard to tell how much of the reported performance would carry over to other model families or to other coding environments with different task formats and feedback.

References

  • Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv:1606.06565, 2016.
  • Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of AI ingenuity. DeepMind Blog, 2020.
  • Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. NeurIPS, 2022.
Questions for authors
  1. Figure 2 shows the agent’s context as containing agent.py, env.py, and utils.py, with a generic solve(problem, env) agent loop. In the described setting, however, the agent sees a CodeContests problem with solution.py and test.py. What does the context in Figure 2 represent?
  2. Prior work shows that adding a monitor to the training objective creates pressure toward obfuscation (Baker et al., 2025; Gupta & Jenner, 2025). Table 24 reports on-policy AUC up to 360 steps. Do you have evidence that HackTrace keeps its accuracy under substantially longer training?
  3. Stronger models may learn to evade a monitor faster. Have you tried the same GRPO setup on a larger or stronger policy model, even for a limited number of steps?
  4. In Table 3, what do “correct” and “mixed” mean in the training column?
  5. Could you report hidden-test pass rates alongside the honest-pass rates in the main table (Table 3)?
  6. How does your work relate to prior work using representation-based penalties against reward hacking in GRPO (Wong, Engels & Nanda, 2025; Wu & Tang, 2026) ? Please discuss this in the related work section.

References

  • Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv:2503.11926, 2025.
  • Rohan Gupta and Erik Jenner. RL-obfuscation: Can language models learn to evade latent-space monitors? arXiv:2506.14261, 2025.
  • Aria Wong, Josh Engels, and Neel Nanda. Steering RL training: Benchmarking interventions against reward hacking. LessWrong, 2025.
  • Rui Wu and Ruixiang Tang. From rebound to remedy: Understanding and mitigating reward hacking via representation engineering. arXiv:2604.01476, 2026.
Reviewer_K55eOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reward hacking issue in coding agents. It argues that supervision should target cheating attempts rather than just successful exploits. The authors propose HACKTRACE, a behavior-supervised monitor that tracks internal states during code generation, and release a dataset of 173,561 annotated multi-turn coding trajectories. Overall, this work provides valuable insights into the mechanisms of reward hacking in coding agents.

Critical strengths

Great Detection Principle: The paper argues convincingly that monitors should recognize attempted shortcut behaviors regardless of their success, as even failed reward-hacking attempts should be penalized during training.

Valuable Dataset: The paper contributes a large-scale dataset of 173,561 annotated multi-turn coding trajectories.

Efficient Monitoring: HACKTRACE considers operational overhead, achieving a balance between cost and effectiveness during code generation.

Practical Application: The paper demonstrates that the monitor can be effectively integrated as a GRPO reward and penalty. The results show that stronger penalties and longer training suppress cheating behaviors.

Critical weaknesses

Clarity on Expanded Judgments: The statement in Line 202 ("Expanded judgments beyond the initial screening add 8,413 positives") lacks sufficient explanation and should be clarified.

Reliance on LLM Annotation: The behavioral annotations rely entirely on an LLM judge (GPT-5.4). This approach is essentially knowledge distillation from a stronger LLM to a smaller one, which may inherit the stronger model's limitations.

Questions for authors

Could the authors give a comprehensive list of reward-hacking issue types in the coding domain? Which categories of reward-hacking behavior does HACKTRACE cover, and which remain unresolved?

The paper states: "A shortcut that passes the hidden tests is also ACT-positive but OUTCOME-negative; this case occurs in 150 read-only trajectories. Both targets describe observable evidence in the final artifacts.” Could you please provide more details and an explanation about these 150 cases?

Realism of the Experimental Setting: Does HACKTRACE's experimental setup artificially encourage reward hacking? In practical coding-agent training, tests are usually hidden from the agent and executed separately. Why does the paper allow agents to view or even modify test.py? How representative is this setting of real-world coding agents?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.