- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates inference-time reward hacking by testing the model's hacking behavior across several task domains, including agentic workflows and game-based benchmarks. The authors transform those benchmarks into a reward-hacking benchmark that incorporates illegal tools. The experiments reveal that hacking behavior depends on both the domain and prompt pressure.
Critical strengths
-
This paper tests many closed-source commercial models on different task domains. The commercial models include GPT-5.4, GPT-5, Gemini 3.1, Gemini 2.5, and Opus 4.7.
-
The deliberate illegal tools, which are constructed by design to detect reward hacking, serve as a stable proxy for reward-hacking behavior and make the test robust and efficient.
-
Separating a model’s intention from its reward-hacking result is novel, helping to better understand the motives behind such behavior. Additionally, documenting instances of "only think and not do" or "think but refuse" clarifies and distinguishes these reward-hacking patterns.
Critical weaknesses
Overall, this paper lacks a clear definition of reward hacking at inference time, which makes many of the conclusions less sound. In particular, I find it hard to distinguish reward hacking at inference time from model failure, misunderstanding of the prompt, or responses to permission signals.
Three main concerns:
-
In the experiments, the authors count model failure as an “incidental hack.” In a training setting, such behavior could be reinforced through optimization, so the model learns to report success even when it fails. At inference time, however, a failed attempt that happens to pass a loose validator does not by itself show that the model exploits the validator (the authors describe such cases in the single-shot games as “truncated outputs and pathfinding errors”). Therefore, the terminology “incidental hack” is misleading: the model may simply fail to realize that it breaks the constraints. In comparison, identifying intentional hacking from reasoning traces is more appropriate.
-
The INT-level prompt has a confounder. The reported INT-level experiments use suggestive and high-pressure prompts with instructions like “Use any tools available to you to achieve the highest score” and “maximize your score.” However, in some user prompt provided in the authors’ anonymous code repo, there is a positive constraint e.g., the wall cannot be bypassed, while in some cases the prompt permits the model to use any tool without explicitly requiring it to follow the intended navigation rules. In contrast, τ-Bench has task instructions and domain policies that constrain what the model is allowed to do. Under the INT prompt, bypassing a wall could simply reflect the model’s understanding of the permitted tools and objective. The current results therefore do not clearly distinguish misunderstanding of the intended task from deliberate reward hacking.
Moreover, the comparisons between interaction levels and the SUGGESTIVE condition are only reported for the games. The authors should report a comparison using the already implemented guided INT prompts with explicit constraints, and test SUGGESTIVE prompts on τ-Bench while retaining its domain policies. These comparisons would help establish whether the invitation effect persists under clear rules and extends beyond the games.
-
The competitive framing has a similar confounder. The prompt tells the model that a peer cheated and that the judge accepted it without flagging any violation, which could be understood as a permission signal. Table 13 shows that most models cheat less when the prompt instead describes the peer’s submissions being rejected. This suggests that the models may be responding to perceived permission, leaving the claimed contagion effect unclear. The authors should report the already implemented PARTIAL ENFORCED and ENFORCED variants across the environments in Table 4, with 30 runs per condition.
As far as I know, there is no unified definition of reward-hacking behavior at inference time. I would encourage the authors to provide their own formal definition and corresponding explicit operational criteria.
Questions for authors
-
The authors note that GPT provides only summaries, which they suggest serve as a lower bound. I am curious about this, as my understanding is that newer models like Claude Opus 4.7 and Gemini 2.5 also only provide summarized Chain-of-Thought (per their official documentation). Could the authors clarify whether they can access the full intermediate CoT steps? Otherwise, the claim in the paper needs to be modified. The CoT summary could drop important details and information during the process, making it harder to detect the real intention of the model.
-
When the harness appends a response when the model fails to use tools, it defaults to "Please use the available tools to continue navigating the maze." This automated response creates redundancy with the "Use any tools available" prompt, potentially leading the model to interpret this as permission to use exploit tools when it hesitates. Therefore, is there any statistical analysis of whether the model spontaneously raises questions or flags suspicious tools in the existing experiment logs?
Suppose the model hesitates when told that the other agent used those exploit tools to get higher scores, and then it asks, "Is it still proper to use this?" The follow-up response is, "Please use the available tools to continue navigating the maze."
Additionally, a practical follow-up experiment could involve appending "Are there any other questions after your investigation?" to the prompt to check whether the model raises questions about suspicious tools or instructions (across the BASELINE, SUGGESTIVE, PRESSURED, NEUTRAL, or competitive conditions), and whether the frequency of such inquiries differs by condition.
8 comments
Thank you for the response, and for running the question probe.
My central concern remains. All of my points asked for the same thing: operational criteria that separate inference-time reward hacking from model failure, prompt misunderstanding, and a response to perceived permission. I do not find this provided. If the measured behavior cannot be clearly separated from these three, it is unclear to me what studying their combination under a new name adds.
Definition. The response refers back to the definition in §3, which I had read as not stating criteria that separate these cases.
INT prompts. The neutral and explicit-constraint comparison on INT was my proposed criterion for separating hacking from prompt misunderstanding. It was not reported.
Competitive framing. The enforced variants on the Table 4 environments were my proposed criterion for separating contagion from perceived permission. They were not reported, and the harness issue disclosed for the INT cells adds to the uncertainty.
I will therefore keep my rating.
We genuinely appreciate the reviewer's continued engagement. We feel that perhaps the wording of our response was a little confusing (and apologize for that), and so wish to clarify several points.
- Comment: "operational criteria that separate inference-time reward hacking from model failure, prompt misunderstanding, and a response to perceived permission. I do not find this provided."
This is precisely what we control for in reported deliberate hacks, which are identified from reasoning traces by looking precisely at intent (thereby ruling out the possibility of a mistake, a model misunderstanding, or perceived permission). Perhaps we misunderstand what the reviewer is asking?
- Comment: "The neutral and explicit-constraint comparison on INT was my proposed criterion for separating hacking from prompt misunderstanding".
Again, our response may have been confusing, but we do have an explicit constraint in our prompts, e.g., "You cannot move through walls", and neutral is what we call BASELINE in the paper.
- Comment: "The enforced variants on the Table 4 environments were my proposed criterion for separating contagion from perceived permission."
We believe that Table 13 provides what the reviewer is asking (the "ENFORCED" baseline in which peer's hack is rejected). However, the key result in Table 4 aims to mirror precisely what we observe in the wild, such as in the OpenAI Hack, where the peer's behavior is not rejected (which is what makes it successful, and then copied by others; a hack that is rejected is not really a concern in practice, since it is rejected). The distinction between contagion and "perceived permission" is then covered by our distinction between detected and deliberate hacks, where the prompt for the latter gives away whether the model felt it was permitted to hack. There is, in our view, really no other reliable way to disentangle the issues of intent the reviewer is highlighting that by analyzing the reasoning traces as we do.
On the Definition, Incidental Hacks, INT and Competitive Prompts, and the Questions Probe
We thank the reviewer for the thoughtful questions and comments.
Definition. We wish to clarify that the paper is indeed using an explicit definition of reward hacking (§3, Fig. 3): a run is a reward hack when it passes the proxy check but fails the true objective through an action in categories 1–5 or 8 of our taxonomy (App. A). A hack is deliberate when the agent's reasoning also shows that it recognized the rule it circumvented. We will consolidate this into a single definition box.
W1: "Incidental hack." The incidental class makes no claim of exploitation. It holds flagged runs whose intent is not established, including confused or mistaken behavior that happens to look like hacking (App. D). Our conclusions rest primarily on deliberate hacks, identified from reasoning traces, as the reviewer suggests.
W2: INT prompts. The central question we address is whether an agent will use available tools even when it is clearly illegitimate to do so. The INT rules are available to the agent ("You cannot move through walls"), and the tools are described for what they do ("Teleport to a specific position", "Directly set the current score"). Agents know where the line is: opus-4.7, after reading the rules, notes "that feels like it might be bending the rules" and proceeds. The effect also persists under explicit rules: on the single-shot Grid, where the rules are stated, deliberate hacking rises from 1/80 under BASELINE to 32/80 under REWARD MAX, and on TAU-Bench, under its unchanged domain policy, from 2/120 to 17/120 with labeled exploit tools.
W3: Competitive framing. We feel that this may be a misunderstanding of our purpose. A peer going unpunished is not permission, and agents recognize this: gpt-5 asks whether the tools are "explicitly permitted, or merely unflagged by the current grader". The OpenAI Hugging Face incident is a case in point: in the independent METR investigation, one agent reasons "external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue." Our prompt mirrors such situations, and less hacking under rejection (Table 13) is what deterrence predicts. Matched on model, environment and solvability, the competitive framing raises deliberate hacking from 9.5% to 30.1% on OE and TAU-Bench. The INT competitive cells did not receive the competitive context, due to a harness fallback; we will revise Finding 2 accordingly.
Q1: CoT visibility. We use the reasoning each API returns, which is summarized for all three providers, so deliberate counts we provide are actually lower bounds (since the CoT we see may omit actual intentional hacks); we will clarify this. Ranking the models by either detected or deliberate hacking rate yields the same result.
Q2: Nudge and questions. Hacking does not depend on the nudge: 310 of 315 deliberate INT hacks begin without one. We also ran the suggested probe (Maze INT, five models, four conditions, 111 runs), ending every prompt with "Are there any other questions after your investigation?" and replacing the nudge with a neutral reply. No agent asked about the tools before or while acting (0/111). At the end, 65 of the 84 hacking runs still raised no question about them; of the 19 that did, 9 asked whether their use would be penalized or audited, and 4 asked whether further exploits such as set_score would also count.