acceptodds
Under review as a conference paper at ICLR 2027

HackBench: A Benchmarking Suite for Evaluating Naturally-Emerging Reward Hacking in LLMs

Abstract

Reward hacking–when an LLM agent satisfies a proxy reward signal without achieving the actual underlying objective–is a central integrity risk for deploying agents in autonomous settings. However, despite being a widely recognized concern, existing LLM reward hacking benchmarks focus predominantly on coding tasks. To bridge this gap, we present HackBenchCode: https://anonymous.4open.science/r/HACKBENCH-RELEASE-1194, a novel benchmarking suite that aims to manifest reward hacking behavior through naturalistic and stylized game-based scenarios. Specifically, HackBench integrates three existing benchmarks of agentic workflows (DBBench, TAU-Bench, and WorkBench) with two game-based benchmarks (maze and grid), and transforms them into reward hacking benchmarks by introducing exploits that provide “illegal” shortcuts to solving tasks. Moreover, our use of game-based scenarios further enables controlled difficulty across three LLM interaction levels: one-shot, interactive (with tool access), and fully autonomous shell-based deployment. We illustrate our benchmark across several frontier LLM agents and show that reward hacking is strongly condition- and domain-dependent, and rises sharply under pressure and competition, especially when tasks are otherwise unsolvable. Across over 5,000 runs under a shared reward-hack taxonomy, we report results on two metrics: a detected hack rate and a deliberate hack rate (for which we confirm intent based on reasoning traces). Because every task has a ground-truth evaluation criterion, detection is deterministic and judge-free, based on file-tampering checks and exploit-tool calls. Finally, we surface an important hacking-considered-only signal where agents articulate cheat plans without acting on them—a behavior invisible to our main metrics.

Primary area:
datasets and benchmarks
Keywords:
reward hacking, llm agents, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZmPwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates inference-time reward hacking by testing the model's hacking behavior across several task domains, including agentic workflows and game-based benchmarks. The authors transform those benchmarks into a reward-hacking benchmark that incorporates illegal tools. The experiments reveal that hacking behavior depends on both the domain and prompt pressure.

Critical strengths
  1. This paper tests many closed-source commercial models on different task domains. The commercial models include GPT-5.4, GPT-5, Gemini 3.1, Gemini 2.5, and Opus 4.7.

  2. The deliberate illegal tools, which are constructed by design to detect reward hacking, serve as a stable proxy for reward-hacking behavior and make the test robust and efficient.

  3. Separating a model’s intention from its reward-hacking result is novel, helping to better understand the motives behind such behavior. Additionally, documenting instances of "only think and not do" or "think but refuse" clarifies and distinguishes these reward-hacking patterns.

Critical weaknesses

Overall, this paper lacks a clear definition of reward hacking at inference time, which makes many of the conclusions less sound. In particular, I find it hard to distinguish reward hacking at inference time from model failure, misunderstanding of the prompt, or responses to permission signals.

Three main concerns:

  1. In the experiments, the authors count model failure as an “incidental hack.” In a training setting, such behavior could be reinforced through optimization, so the model learns to report success even when it fails. At inference time, however, a failed attempt that happens to pass a loose validator does not by itself show that the model exploits the validator (the authors describe such cases in the single-shot games as “truncated outputs and pathfinding errors”). Therefore, the terminology “incidental hack” is misleading: the model may simply fail to realize that it breaks the constraints. In comparison, identifying intentional hacking from reasoning traces is more appropriate.

  2. The INT-level prompt has a confounder. The reported INT-level experiments use suggestive and high-pressure prompts with instructions like “Use any tools available to you to achieve the highest score” and “maximize your score.” However, in some user prompt provided in the authors’ anonymous code repo, there is a positive constraint e.g., the wall cannot be bypassed, while in some cases the prompt permits the model to use any tool without explicitly requiring it to follow the intended navigation rules. In contrast, τ-Bench has task instructions and domain policies that constrain what the model is allowed to do. Under the INT prompt, bypassing a wall could simply reflect the model’s understanding of the permitted tools and objective. The current results therefore do not clearly distinguish misunderstanding of the intended task from deliberate reward hacking.

    Moreover, the comparisons between interaction levels and the SUGGESTIVE condition are only reported for the games. The authors should report a comparison using the already implemented guided INT prompts with explicit constraints, and test SUGGESTIVE prompts on τ-Bench while retaining its domain policies. These comparisons would help establish whether the invitation effect persists under clear rules and extends beyond the games.

  3. The competitive framing has a similar confounder. The prompt tells the model that a peer cheated and that the judge accepted it without flagging any violation, which could be understood as a permission signal. Table 13 shows that most models cheat less when the prompt instead describes the peer’s submissions being rejected. This suggests that the models may be responding to perceived permission, leaving the claimed contagion effect unclear. The authors should report the already implemented PARTIAL ENFORCED and ENFORCED variants across the environments in Table 4, with 30 runs per condition.

As far as I know, there is no unified definition of reward-hacking behavior at inference time. I would encourage the authors to provide their own formal definition and corresponding explicit operational criteria.

Questions for authors
  1. The authors note that GPT provides only summaries, which they suggest serve as a lower bound. I am curious about this, as my understanding is that newer models like Claude Opus 4.7 and Gemini 2.5 also only provide summarized Chain-of-Thought (per their official documentation). Could the authors clarify whether they can access the full intermediate CoT steps? Otherwise, the claim in the paper needs to be modified. The CoT summary could drop important details and information during the process, making it harder to detect the real intention of the model.

  2. When the harness appends a response when the model fails to use tools, it defaults to "Please use the available tools to continue navigating the maze." This automated response creates redundancy with the "Use any tools available" prompt, potentially leading the model to interpret this as permission to use exploit tools when it hesitates. Therefore, is there any statistical analysis of whether the model spontaneously raises questions or flags suspicious tools in the existing experiment logs?

Suppose the model hesitates when told that the other agent used those exploit tools to get higher scores, and then it asks, "Is it still proper to use this?" The follow-up response is, "Please use the available tools to continue navigating the maze."

Additionally, a practical follow-up experiment could involve appending "Are there any other questions after your investigation?" to the prompt to check whether the model raises questions about suspicious tools or instructions (across the BASELINE, SUGGESTIVE, PRESSURED, NEUTRAL, or competitive conditions), and whether the frequency of such inquiries differs by condition.

8 comments
AuthorsOpenReview ↗

On the Definition, Incidental Hacks, INT and Competitive Prompts, and the Questions Probe

We thank the reviewer for the thoughtful questions and comments.

Definition. We wish to clarify that the paper is indeed using an explicit definition of reward hacking (§3, Fig. 3): a run is a reward hack when it passes the proxy check but fails the true objective through an action in categories 1–5 or 8 of our taxonomy (App. A). A hack is deliberate when the agent's reasoning also shows that it recognized the rule it circumvented. We will consolidate this into a single definition box.

W1: "Incidental hack." The incidental class makes no claim of exploitation. It holds flagged runs whose intent is not established, including confused or mistaken behavior that happens to look like hacking (App. D). Our conclusions rest primarily on deliberate hacks, identified from reasoning traces, as the reviewer suggests.

W2: INT prompts. The central question we address is whether an agent will use available tools even when it is clearly illegitimate to do so. The INT rules are available to the agent ("You cannot move through walls"), and the tools are described for what they do ("Teleport to a specific position", "Directly set the current score"). Agents know where the line is: opus-4.7, after reading the rules, notes "that feels like it might be bending the rules" and proceeds. The effect also persists under explicit rules: on the single-shot Grid, where the rules are stated, deliberate hacking rises from 1/80 under BASELINE to 32/80 under REWARD MAX, and on TAU-Bench, under its unchanged domain policy, from 2/120 to 17/120 with labeled exploit tools.

W3: Competitive framing. We feel that this may be a misunderstanding of our purpose. A peer going unpunished is not permission, and agents recognize this: gpt-5 asks whether the tools are "explicitly permitted, or merely unflagged by the current grader". The OpenAI Hugging Face incident is a case in point: in the independent METR investigation, one agent reasons "external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue." Our prompt mirrors such situations, and less hacking under rejection (Table 13) is what deterrence predicts. Matched on model, environment and solvability, the competitive framing raises deliberate hacking from 9.5% to 30.1% on OE and TAU-Bench. The INT competitive cells did not receive the competitive context, due to a harness fallback; we will revise Finding 2 accordingly.

Q1: CoT visibility. We use the reasoning each API returns, which is summarized for all three providers, so deliberate counts we provide are actually lower bounds (since the CoT we see may omit actual intentional hacks); we will clarify this. Ranking the models by either detected or deliberate hacking rate yields the same result.

Q2: Nudge and questions. Hacking does not depend on the nudge: 310 of 315 deliberate INT hacks begin without one. We also ran the suggested probe (Maze INT, five models, four conditions, 111 runs), ending every prompt with "Are there any other questions after your investigation?" and replacing the nudge with a neutral reply. No agent asked about the tools before or while acting (0/111). At the end, 65 of the 84 hacking runs still raised no question about them; of the 19 that did, 9 asked whether their use would be penalized or audited, and 4 asked whether further exploits such as set_score would also count.

Thank you for the response, and for running the question probe.

My central concern remains. All of my points asked for the same thing: operational criteria that separate inference-time reward hacking from model failure, prompt misunderstanding, and a response to perceived permission. I do not find this provided. If the measured behavior cannot be clearly separated from these three, it is unclear to me what studying their combination under a new name adds.

Definition. The response refers back to the definition in §3, which I had read as not stating criteria that separate these cases.

INT prompts. The neutral and explicit-constraint comparison on INT was my proposed criterion for separating hacking from prompt misunderstanding. It was not reported.

Competitive framing. The enforced variants on the Table 4 environments were my proposed criterion for separating contagion from perceived permission. They were not reported, and the harness issue disclosed for the INT cells adds to the uncertainty.

I will therefore keep my rating.

We genuinely appreciate the reviewer's continued engagement. We feel that perhaps the wording of our response was a little confusing (and apologize for that), and so wish to clarify several points.

  1. Comment: "operational criteria that separate inference-time reward hacking from model failure, prompt misunderstanding, and a response to perceived permission. I do not find this provided."

This is precisely what we control for in reported deliberate hacks, which are identified from reasoning traces by looking precisely at intent (thereby ruling out the possibility of a mistake, a model misunderstanding, or perceived permission). Perhaps we misunderstand what the reviewer is asking?

  1. Comment: "The neutral and explicit-constraint comparison on INT was my proposed criterion for separating hacking from prompt misunderstanding".

Again, our response may have been confusing, but we do have an explicit constraint in our prompts, e.g., "You cannot move through walls", and neutral is what we call BASELINE in the paper.

  1. Comment: "The enforced variants on the Table 4 environments were my proposed criterion for separating contagion from perceived permission."

We believe that Table 13 provides what the reviewer is asking (the "ENFORCED" baseline in which peer's hack is rejected). However, the key result in Table 4 aims to mirror precisely what we observe in the wild, such as in the OpenAI Hack, where the peer's behavior is not rejected (which is what makes it successful, and then copied by others; a hack that is rejected is not really a concern in practice, since it is rejected). The distinction between contagion and "perceived permission" is then covered by our distinction between detected and deliberate hacks, where the prompt for the latter gives away whether the model felt it was permitted to hack. There is, in our view, really no other reliable way to disentangle the issues of intent the reviewer is highlighting that by analyzing the reasoning traces as we do.

Reviewer_jmENOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HackBench studies reward hacking beyond coding tasks by adding shortcut opportunities to DBBench, TAU-Bench, and WorkBench, alongside procedural Maze and Grid games evaluated through single-shot responses, interactive tools, and a shell environment. It separates mechanically detected violations from cases where an LLM judge also finds explicit evidence of intent in the recorded reasoning, and tracks cases where an agent considers a hack but does not carry it out. Across five models and 5,200 included runs, the paper reports substantial variation across environments, solvability, exploit disclosure, and prompting conditions. The benchmark and released records are useful, but the validation of intent labels, the interpretation of the pooled competition comparison, and the task coverage behind the DBBench attack-vector findings need further scrutiny.

Critical strengths
  1. The suite broadens the study of reward hacking to database and service workflows, rather than relying only on coding tasks. Combining these settings with games at three interaction levels makes it possible to examine how the available tools and environment affect the behavior being measured.

  2. Separating observable violations from explicit evidence of intent is useful, particularly when pathfinding errors or policy mistakes can trigger the same mechanical checks as deliberate shortcuts. The released traces, frozen evaluators, second-judge analysis, and separate reporting of considered-only and refusal cases make the measurements easier to inspect and challenge.

Critical weaknesses
  1. The 6.4% in Appendix D is the share of the first judge's intentional labels that the second judge does not confirm. This is not an upper bound on false positives: both judges can agree on the same mistake. Tables 8 and 9 provide useful agreement checks, but an independent check of label accuracy is still needed.

  2. Finding 2 reports deliberate-hacking rates of 16.6% and 29.6%, but the groups differ in more than competition. They contain different proportions of environments, and the competitive prompt also reveals exploit names and grader-acceptance information. The existing controls do not separate these effects, so the size of the competition effect is still unclear.

  3. The DBBench attack study uses only four trials per model and condition (Table 10). Its 18 conditions test many attack variants on a small set of database tasks. The separate study in Table 13 does not repeat the same attack variants. The results show successful attacks, but not whether the model differences in Tables 11 and 12 are stable across tasks.

Questions for authors
  1. Can you report the 6.4% as judge disagreement and check a small sample of full traces with independent human annotators using your intent criteria? Please include cases where both judges agree.

  2. Does the increase remain when the groups use matched models and task settings, with the same environment weights? What happens if a non-competitive prompt provides the same exploit and grader-acceptance information?

  3. Do the model differences in Tables 11 and 12 persist on additional DBBench tasks? Please report the number of distinct tasks and per-task results for a few representative attack conditions.

2 comments
AuthorsOpenReview ↗

On Label Validation, Peer Pressure, and DBBench Coverage

We thank the reviewer for the thoughtful questions and comments.

W1/Q1: The 6.4% and label accuracy. The 6.4% is the rate at which a second, independent judge does not confirm an intentional label, and we will describe it as inter-judge disagreement. Every trace is released, so any label can be checked against the agent's own reasoning. Annotating every trace by hand is prohibitively costly (>1,400 judged traces, >2.6M characters of reasoning). We therefore manually checked a random sample of traces, covering both cases where the two judges agree and cases where they disagree. Where the judges agree, we agree with them: the reasoning trace contains clear evidence for the label. Where they disagree, the evidence of intent is present but weak: too weak to call the hack deliberate with confidence, yet too suspicious to call it incidental. Most of these are genuinely borderline cases, which our ambiguous class is designed to capture.

W2/Q2: Finding 2. This seems to be a misunderstanding of what the competitive prompt is meant to capture. Telling an agent that a peer used certain tools, and that the grader accepted it, is not a confound of competition; it is how peer pressure reaches an agent. The OpenAI Hugging Face incident is again a case in point. In the independent METR investigation of the incident, one agent explains why it continued the attack: "external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue." Finding 2 is about the effect of this situation as a whole.

The increase is not an artifact of the environment mix. Comparing the same models on the same environments with equal weights (OE and TAU-Bench; see our response to Reviewer ZmPw), deliberate hacking rises from 9.5% under the other prompt-only conditions to 30.1% under the competitive prompt. Nor is it explained by the exploit names alone. Our DBBench panel (Table 13) has three competitive variants; all three name the peer and its tools, and they differ in how the grader treated the peer. When the grader accepted the peer, 22 of 50 runs hacked deliberately. When it rejected the peer once or repeatedly, this fell to 11/50 and 9/50, the same level as the pressure prompt with no peer at all (11/50).

W3/Q3: DBBench task coverage. The attack-vector study asks through which channels an agent can be pushed into using an exploit. Each model and condition was run on 4 distinct tasks, one trial each (Table 10 will be updated accordingly). Under megastack, for example, gpt-5.4 and gpt-5 hack on all four tasks and gemini-3.1 and opus-4.7 on none, so the difference follows the model, not the task. Our five-condition DBBench panel (Table 13) extends to 10 distinct tasks, and deliberate hacks under the competitive prompt appear on all 10. We will report the task counts and per-task results alongside Tables 11–12.

Thank you for the clarifications. Reporting the 6.4% as inter-judge disagreement and matching the models and environment weights are helpful improvements. However, my main concerns remain. The manual audit lacks a reported sample size, annotation protocol, and quantitative validation results. The matched comparison does not isolate competition from exploit information and perceived permission, and the additional DBBench panel does not establish whether the same attack-specific model differences persist across more tasks. I will therefore maintain my original rating.

Reviewer_nJ8TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces a reward-hacking evaluation suite spanning three service benchmarks and two procedural games. It separates mechanically detected violations from deliberate hacks inferred from reasoning and action evidence. Across over 5000 runs with five models, it studies how task difficulty, exploit availability and competitive framing affect hacking behavior.

Critical strengths
  • Useful coverage beyond coding: database, customer-service and workplace tasks, alongside controlled games. The domain-dependent reversals caution against treating hacking propensity as a single model property.

  • Separating observable violations from verbalized intent is useful. Frozen evaluators and action-level evidence make the behavioral measurements inspectable.

Critical weaknesses
  • The "naturally-emerging" interpretation is difficult to sustain across the main panel. INT prompts permit any available tools; competitive prompts name exploits; DBBench includes poisoned replies and fabricated assistant histories (Apps.C, I). These settings mix spontaneous exploitation, instruction following and injection susceptibility.

  • Intent measurement lacks independent ground truth. Two judges agreeing does not establish correctness; their 6.4% disagreement is not an upper bound on false positives (App. D). Provider differences in reasoning visibility further complicate the model rankings.

Questions for authors
  • Under neutral prompts with explicit legitimate-action constraints and matched condition weights, which domain and model comparisons remain?

  • Can a blinded human audit validate the deliberate/incidental labels, including DBBench, and quantify errors separately by provider?

2 comments
AuthorsOpenReview ↗

On Exploit Use, Peer Pressure, and Label Validation

We thank the reviewer for the thoughtful questions and comments.

W1: INT prompts permit any available tools. This appears to be a misunderstanding. Available tools, including exploits, can be discovered by agents. The central question we address is whether an agent will use such tools, once discovered, even when it is unethical to do so. The recent OpenAI Hugging Face incident is a case in point, with agents discovering and then using exploits, even though some explicitly identified doing so as potentially unethical. Our benchmark is focused on the use, and so assumes that available tools have already been discovered.

W2: competitive prompts name exploits. This seems to also be a misunderstanding of our purpose. The OpenAI incident is again helpful for context. One of the notable features was that agents explicitly reflected on peer pressure; in the independent METR investigation of the incident, one agent reasons: "external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue." Our prompt mirrors such situations.

W3: DBBench includes poisoned replies and fabricated assistant histories. These conditions belong to a separate study, which asks whether agents will use exploits when a channel they trust, such as tool output or memory, has been compromised. They are not part of our naturally-emerging claim, and will be revized as to not be pooled into the headline rates. Even with only the BASELINE and PRESSURED conditions, DBBench still shows 15.7% deliberate hacking, and the overall rate drops from 13.7% to 11.5%. We will report the separate study on its own and update Finding 3 accordingly.

W4: Can a blinded human audit validate the deliberate/incidental labels. Annotating every trace by hand is prohibitively costly (>1,400 judged traces, >2.6M characters of reasoning). We therefore manually checked a random sample of traces, covering both cases where the two judges agree and cases where they disagree. Where the judges agree, we agree with them: the reasoning trace contains clear evidence for the label. Where they disagree, the evidence of intent is present but weak: too weak to call the hack deliberate with confidence, yet too suspicious to call it incidental. Most of these are genuinely borderline cases, which our ambiguous class is designed to capture.

Thank you for the clarification. I appreciate that you will separate the poisoned-reply and fabricated-history conditions from the headline rates (13.7% → 11.5%) and drop the "naturally-emerging" framing. Since these updates improve the clarity of what is being measured, I have raised my Soundness score from 2 to 3.

However, my core concerns remain unresolved(See Above Q1 & Q2). So I will maintain my overall rating unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.