acceptodds
Under review as a conference paper at ICLR 2027

Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack

Abstract

Agent benchmarks have become the de facto measure of frontier AI competence, guiding model selection, investment, and deployment. However, reward hacking, where agents maximize a score without performing the intended task, emerges spontaneously in frontier models without overfitting. We argue that benchmarks must be secure by design. From past incidents of reward hacks, we derive a taxonomy of eight recurring flaw patterns and compile them into the Agent-Eval Checklist for benchmark designers. We condense the insights into BenchJack, an automated red-teaming system that drives coding agents to audit benchmarks and identify possible reward-hacking exploits in a clairvoyant manner. Moreover, we extend BenchJack to an iterative generative-adversarial pipeline that discovers new flaws and patches them iteratively to improve benchmark robustness. We apply BenchJack to 10 popular agent benchmarks spanning software engineering, web navigation, desktop computing, and terminal operations. BenchJack synthesizes reward-hacking exploits that achieve near-perfect scores on most of the benchmarks without solving a single task, surfacing 219 distinct flaws across the eight classes. Moreover, BenchJack's extended pipeline reduces the hackable-task ratio from near 100% to under 10% on four benchmarks without fatal design flaws, fully patching WebArena and OSWorld within three iterations. Our results show that evaluation pipelines have not internalized an adversarial mindset, and that proactive auditing could help close the security gap for the fast-paced benchmarking space.

Primary area:
datasets and benchmarks
Keywords:
benchmark, reward hacking, benchmark auditing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zspPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work presents a taxonomy of eight major classes of reward-hacking vulnerabilities in agent benchmarks and derives an auditing checklist from it. Building on this checklist, the authors develop an automated pipeline for detecting reward-hacking exploits. They then use the findings to patch existing benchmarks and analyze the effectiveness of these patches. Overall, the work identifies a genuine concern in current evaluation practice: most of the audited agent benchmarks are susceptible to reward hacking due to weaknesses in their evaluation design or implementation. In my view, the proposed checklist also addresses practical concerns relevant to current agent evaluation.

Critical strengths
  1. The paper organizes recurring reward-hacking vulnerabilities into a taxonomy and an actionable checklist, then incorporates this guidance into an automated auditing pipeline. The pipeline goes beyond identifying potential issues by constructing executable exploits, making the findings useful for benchmark developers.

  2. The study examines multiple widely used agent benchmarks and evaluates a patch-and-re-audit workflow. The reported reduction in attack success rates on some benchmarks provides evidence of the practical value of proactive auditing and targeted mitigation.

Critical weaknesses
  1. The checklist's incremental contribution over Zhu et al.'s Agentic Benchmark Checklist (ABC) is insufficiently established. Although the paper cites ABC, it does not clearly distinguish requirements already covered by that work from more specific implementation guidance or substantively new checks. Several central concerns, including ground-truth isolation, adversarial inputs to LLM judges, weak string matching, and evaluation shortcuts, are already addressed by ABC. A direct comparison is needed to establish the checklist's novelty as a standalone contribution.

  2. The contribution and potential limitations of taxonomy-guided auditing are not adequately evaluated. The experiments use a strong coding-agent backend, but lack a controlled comparison against the same agent auditing without the predefined taxonomy. Consequently, it is unclear how much of the vulnerability discovery capability comes from the proposed guidance rather than the underlying model. Conversely, organizing the search around eight predefined classes may bias exploration and leave other vulnerabilities undiscovered; this possibility is not assessed.

Questions for authors

If possible, I would like to see a comparison of the pipeline's vulnerability detection results using the proposed checklist, the Agentic Benchmark Checklist (ABC) from https://arxiv.org/abs/2507.02825 and no checklist. This would directly demonstrate the effectiveness of the proposed checklist.

1 comment

We genuinely appreciate the reviewer for the comprehensive comments concerning BenchJack. Please see our point-to-point response to your comments below.

Incremental contribution over the Agentic Benchmark Checklist

We acknowledge that we did not sufficiently discuss Agentic Benchmark Checklist (ABC) in our previous version of the paper. However, we do note that ABC only considers the construct validity of the benchmark. That is, whether the task is solvable by an honest-but-capable agent. The only detailed overlap between ABC and reward hacking is in the information acquisition section, where ABC mentions the robustness against adversarial inputs for LLM-as-judge and robustness in string matching. Our taxonomy, on the other hand, targets specifically reward hacking, and in turn models reward hacking more thoroughly. Concretely, the reward hacking risks covered in ABC is a subset of the V4-6. The other vulnerabilities in the taxonomy covers novel hacking behaviors not captured by ABC.

To showcase the effectiveness of our taxonomy, we conduct an ablation study comparing BenchJack with two variants, ABC and no checklist. We run the three agents on Terminal-Bench and report the results below.

BenchJackABCNo Checklist
Tasks hacked100%100%100%
Exploit attributionV1 + V8T.10 only-
Findings5411 (FP)-

Overall, we find that the hack rates are all 100% as Terminal-Bench has a universal exploit that easily generalizes to hack all the tasks. However, the three shows massive difference on the findings and the exploit attributions. Without any checklists, the agent does not form any concrete findings. With ABC, the agent very generally categorizes everything under T.10, which corresponds to the general reward hacking category. Additionally, the produced findings are all related to the quality of the tasks rather than the reward hackability of the implementations. In comparison, BenchJack attributes the exploit to V1 and V8 correctly and also produces more critical findings that can lead to reward hacks.

We will include this ablation in the next revision of the paper and clarify our contribution in comparison to ABC.

Does organising the search around eight classes bias exploration?

We acknowledge that the eight classes may lead to unbalanced exploration for potential reward-hacking flaws. However, our work aims to solve this via two key designs. First, the agent is designed to maximize hack rate as its ultimate goal, not the findings under the taxonomy. Thus, even when there are new flaws surfacing outside of the current taxonomy, the agent will still use it to achieve higher hack rates. Additionally, we introduce the adversarial patching loop such that after all of the flaws related to the taxonomy are depleted, other reward hacking flaws can still be detected by the hacking agent. We will discuss this in the next revision of the paper.

Reviewer_Cm1pOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an eight-class taxonomy of benchmark vulnerabilities and BenchJack, an automated pipeline for discovering, constructing and testing reward-hacking exploits. It audits ten agent benchmarks, reports 219 flaws, and demonstrates widespread exploitability. An iterative attack-and-patch study evaluates whether discovered vulnerabilities can be mitigated and whether new exploits remain.

Critical strengths
  • Concrete exploit constructions across ten widely used benchmarks. Several examples expose consequential failures in evaluator isolation and trust in agent-controlled outputs.

  • Re-attacking patched benchmarks is valuable. It demonstrates why blocking one payload can leave the underlying vulnerability intact.

Critical weaknesses
  • Some reported attacks do not clearly respect the stated agent permissions. In WebArena, the mock also intercepts evaluator LLM calls and returns favorable judgments (App. G.10, p. 43). No agent-accessible path to this control is established, weakening the claimed 812/812 result.

  • Reduced hack rates do not establish preserved benchmark validity. The OSWorld patch forces certain task configurations to score zero (App. H.5, p. 47), without showing that legitimate solutions still pass. Failure of the same auditor to find another exploit also does not justify “unhackable.”

Questions for authors
  • What are the success rates when the mock controls only the evaluated agent, while evaluator models and host-side configuration remain untouched?

  • How many original tasks remain solvable after patching? Please report reference-solution or benign-agent performance alongside independently re-audited hack rates.

1 comment

We genuinely appreciate the reviewer for the comprehensive comments concerning BenchJack. Please see our point-to-point response to your comments below.

Attacks that exceed the stated agent permissions

Thank you for catching this error. We re-inspected the exploits generated by BenchJack and confirmed that the WebArena mock of llm_fuzzy_match / llm_ua_match does not affect the reported numbers. Since the exploit uses the file:// to access the golden solution, it already achieves 100% on all of the tasks and is judge-independent. The mock to the two functions is generated when the agent is trying to bypass the cost of calling any LLMs during the exploit validation phase. We re-inspected all the other exploits and they do not contain any threat model violations.

Preserved validity, and how many tasks remain solvable after patching

We agree that reduced hack rates do not necessarily directly indicate preserved validity. Indeed, the OSWorld patch would directly reject 5/369 (1.36%) of the tasks due to them comparing the agent output to another file in the VM that agent can potentially touch and modify. We run the ground-truth solutions on the other patches and they do not affect any benign submissions.

However, we note that although the OSWorld patch directly rejects some of the tasks due to the their configurations, this does not negatively impact the quality of the benchmark. Since these tasks are configured to use files in the VM as ground-truth answers, they are heavily prone to reward-hacking agents trying to tamper with the golden files and hack the rewards, which is structurally un-patchable without heavy per-case redesign of the verifier. Although fully patching this flaw is possible, it is orthogonal to this paper and is left to future works to explore.

We will clarify this in the patching and discussion section in the next revision of the paper.

Reviewer_6Cz2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a pipeline for finding reward hackable exploits in benchmarks. They categorize reward hacking exploits in 8 different classes. Further they build a system which automatically finds flaws in benchmarks, given those 8 classes. They then try to validate their system by finding that many benchmarks (including OSWorld, SWE-bench Verified and Pro, Terminal-Bench) contain close to 100% reward hackable tasks.

Critical strengths
  • timely and important area. It is important to build non-rewardhackable benchmarks / environments and a system in this direction is appreciated.
Critical weaknesses
  • unfortunately it is hard to review the paper, because core aspects of the methodology are missing. How are the 8 flaw categories generated in detail? What was the exact method. Further, how is the "hack percentage" in fig 7 and 8 computed?
  • evaluation of their method is flawed in places. In particular: No baselines (like measuring just asking an agent to find flaws, without BenchJack), no false-positive rate (are all the reward hacks which the agent finds actual reward hacks), no measurement of how often those reward hacks are actually exploited by in-the-wild agents.
Questions for authors
  1. "we review these reward-hacking instances, summarize the root causes from a security perspective, and manually verify that the resulting patterns cover all of the reported problems in our review". This part seems crucial. Can you give more details on the procedure you used? Benchmarks, agents, etc.
  2. How often did agents actually exploit the harm categories which you found, without being prompted for it?
  3. How did the reward hackability change before and after you fixed the benchmarks? E.g. try with an agent which reward hacks a lot (like deepseek-v4-flash) and check if after your fixes, reward hacks still occur / to which amount they still occur. Maybe you already did this with Fig 7 & 8, although it is unclear to me what the "Hack Percentages" there refer to.
  4. What do the "hack percentages" in fig 7 & 8 refer to, how are they measured?
  5. On future work: A main problem in current benchmarks is wrong specification of what is being rewarded. (See e.g. the problems with SWE-Bench Pro). Could BenchJack be extended to such problems as well and to which degree?
  6. How does your system compare to some simple baselines, like just asking an agent to find reward hacks?
  7. Which version of Terminal-Bench do you use?
3 comments

We genuinely appreciate the reviewer for the comprehensive comments concerning BenchJack. Please see our point-to-point response to your comments below.

How the taxonomy was derived

We acknowledge that this is not properly documented in the submission. We built a set of previously-reported reward-hacking incidents from the several sources cited in our paper, including [1,2,3]. We then independently inspected each incidents by their mechanisms and, generalizing to similar security concepts, derived the classes in the taxonomy. We will elaborate this more clearly in the next version of the paper.

What the hack percentages in figure 7 and 8 indicate

Figure 7 and 8 are acquired by rerunning BenchJack on the patched version of the benchmarks. The hack percentage indicates the effectiveness of the exploits found by BenchJack, i.e., how much percentage of the tasks are reward-hacked by the exploit that BenchJack generates. We think that this is a reliable proxy to the actual strength of the benchmark against reward hacking as we demonstrate in Figure 5 the effectiveness of the tool to hack benchmarks.

Comparison against simple baselines

We agree that our evaluation did not sufficiently disentangle the contributions of our harness. To showcase the improvement from the BenchJack tool, we run a simple baseline that asks the agent to reward hack the benchmark on Terminal-Bench and SWE-bench Pro.

AblationTerminal-Bench Hack rateTerminal-Bench FindingsSWE-bench Pro Hack rateSWE-bench Pro Findings
BenchJack100%54100%21
Baseline100%2355%10

We find that asking the agent to hack yields similar hack rates on Terminal-Bench. However, the generated exploit hacks only 55% of the tasks on SWE-bench Pro. Additionally, the baseline yields far fewer findings, showing the improvement on audit breadth by the BenchJack orchestration.

We will present the results in the paper to provide more insights and supplement our current evaluation.

False-positive rate

The hack rate reported by BenchJack have 0 false-positives in our evaluation. We designed BenchJack such that the generated exploits are actually run through the official entrypoint of the benchmark. Thus, the hack rates reported are machine-verified.

How often do agents exploit these flaws unprompted

We note that the exact frequency is orthogonal to this work and is itself an interesting question worth studying. However, we do show in our paper that empirically the unprompted reward hacks are relatively frequent and could appear in evaluations for both frontier models and smaller moderate-sized models. The exploitations may not use the exact flaws found by our auditing. However, they follow the same patterns as listed in our taxonomy. We thus propose clairvoyant auditing of possible reward-hacking flaws by BenchJack, detecting and patching the flaws before they are actually used to hack.

How hackability changed before and after the fixes

Figure 7 and Figure 8 in the paper shows this comparison by running BenchJack on the versions before and after the fixes. We find that although some benchmarks with good designs (like isolated evaluators, structured verifiers, etc.) can be iteratively patched, some benchmarks have deeply-rooted culprits that call for more advanced and human-integrated patching methods (like Terminal Bench, which needs to migrate all of the tasks to isolate the verifier from the agent runtime environment).

Extending BenchJack to mis-specified rewards

This is indeed an interesting direction. V5 and V6 in our taxonomy are outcome-validity classes and already catch the mechanical half of the problem. However, BenchJack does not judge whether a task's reward specification matches the claims of the task and the benchmark. We note that although BenchJack can potentially be altered to provide an estimate, fully capturing the mis-specified rewards require the developer's intent which is not necessarily present.

Which version of Terminal-Bench

Thank you for pointing this out. We audit Terminal-Bench 1 in our paper as is mentioned in Appendix C. More specifically, harbor-framework/terminal-bench at commit 1a6ffa9. We will correct the entry in table 1 in the next revision of the paper.

[1] IQuestLab. Reward hacking of SWE-Bench. https://github.com/IQuestLab/IQuest-Coder-V1/issues/14, 2026. GitHub issue

[2] Beth Barnes Sydney Von Arx, Lawrence Chan. Recent frontier models are reward hacking, 2025. URL https://metr.org/blog/2025-06-05-recent-reward-hacking/.

[3] Anthropic. Alignment risk update: Claude mythos preview, 2026. URL https://www-cdn.anthropic.com/3edfc1a7f947aa81841cf88305cb513f184c36ae.pdf.

Thank you for your response.

We note that the exact frequency is orthogonal to this work and is itself an interesting question worth studying.

I disagree. In my opinion this is important, because if the reward hacks are not exploited in practice, they might not be relevant. Whether or not this paper is relevant hinges on the question if those reward hacks are an issue or not. I think this would be an important analysis.

E.g. you can use deepseek v4 flash, which is a model which likes to reward hack.

However, we do show in our paper that empirically the unprompted reward hacks are relatively frequent and could appear in evaluations for both frontier models and smaller moderate-sized models.

Where do you show this?

Comparison against simple baselines

Which model did you use?

I'd advise you to use deepseek v4 flash, as it is a model which likes to reward hack a lot.

We agree that showcasing that the models are capable to reward hack is important to the paper. In our introduction we cited [1,2,3], which showcases the prominence of reward hacking. We do note that there has also been other evidences for the frequency of reward hacking. In [4], METR showcases 44 high-risk misalignment incidents in Figure 10 from their internal evaluation over three frontier models from OpenAI, Anthropic, and Google DeepMind, which are mostly attempts to reward hack. Additionally, [5] also shows that on two of the tasks (Rust C compiler and VLIW SIMD kernel optimization), Gemini 3.1 pro, Kimi K2.6, and GPT 5.5 hacks in 3/5 (60%) of the attempts. SWE-Bench has also reported the same reward hacks with conftest.py that we found in our paper [6]. These reward hacks in [4,5,6] spans V1-V8 identified in the taxonomy, as they involve sandbox escapes, lookup tables, using admin permission to peak ground-truth tests, etc. We will include these extra evidence in the next revision of the paper.

In order to provide more direct evidence of reward hacking, we further evaluate DeepSeek-V4-Flash on a subset of the tasks and inspect for any reward hacks. In four of the benchmarks (SWE-bench Pro, Terminal-Bench, SkillsBench, FrontierSWE), we randomly sample 10 problems and run each problem 5 times. Unprompted, the model reward hacks two of the benchmarks: on SWE-bench Pro the model directly reads .git/ for future commits; on SkillsBench it hacks fix-druid-loophole-cve by finding the ground-truth patch online. We will include this experiment in the next revision of the paper to further support our claims.

We do note that these reward hacks are not specifically the ones BenchJack used to reward hack the benchmarks, but BenchJack found these hacks and listed them in the audit findings. We thus do not claim that the current models will exploit every hacks found in our evaluation. However, with the models become stronger, more and more existing flaws in the benchmarks can be found and used to exploit. Thus, we propose adversarial reward hack detection and patching with BenchJack to resolve potential reward hacks before they actually appear in the full evaluation run.

[4] METR. Frontier Risk Report (February to March 2026), 2026. URL https://metr.org/blog/2026-05-19-frontier-risk-report/

[5] Abundant, SWE-Marathon, 2026. URL https://www.swe-marathon.org/

[6] SWE-bench, A model patch containing only a conftest.py is scored resolved, https://github.com/swe-bench/SWE-bench/issues/631, 2026. GitHub issue

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.