- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work presents a taxonomy of eight major classes of reward-hacking vulnerabilities in agent benchmarks and derives an auditing checklist from it. Building on this checklist, the authors develop an automated pipeline for detecting reward-hacking exploits. They then use the findings to patch existing benchmarks and analyze the effectiveness of these patches. Overall, the work identifies a genuine concern in current evaluation practice: most of the audited agent benchmarks are susceptible to reward hacking due to weaknesses in their evaluation design or implementation. In my view, the proposed checklist also addresses practical concerns relevant to current agent evaluation.
Critical strengths
-
The paper organizes recurring reward-hacking vulnerabilities into a taxonomy and an actionable checklist, then incorporates this guidance into an automated auditing pipeline. The pipeline goes beyond identifying potential issues by constructing executable exploits, making the findings useful for benchmark developers.
-
The study examines multiple widely used agent benchmarks and evaluates a patch-and-re-audit workflow. The reported reduction in attack success rates on some benchmarks provides evidence of the practical value of proactive auditing and targeted mitigation.
Critical weaknesses
-
The checklist's incremental contribution over Zhu et al.'s Agentic Benchmark Checklist (ABC) is insufficiently established. Although the paper cites ABC, it does not clearly distinguish requirements already covered by that work from more specific implementation guidance or substantively new checks. Several central concerns, including ground-truth isolation, adversarial inputs to LLM judges, weak string matching, and evaluation shortcuts, are already addressed by ABC. A direct comparison is needed to establish the checklist's novelty as a standalone contribution.
-
The contribution and potential limitations of taxonomy-guided auditing are not adequately evaluated. The experiments use a strong coding-agent backend, but lack a controlled comparison against the same agent auditing without the predefined taxonomy. Consequently, it is unclear how much of the vulnerability discovery capability comes from the proposed guidance rather than the underlying model. Conversely, organizing the search around eight predefined classes may bias exploration and leave other vulnerabilities undiscovered; this possibility is not assessed.
Questions for authors
If possible, I would like to see a comparison of the pipeline's vulnerability detection results using the proposed checklist, the Agentic Benchmark Checklist (ABC) from https://arxiv.org/abs/2507.02825 and no checklist. This would directly demonstrate the effectiveness of the proposed checklist.
We genuinely appreciate the reviewer for the comprehensive comments concerning BenchJack. Please see our point-to-point response to your comments below.
Incremental contribution over the Agentic Benchmark Checklist
We acknowledge that we did not sufficiently discuss Agentic Benchmark Checklist (ABC) in our previous version of the paper. However, we do note that ABC only considers the construct validity of the benchmark. That is, whether the task is solvable by an honest-but-capable agent. The only detailed overlap between ABC and reward hacking is in the information acquisition section, where ABC mentions the robustness against adversarial inputs for LLM-as-judge and robustness in string matching. Our taxonomy, on the other hand, targets specifically reward hacking, and in turn models reward hacking more thoroughly. Concretely, the reward hacking risks covered in ABC is a subset of the V4-6. The other vulnerabilities in the taxonomy covers novel hacking behaviors not captured by ABC.
To showcase the effectiveness of our taxonomy, we conduct an ablation study comparing BenchJack with two variants, ABC and no checklist. We run the three agents on Terminal-Bench and report the results below.
Overall, we find that the hack rates are all 100% as Terminal-Bench has a universal exploit that easily generalizes to hack all the tasks. However, the three shows massive difference on the findings and the exploit attributions. Without any checklists, the agent does not form any concrete findings. With ABC, the agent very generally categorizes everything under T.10, which corresponds to the general reward hacking category. Additionally, the produced findings are all related to the quality of the tasks rather than the reward hackability of the implementations. In comparison, BenchJack attributes the exploit to V1 and V8 correctly and also produces more critical findings that can lead to reward hacks.
We will include this ablation in the next revision of the paper and clarify our contribution in comparison to ABC.
Does organising the search around eight classes bias exploration?
We acknowledge that the eight classes may lead to unbalanced exploration for potential reward-hacking flaws. However, our work aims to solve this via two key designs. First, the agent is designed to maximize hack rate as its ultimate goal, not the findings under the taxonomy. Thus, even when there are new flaws surfacing outside of the current taxonomy, the agent will still use it to achieve higher hack rates. Additionally, we introduce the adversarial patching loop such that after all of the flaws related to the taxonomy are depleted, other reward hacking flaws can still be detected by the hacking agent. We will discuss this in the next revision of the paper.