acceptodds
Under review as a conference paper at ICLR 2027

Auditing Reward Hacking for Agent Environments via World Modeling

Abstract

Reward hacking allows language-model agents to earn high benchmark scores without completing the intended task. Existing approaches often detect hacking only after an agent has finished, or search for particular attack patterns whose coverage of other exploits remains uncertain. We introduce RewardProof, an agent-driven world-modeling framework for proactive auditing of agent environments. Its auditing agent explores and simulates potential hacking worlds, such as a state containing a forged reward file, and validates their effects by running submissions with and without the candidate attack before deployment. We introduce a fine-grained auditing protocol that distinguishes validated reward effects, unsuccessful attempts targeting the verifier, other unvalidated proposals, and cases with insufficient evidence. Across 567 environments from 18 benchmark families, we validate 1,116 attacks in 79.9% of environments and identify 44 environments whose scorers do not distinguish the tested submissions. Seventeen attacks also reproduce in the upstream SWE-bench Verified harness. On 72 shared tasks, our proposal method finds more validated attacks at higher precision than the evaluated proposal baselines. These results show how proactive auditing can give benchmark developers concrete evidence of scoring shortcuts before deploying an evaluation environment.

Primary area:
generative models
Keywords:
World Modeling, Language Model Agents, Agent Evaluation, Reward Hacking, Counterfactual Analysis, Benchmark Auditing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iG2WOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses agents to perform a large-scale scan for potential reward hacking across multiple benchmarks and validates the candidate attacks with a deterministic verification framework. Across 567 environments from 18 benchmark families, the authors validate 1,116 attacks in 79.9% of the environments and then use an LLM to categorize these attacks.

Critical strengths

The paper runs audits on 567 environments from 18 benchmark families and validates 1,116 attacks in 79.9% of the environments through paired execution. Of 25 attacks tested on the original SWE-bench evaluation, 17 reproduce, indicating that the findings do not depend solely on the audit environment and can provide benchmark developers with concrete evidence of vulnerabilities.

Critical weaknesses
  1. The method relies only on agent auditing and paired testing to identify tasks that may contain reward hacking, and its outputs lack human verification, so the overall novelty is limited.
  2. The hacking behaviors found in Harbor environments are difficult to verify in the original test environments, and the existing validation is limited to SWE-bench, so whether these behaviors transfer remains undetermined.
Questions for authors
  1. Could the authors discuss how well the hacking issues they identify transfer to other benchmarks?
  2. Could the authors disclose the models and agent frameworks used in the audit, along with the associated costs?
Reviewer_seSJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RewardProof, an auditing framework that uses an LLM agent to identify potential reward-hacking channels across evaluation data, scoring code, execution state, and reporting paths. Candidate attacks are validated through paired executions with and without the suspected exploit, using a fine-grained protocol to distinguish verified reward gains from unsuccessful or inconclusive attempts. Across 567 environments from 18 benchmark families, the framework validates over 1,000 attacks affecting 79.9% of environments, outperforms source-only proposal baselines, and reproduces 17 attacks in the upstream SWE-bench Verified harness.

Critical strengths
  • Systematic auditing with execution-based validation. The paper presents an automated workflow for proactively discovering reward-hacking channels across diverse agent environments, which is important in agent evaluation. Its matched-ablation protocol provides useful evidence of an operation's effect on reward, while the fine-grained verdicts distinguish successful, unsuccessful, and inconclusive attempts.

  • Broad empirical evaluation. The experiments span 567 environments across 18 benchmark families, with 1,116 validated findings. The authors additionally evaluate documented and planted exploit recovery and reproduce 17 attacks using the original SWE-bench harness, providing evidence beyond their primary auditing setup.

Critical weaknesses
  • Closely related work such as BenchJack already performs proactive, execution-enabled vulnerability discovery. RewardProof's main distinction appears to be its counterfactual validation protocol rather than a fundamentally new exploration method. Moreover, the baselines lack execution access, and the evaluation emphasizes proposal precision without reporting end-to-end costs or considering combinations of complementary baselines. Since all proposals undergo automated validation, the practical significance of the precision improvement remains unclear (Section 4.5, Table 4).

  • Unclear validity, uniqueness, and generalizability of discovered vulnerabilities. The experiments primarily audit benchmarks staged through Harbor rather than their original evaluation harnesses; notably, 8 of 25 attacks fail to reproduce upstream due to audit-specific state (Appendix D). Furthermore, the counterfactual validation establishes reward differences without independently confirming genuine reward hacking, while file-level deduplication does not eliminate semantically equivalent attacks. These limitations make the interpretation unclear for the number of reported attacks.

Questions for authors

See weaknesses and following questions:

  • Could the authors quantify what proportion of validated attacks remain effective in the original benchmark evaluation harnesses? In particular, how many findings exploit Harbor-specific files, permissions, or execution state rather than vulnerabilities inherent to the original benchmarks?
  • How does the validator independently ensure that the ablated submission differs only in the proposed attack operation? Could the authors manually adjudicate a representative sample to distinguish genuine reward hacking from legitimate reward-affecting changes, and report the number of distinct vulnerabilities after root-cause-level deduplication?
Reviewer_hEfPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Reward hacking can substantially inflate the benchmark scores for studied LLM agents. According to the paper, while there exist approaches for detecting such issues, they are usually applied after the agents finished their trajectories or simply scan for suspicious attack patterns. To this end, this paper proposes RewardProof to proactively audit agent environments via an agent-driven world modeling framework. The proposed approach introduces a fine-grained auditing protocol to distinguish verified reward gains from unsuccessful targeting. The experimental results show that the proposed approach can validate attacks in 79.9% of the studied environments and outperform some baselines.

Critical strengths

Reward hacking can provide false signals for agent training and applications, and it is great to see this paper targeting this critical problem. The overall idea of operation-level auditing protocol is also reasonable and can be promising.

The experimental study has been performed on over 500 real-world environments from various application domains, including software repair, terminal tasks, programming, reasoning, and professional work and research. I truly appreciate such large-scale evaluations, which clearly demonstrated the generalizability of the proposed approach.

Critical weaknesses

It is great to see the paper studied whether the discovered attacks can be reproduced in the original evaluation environment. Meanwhile, the study only focused on a subset of the attacks for one of the 18 studied benchmark families. It is not clear how the discovered attacks can transfer to real-world impacts on other benchmark families, and I highly recommend the paper include more studies on other benchmarks.

The studied baselines also seem a bit weak. It would be great if the paper can further justify why the included baselines are sufficient. For example, BenchJack aims to drive coding agents to audit benchmarks and identify possible reward-hacking exploits; it is not quite clear why such existing techniques were excluded in the evalutation.

I also feel the “world modeling” framing adds limited value and can be confusing. The agent proposes modifications, executes them in the actual environment, and evaluates an ablation. There is no learned predictive world model, and the structural-counterfactual notation largely formalizes this execution comparison.

Questions for authors

Q1: How would the discovered attacks translate into real-world impacts on other benchmark families besides SWE-bench Verified? Q2: Can you please elaborate on the included baselines and justify why they are sufficient? Q3: Why is it called “world-modeling”?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.