acceptodds
Under review as a conference paper at ICLR 2027

Scalable Weak-to-Strong Verifier Hardening for Agent Environments

Abstract

Agent environments are essential for large language model (LLM) development. They use programmatic verifiers to turn complex outcomes into rewards. Verifiers can be attacked into awarding full credit to cheating agents without completing the intended task, causing inaccurate evaluation and misalignment in training. As agent environments scale in numbers and difficulty, so must the repairing of exploitable verifiers. To this end, we ask whether today's model with scaled compute can automatically harden agent environments to prevent potentially stronger future models from reward hacking. We introduce a scalable, fully automatic approach to weak-to-strong verifier hardening. It first sets up a separate grading environment that agents cannot directly modify, then uses a hacker-fixer loop to find and repair remaining verifier vulnerabilities. We apply this loop to Terminal Bench, KernelBench, and SETA, substantially reducing attack success across these agent environments. The iterative design scales with the hardening budget, producing progressively stronger defenses as the number of loop iterations increases and outperforming the static, non-iterative baseline. An earlier version of our hacker-fixer loop has been adopted by the official Terminal Bench repository as a way to audit exploits in its environments.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward hacking, LLM agents, verifier hardening

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m6opOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper audits reward hacking patterns across several agent benchmarks and proposes a weak-to-strong verifier hardening method built on an isolated sandbox and a hacker-fixer loop, in which weaker models repair the verifier against stronger agents. On TerminalBench, KernelBench and SETA, attack success rates drop substantially while benign solve rates remain largely unchanged, and two ablation groups separate the contributions of hardening capability and benign validity.

Critical strengths
  1. The problem is clearly posed and targets reward hacking in benchmarks, which has clear practical relevance.
  2. The ablations are thorough, and the multiple comparisons both identify the mechanisms behind reward hacking and support comparisons across method components.
Critical weaknesses
  1. Evaluation relies on a single judge backend from the GPT family, which may introduce same-family bias, and no agreement rate between this judge and human annotation is reported.
  2. Patch acceptance depends on whether the current model can solve the task, and hard-task settings are discussed insufficiently.
Questions for authors
  1. Could the authors report ASR re-judged by additional judge models, so as to show how sensitive the conclusions are to the choice of judge?
  2. Could the authors walk through the hardening process on a few hard tasks, showing how a patch is accepted or rejected when the weaker model cannot solve the task?
  3. Benign pass rates drop after hardening (on SETA, Gemini falls from 92.5% to 84.5%). Could the authors explain where these drops come from?
Reviewer_cQkLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether a weaker model can harden the verifiers of agentic task environments against stronger attackers, so to answer if the current model can help avoid cheating of future models. It first moves the verifier into a separate sandbox that only receives the agent's declared output files. Then a weaker model with access to the verifier source repeatedly finds exploits and patches the verifier, with checks that legitimate solutions still pass. The hardened environments are evaluated on Terminal Bench, KernelBench and SETA with stronger models as attackers.

Critical strengths

1/ Reward hacking in agent environments is an important problem. The experiments are quite complete, with three benchmarks, three held-out attackers and ablations of each component, and the validation keeps legitimate solve rates close to the original.

Critical weaknesses

1/ The weak-to-strong part, which is the core contribution, is not well supported, and a comparison with a strong hardener is missing. Specifically:

a) Using a weaker model to defend against stronger ones is a common setup (e.g. trusted monitoring in AI control [1]), and here the stronger models are already available. Hardening is done once per environment, so why not use the strongest model to fix it? The paper has no strong hardener baseline. It also doesn't test whether a stronger model in the loop can still find new exploits and patches after the weak loop stops. The results suggest it can: with the verifier source, Opus 5 still hacks 71.6% of the hardened Terminal Bench tasks, and Appendix E.2 shows a new exploit found by Opus after the loop stopped.

b) The validation only uses legitimate solutions from weak models (the solution corpus and the fresh solver). So patches may block legitimate solutions from stronger models that the weak model never tried (e.g. Gemini's benign rate on SETA drops from 92.5% to 84.5%). Also, the hack labels in the loop and in evaluation all come from LLM judges, and the only human check is on 49 environments of the separate audit. Therefore, it is unclear whether the hardened verifiers are both robust and still fair to stronger models.

2/ Verifier separation and adversarial hacker-fixer loops are already well established. BenchJack [2] already finds and patches benchmark exploits iteratively. It has also already attacked Terminal Bench: four of the eight hinted exploits in this paper come from its disclosure. The paper only cites its blog post, and doesn't explain the difference or compare with it, or with other similar work.

[1] AI Control: Improving Safety Despite Intentional Subversion. Greenblatt et al.

[2] Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack. Wang et al.

Questions for authors

1/ Can authors compare with hardening by a stronger model at the same cost? Can a stronger model in the loop still find new exploits after the weak hardening?

2/ Can authors report how often legitimate solutions from stronger models are blocked after hardening? Have the hack labels and accepted patches been checked by humans?

Reviewer_3hiLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses reward hacking, this happens when the agents exploit weaknesses in the verifiers to obtain rewards without completing the tasks. It introduces a scalable weak to strong hardening method which uses a weaker AI model to improve verifiers against stronger model attacks. This is done using an iterative pipeline. The authors evaluate the method on Terminal Bench, KernelBench, and SETA. The results show a substantial reduction in ASR (attack success rates) when using full hardening while preserving the benign task performance.

Critical strengths
  • Idea is simple and the results show that it works well on the three tested evaluators and benchmarks
  • The paper addresses an important and timely problem of AI agents being able to exploit weaknesses in automated verifiers by reward hacking.
  • The idea of using a weak to strong approach to address reward hacking is well motivated.
Critical weaknesses
  • W1 The paper is hard to follow at times, and writing can be improved. Figure 2 can be made more clear to illustrate the workflow better.
  • W2 The reasoning behind having fresh blind solve in the pipeline is unclear and can be clarified.
  • W3 The selection of only a subset of tasks from the selected benchmarks makes the evaluation limited to a few cases.
  • W4 The explanation behind using different models for TerminalBench and KernelBench for hardening is not mentioned.
Questions for authors
  • Q1 Why is Hinted ASR lower than unhinted ASR ? It looks counterintuitive and in what case does this happen?
  • Q2 For verifier aware attacks after hardening, the ASR is still high. Also, using a weak model still has a chance of missing an exploit the stronger model might be able to use. Is this cause of concern and what explains this ?
  • Q3 Table 2 shows the verifier-aware ASR comparison between full hardening and static-15. But the two differ on more than one thing as full hardening has much more computation than static-15. Can a controlled experiment be performed for this.
  • Q4 There is a task pre-check for each benchmark using the weaker model and only those tasks are accepted which that model can solve. Won't this limit the method to only work in this particular setting ? What happens to the tasks which the stronger model can solve but the weaker model can't ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.