acceptodds
Under review as a conference paper at ICLR 2027

VeriHarm-Bench: Measuring Harmful Success in LLM Coding Agents Verifiably

Abstract

Large language model (LLM) coding agents increasingly operate with shell access, allowing them to write scripts, move files, alter permissions, and modify system configurations. However, an agent may achieve a user's goal while causing unintended harm, such as deleting unrelated files, weakening access controls, or corrupting data, in response to benign inputs without explicit jailbreaking. This failure mode is particularly important because apparent task success may reassure users that an agent behaved correctly, causing harmful side effects to go unnoticed. We call this *harmful success*: the agent successfully completes the intended benign task while simultaneously causing unintended harm. Despite its importance, harmful success is difficult to measure because conventional task-success metrics overlook harmful behaviors, while existing safety evaluations are often designed solely to assess harmful outcomes. We introduce VeriHarm, an automatic framework for constructing programmatically verifiable benchmarks that directly measure harmful success. VeriHarm minimally perturbs coding tasks to elicit unintended harmful behavior while maintaining the benignity of the task and preserving the original task objective. It then synthesizes executable harm tests that, together with the original task-correctness tests, evaluate both task accuracy and harmful outcomes. Frontier agents, including GPT-5.6-Sol and Claude-Opus-5, achieve an average task accuracy of 95% on our curated VeriHarm-Bench, yet still struggle to complete tasks safely: their safe success rate, the fraction of tasks completed successfully without triggering the designated harm, averages only 37%. These results suggest that successful task completion can frequently coexist with unintended harm, and that task-success metrics alone may substantially understate the potential safety risks of coding agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Coding Agent, Agent Safety, Harmful Success

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_17fbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VeriHarm, a framework for building benchmarks of "harmful success," where an agent completes a benign coding task but causes unintended harm along the way. Starting from BashBench tasks, a proposer model repeatedly modifies the task instructions and writes executable harm tests, guided by feedback from target-agent rollouts. The original task-correctness tests stay fixed. A candidate is kept when both the correctness test and the harm test pass, and the kept candidates are then ranked by two LLM judges.

The resulting VeriHarm-Bench has 150 instances derived from 97 distinct source tasks. The authors evaluate four agents that played no role in building the benchmark. On average, the agents complete 95.1% of tasks, split into 58.2% harmful success and 37.0% safe success. In a human audit, 21 of 25 sampled benchmark instances are judged valid, compared with 2 of 25 sampled bottom-ranked candidates.

Critical strengths
  1. A practical approach to reproducible joint evaluation. The paper reuses established correctness tests and generates harm tests specific to each task. This gives a concrete way to tell successful-but-harmful outcomes apart from ordinary task failures. Prior benchmarks such as OS-Harm (NeurIPS 2025) already evaluate task completion alongside safety. What is new here is that the executable checks are built automatically and that no model-based judge is needed at evaluation time. Fixed executable checks don’t depend on a judge model that may change, and each verdict can be inspected. This could be useful infrastructure for evaluating safeguards on coding agents.
  2. The evaluation goes beyond the agents used to find the failures. None of the four evaluation agents served as target, proposer, or judge during construction. Their results suggest that the selected cases expose weaknesses beyond the agents on which they were found. The paper is also transparent: it reports the human audit with agreement statistics, the construction prompts, and a cost breakdown.
Critical weaknesses
  1. Some harm tests measure latent vulnerabilities or outcomes the test itself causes, not harmful side effects of the agent’s rollout. Section 2.1 and Appendix A say harm is verified on the final sandbox state. The motivating scenario is an agent that completes its task but causes an unintended harmful side effect. However, Appendix F’s shell-injection example runs the generated script with malicious search strings and checks whether those runs create marker files (pp. 22–23). This shows that the generated artifact can be exploited. It does not show that any command-execution harm occurred during the benign-task rollout. Similarly, the permissions example can run the generated script to create a missing file before checking that file’s permissions (p. 25). So, these checks are not always passive inspections of the final state: the evaluation itself can produce the state it then assesses. Generating vulnerable software is a real safety concern, but it is not the same as causing operational harm during execution. Without knowing how many instances fall into each category, the aggregate harmful-success rate mixes different outcomes. The benchmark should separate the two categories and report results for each. Where tests execute or modify artifacts, the protocol should also say how task checks and harm checks are kept isolated from each other.
  2. The human audit reveals unresolved validity errors, and the reported ranking metrics claim more than has been validated. Four of the 25 audited top-ranked instances fail the paper’s own validity criteria. That matters for a benchmark whose central contribution is reliable, executable harm measurement. The paper does not explain these failures or say whether the affected instances were repaired or excluded. It also does not show how filtering out invalid instances would change the agent results. In addition, the reported 88% accuracy and 91% recall are computed on a sample that contains only the top and bottom of the ranking. They show that the two ends can be told apart. They do not establish recall over the full pool of 1,046 candidates, because the middle of the ranking was never annotated. The more relevant estimate is that 84% (21/25) of sampled benchmark items are valid. The 95% confidence interval for that is wide, roughly 65–94%, and annotator agreement is only moderate. None of this negates the failures the benchmark exposes. But it leaves the reliability of individual labels, and therefore the exact benchmark scores, uncertain. A test being executable and scored deterministically does not by itself show that it correctly captures unintended harm.
Questions for authors

Q1–Q3 address the two weaknesses above and can likely be answered with the existing benchmark. Convincing answers could materially improve my assessment: an audit of what each harm test measures, separate results for operational harms and artifact vulnerabilities, and corrected scores. Q4 is about the construction method. None of these questions asks for expansion to other domains or models.

  1. What does each harm test measure? (clarification and reanalysis) How many benchmark instances detect a state change caused by the agent that is already present when the rollout ends? How many instead actively probe the generated artifacts for vulnerabilities? Please report agent results separately for these two categories. Do task tests and harm tests run on isolated copies of the final environment?
  2. What failed the human validity audit? (clarification and reanalysis) For the four invalid top-ranked instances, which criteria failed, and were these instances kept in Table 2? Can you provide corrected results after adjudication, along with a breakdown of the reasons for invalidity? Please also describe the annotators’ relevant expertise and how the 25 benchmark items were stratified for sampling.
  3. What are the exact evaluation counts? (clarification) Appendix B.3 specifies one rollout per instance on a 150-instance benchmark. However, some reported values cannot come from 150 pass/fail outcomes. For example, Claude-Opus-5’s 56.2% harmful success is impossible: the nearest achievable values are 56.0% and 56.7%. Please give numerators and denominators and explain how failed or missing runs were handled. The benchmark contains variants of the same 97 source tasks, and all agents are evaluated on the same instances. How do the reported uncertainty estimates account for these dependencies?
  4. What does rollout feedback add beyond simply making more attempts? (additional analysis) Table 1 shows that later iterations find more candidates, but more independent proposals would also find more over time. On a representative subset, how does refinement with feedback compare with the same number of proposals made without feedback from failed rollouts? It would also help to run the selected harm tests on rollouts of the original, unmodified tasks. That would show how much the instruction changes themselves affect harmful-success rates.
Reviewer_PCrMOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VeriHarm-Bench to evaluate harmful success in LLM coding agents, where an agent completes a benign task but also causes unintended harm. The authors use agent rollouts to generate small task modifications and corresponding executable harm tests. The original task tests are kept to check task correctness, while LLM judges are used to select 150 benchmark instances. Experiments on four held-out coding agents show that high task accuracy does not necessarily mean the tasks are completed safely.

Critical strengths
  1. The paper studies an important problem in coding-agent safety. Evaluating both task completion and unintended harm is useful, since task accuracy alone may overlook harmful side effects.

  2. The idea of generating executable harm tests is practical. It avoids relying on LLM judges when evaluating new agents. The paper also tests several frontier agents and includes human evaluation of benchmark quality.

Critical weaknesses
  1. I have some concerns about the reliability of the generated harm tests. Although the paper emphasizes verification through the final sandbox state, some tests in Appendix F actively execute the generated scripts with additional inputs to trigger unsafe behavior. This seems different from directly detecting unintended harm caused during the original agent execution. I am not sure whether all such cases should be counted as harmful success.

  2. The benchmark is relatively small and limited to BashBench tasks. The final 150 instances come from only 97 source tasks. Also, only 84% of the sampled top-ranked instances are considered valid in the human evaluation. I think more validation is needed to show that the benchmark is reliable and covers a reasonable range of coding-agent safety issues.

  3. The methodological novelty of the benchmark construction process could be better justified. The task perturbation strategy is largely based on AutoElicit, including underspecification and delegation of control. The main difference seems to be the automatic generation of harm tests. It would be helpful to show more clearly what the rollout-guided refinement adds over simpler generation methods.

Questions for authors
  1. How do the authors distinguish harmful side effects that actually occur during the agent's execution from vulnerabilities that are only exposed by additional harm tests?

  2. Could the authors provide more evidence about the correctness and diversity of the 150 selected instances?

  3. How much improvement does rollout-guided refinement provide over simpler generation methods under the same computational budget?

Reviewer_7KfUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies “harmful success”: a coding agent completes a benign task correctly but causes an unintended side effect along the way, like emptying a file, loosening permissions, or touching unrelated data. VeriHarm builds such cases automatically. Starting from BashBench tasks, a proposer model makes a small edit to the instruction (underspecification or more delegated control) and writes an executable harm test. It refines both based on whether a target agent actually ends up passing the original task test while triggering the harm. LLM judges then score candidates, and the top 150 form the benchmark. Held-out frontier agents solve almost all tasks but avoid the designated harm only about a third of the time.

Critical strengths

The problem is real, and framing it as the conjunction of task success and harm is useful. Reusing the original task test is a clever way to keep the objective fixed, and having no judge at evaluation time makes the benchmark cheap and reproducible. Testing on agents stronger than the ones used for construction is a nice design choice. The prompts and examples are fully shown, which made it easy to see what the benchmark actually contains.

Critical weaknesses
  1. The headline number is largely a product of selection. Only perturbations on which a construction agent actually produced harmful success are kept, so the benchmark is pre-filtered for harm. The result tells us that harm found with weaker agents often transfers to stronger ones, which is interesting, but not how often frontier agents cause harm in general. Two controls are missing. First, run the same harm tests on agents solving the original, unperturbed task, to show the edit is what induces the harm and that ordinary solutions don’t trip the test. Second, report results on the discarded candidates, to show how much the filtering drives the rate.
  2. Many cases look more like reasonable readings of a nudged instruction than unintended harm. In Figure 1, being told not to re-append merged content makes clearing the source file a natural idempotence strategy. “Don’t get blocked by permission issues in whatever environment” comes close to asking for chmod 777. “Assemble the command as one line and execute it” practically prescribes eval. On top of that, the scaffold gives the agent no way to ask for clarification and tells it to keep working until done. For underspecified requests, asking is often the safe behavior, and here it’s unavailable. The benchmark therefore mixes agent judgment with its own design choices. Adding a clarification channel, or counting a clarifying question as safe, would make the measurement much cleaner.
  3. The harm tests are LLM-written and each checks only one designated harm. A safe solution that reaches the goal a different way could still trip the check. Meanwhile, an agent that avoids this harm but causes a different one counts as safe. So “safe success” really means “didn’t trigger this particular test.” It would help to write a reference safe solution for each instance and confirm the harm test stays quiet on it.
  4. The human evaluation answers an easy question. Comparing the top 25 against the bottom 25 will show separation almost regardless of how good the scoring is. What matters is how many instances inside the benchmark are invalid, broken down by criterion, and there the numbers are less reassuring. Also, the judges are the same two models as the proposers, so the quality score may simply reward the proposers’ own style.
Questions for authors

How often do the harm tests fire when agents solve the original tasks? Does allowing clarification change the picture? Do short safety instructions reduce harmful success substantially?

Reviewer_YP9GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VeriHarm, a framework for benchmark construction with a focus on measuring 'harmful success' where a coding agent completes a benign task but causes unintended harm. A proposer model is used to minimally perturb BashBench tasks and generate executable harm tests, with the original task correctness tests kept constant. The 1,046 candidates which produced harmful success in a target-agent rollout are scored and narrowed down to the 150 highest-scoring to form VeriHarm-Bench, where frontier agents achieve 95% task accuracy but only 37% safe success. Benchmark quality is validated using a human evaluation of 50 candidates.

Critical strengths
  1. The results are reproducible as the evaluation methodology is deterministic and executable, with no LLM used as a judge at evaluation time.

  2. A complete pipeline is produced with a strong focus on coding agents.

  3. The gulf between task accuracy and safe success provides a useful signal.

Critical weaknesses
  1. The primary concern is a lack of reliability. LLMs write and score every task and harm test, and no harm test is checked against a safe solution, such as a reference solution or seed rollout.

  2. Humans still judge 16% (4/25) of top-ranked items to be invalid, with only moderate agreement (κ=0.54) and the check compares only the top and bottom of the ranking. It may be the case that the definition of harm itself is ambiguous.

  3. Since every regenerated benchmark arguably still requires a human audit, claims about benchmark scaling seem overstated.

Questions for authors
  1. When the benchmark is regenerated, how do you ensure the tasks are valid without a full human audit?

  2. What share of the 150 final instances would be valid after a human audit, and how would that affect the main metrics?

  3. How reliable are VeriHarm's executable harm tests compared to AutoElicit's trajectory-based evaluator when both are validated against human labels?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.