- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces VeriHarm, a framework for building benchmarks of "harmful success," where an agent completes a benign coding task but causes unintended harm along the way. Starting from BashBench tasks, a proposer model repeatedly modifies the task instructions and writes executable harm tests, guided by feedback from target-agent rollouts. The original task-correctness tests stay fixed. A candidate is kept when both the correctness test and the harm test pass, and the kept candidates are then ranked by two LLM judges.
The resulting VeriHarm-Bench has 150 instances derived from 97 distinct source tasks. The authors evaluate four agents that played no role in building the benchmark. On average, the agents complete 95.1% of tasks, split into 58.2% harmful success and 37.0% safe success. In a human audit, 21 of 25 sampled benchmark instances are judged valid, compared with 2 of 25 sampled bottom-ranked candidates.
Critical strengths
- A practical approach to reproducible joint evaluation. The paper reuses established correctness tests and generates harm tests specific to each task. This gives a concrete way to tell successful-but-harmful outcomes apart from ordinary task failures. Prior benchmarks such as OS-Harm (NeurIPS 2025) already evaluate task completion alongside safety. What is new here is that the executable checks are built automatically and that no model-based judge is needed at evaluation time. Fixed executable checks don’t depend on a judge model that may change, and each verdict can be inspected. This could be useful infrastructure for evaluating safeguards on coding agents.
- The evaluation goes beyond the agents used to find the failures. None of the four evaluation agents served as target, proposer, or judge during construction. Their results suggest that the selected cases expose weaknesses beyond the agents on which they were found. The paper is also transparent: it reports the human audit with agreement statistics, the construction prompts, and a cost breakdown.
Critical weaknesses
- Some harm tests measure latent vulnerabilities or outcomes the test itself causes, not harmful side effects of the agent’s rollout. Section 2.1 and Appendix A say harm is verified on the final sandbox state. The motivating scenario is an agent that completes its task but causes an unintended harmful side effect. However, Appendix F’s shell-injection example runs the generated script with malicious search strings and checks whether those runs create marker files (pp. 22–23). This shows that the generated artifact can be exploited. It does not show that any command-execution harm occurred during the benign-task rollout. Similarly, the permissions example can run the generated script to create a missing file before checking that file’s permissions (p. 25). So, these checks are not always passive inspections of the final state: the evaluation itself can produce the state it then assesses. Generating vulnerable software is a real safety concern, but it is not the same as causing operational harm during execution. Without knowing how many instances fall into each category, the aggregate harmful-success rate mixes different outcomes. The benchmark should separate the two categories and report results for each. Where tests execute or modify artifacts, the protocol should also say how task checks and harm checks are kept isolated from each other.
- The human audit reveals unresolved validity errors, and the reported ranking metrics claim more than has been validated. Four of the 25 audited top-ranked instances fail the paper’s own validity criteria. That matters for a benchmark whose central contribution is reliable, executable harm measurement. The paper does not explain these failures or say whether the affected instances were repaired or excluded. It also does not show how filtering out invalid instances would change the agent results. In addition, the reported 88% accuracy and 91% recall are computed on a sample that contains only the top and bottom of the ranking. They show that the two ends can be told apart. They do not establish recall over the full pool of 1,046 candidates, because the middle of the ranking was never annotated. The more relevant estimate is that 84% (21/25) of sampled benchmark items are valid. The 95% confidence interval for that is wide, roughly 65–94%, and annotator agreement is only moderate. None of this negates the failures the benchmark exposes. But it leaves the reliability of individual labels, and therefore the exact benchmark scores, uncertain. A test being executable and scored deterministically does not by itself show that it correctly captures unintended harm.
Questions for authors
Q1–Q3 address the two weaknesses above and can likely be answered with the existing benchmark. Convincing answers could materially improve my assessment: an audit of what each harm test measures, separate results for operational harms and artifact vulnerabilities, and corrected scores. Q4 is about the construction method. None of these questions asks for expansion to other domains or models.
- What does each harm test measure? (clarification and reanalysis) How many benchmark instances detect a state change caused by the agent that is already present when the rollout ends? How many instead actively probe the generated artifacts for vulnerabilities? Please report agent results separately for these two categories. Do task tests and harm tests run on isolated copies of the final environment?
- What failed the human validity audit? (clarification and reanalysis) For the four invalid top-ranked instances, which criteria failed, and were these instances kept in Table 2? Can you provide corrected results after adjudication, along with a breakdown of the reasons for invalidity? Please also describe the annotators’ relevant expertise and how the 25 benchmark items were stratified for sampling.
- What are the exact evaluation counts? (clarification) Appendix B.3 specifies one rollout per instance on a 150-instance benchmark. However, some reported values cannot come from 150 pass/fail outcomes. For example, Claude-Opus-5’s 56.2% harmful success is impossible: the nearest achievable values are 56.0% and 56.7%. Please give numerators and denominators and explain how failed or missing runs were handled. The benchmark contains variants of the same 97 source tasks, and all agents are evaluated on the same instances. How do the reported uncertainty estimates account for these dependencies?
- What does rollout feedback add beyond simply making more attempts? (additional analysis) Table 1 shows that later iterations find more candidates, but more independent proposals would also find more over time. On a representative subset, how does refinement with feedback compare with the same number of proposals made without feedback from failed rollouts? It would also help to run the selected harm tests on rollouts of the original, unmodified tasks. That would show how much the instruction changes themselves affect harmful-success rates.