- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper argues for the importance of considering the security of agentic harnesses, and whether LLM agents themselves can find vulnerabilities in them. Its main contributions are:
- Collating a large dataset of these vulnerabilities
- Turning them into a benchmark, HarnessSecGym, which tests agents on whether they can find these vulnerabilities under different harness and prompting conditions
- Testing frontier agents and harnesses on HarnessSecGym, and analysing the results
The main findings from this analysis are:
- Agent's reviews have high precision but lower recall
- The main variance in whether a vulnerability is found comes from the type of vulnerability, not the LLM conducting the review
- There's evidence that the harness only matters for the quality of the review when the LLM is not told in which file the vulnerability is
Critical strengths
The paper explores a timely and important issue not covered (to the reviewers knowledge) by prior literature: how secure the harnesses used to turn LLMs into agents are, and if their vulnerabilities be found by agents autonomously. This is important due to increasing concerns around the safety of these agents, as well as reliance upon them for software engineering tasks and security work.
The paper is fairly comprehensive, as it aggregates a wide range of vulnerabilities, turns them into a useful benchmark with several types of review, and carefully evaluates many frontier models and harnesses with scientific rigour.
Critical weaknesses
The paper is written in a way that is generally hard to parse and confusing. This makes understanding the paper and its contributions difficult, time-consuming, and mentally taxing. Additionally, the excessive use of mid-sentence bolding is also bad style, and indicative that care has not been taken in some parts of the paper to lay out its prose such that these claims and points are naturally salient to the reader.
Some of the paper's main conclusions are over-stated compared to the empirical results. The experiments show that certain types of vulnerability are harder to detect, but do not actually establish that vulnerability type matters more than the choice of model as the examples in each category also differ in their code and difficulty. Similarly, the finding that the harness matters mainly for repository-wide review comes from testing one model across three harnesses, so it is unclear whether this finding generalises to other models. Whilst they are useful insights, these findings should be presented as patterns observed in this benchmark, rather than general features of agent performance. This limits the strength of the paper's conclusions, although it does not undermine the value of the benchmark itself.
Questions for authors
- Given the (acknowledged) use of AI in drafting of the paper, to what extent was it edited and checked by humans to make sure its contents made sense and properly reflected the views and understanding of the authors?
- What evidence supports the conclusion that vulnerability type matters more than model choice, beyond the difference in average detection rates? Can you distinguish this from differences in the repositories and difficulty of the examples?
- How were the 20 reports used to check the automated judge selected? Did they cover both missed planted defects and additional findings, including reports on control tasks?
6 comments
Authors' Response to Reviewer 7D6t (2/4)
Q1. Given the (acknowledged) use of AI in drafting of the paper, to what extent was it edited and checked by humans to make sure its contents made sense and properly reflected the views and understanding of the authors?
Re: The authors determined the paper's claims, structure and experiments, and checked every statement and number in it. In fact, this work contains large human efforts in collecting, reviewing, and verifying the data at different difficulty levels, and also for reviewing the evaluation results. For the LLM tools, it did assist with drafting and editing under the authors' direction. However, as the AI use statement says, the authors have reviewed all the content that was edited by LLMs and take responsibility for the content. For experiments, every experimental number is computed from per-run results by human-written scripts.
Authors' Response to Reviewer 7D6t (3/4)
Q2. What evidence supports the conclusion that vulnerability type matters more than model choice, beyond the difference in average detection rates? Can you distinguish this from differences in the repositories and difficulty of the examples?
Re: Beyond the average detection rates, we ran three checks. They show that the vulnerability type matters more than the model, and that this result does not come from the repositories or from the difficulty of the examples. The revised paper reports the three checks in Section 5 and Appendix O.
All three checks use the file-scope tasks (the tasks that name the file to review). On these tasks, the six LLM agents of the paper (four models in five harnesses) review the same 39 defects. The vulnerability type of a defect is what the paper calls its trust plane. A target is a defect planted in one task, so a defect that is planted in two tasks counts as two targets.
As the paper does, the checks compare two groups of vulnerability types: the three types that guard the model's inputs, its actions and the operator's view (INSTRUCTION, EXEC, RENDER), and the three conventional access-control types (NETWORK, FILESYSTEM, IDENTITY_POLICY). The first group holds 15 of the 39 defects, and the second group holds 20. The six LLM agents miss 60 of the 120 targets in the first group (50.0%) and 23 of the 144 targets in the second group (16.0%), a difference of 34.0 points.
1. Does the result depend on the defect or on the model? For each of the 39 defects and each of the six LLM agents, we recorded whether the agent found the defect. We then split the variance of these results into three parts: the differences between defects, the differences between LLM agents, and the rest. The differences between defects make up 68.6% of the variance, the differences between LLM agents make up 1.2%, and the rest is 30.2%. So whether a defect is found depends mostly on the defect, and very little on which LLM agent reviews it. Each of the six LLM agents also shows the difference between the two groups on its own: each agent misses 35% to 70% of the targets in the first group and 8% to 25% in the second group. We repeated the split of the variance on new data, in which a fifth model, GLM-5.3-Flash, reviews 78 defects in two rounds in each of two harnesses (Claude Code and Pi): the differences between defects make up 66.5% of the variance, and the differences between the four rounds make up 0.6%.
2. Does the result come from particular repositories? No. We left out each repository in turn and computed the difference between the two groups again. The difference stays between 30.0 and 40.4 points, so no single repository produces it. Two single types show the same. EXEC stays the most-missed type on the file-scope tasks whichever repository is left out (60.4% to 72.2% missed). INSTRUCTION is missed 44.4% of the time without the Codex repository and 45.2% with the Codex repository.
3. Does the result come from the difficulty of the examples? No. Random groupings of the same defects rarely give a difference of this size. To verify whether the difference between the two groups comes only from some examples being harder than others, we further conducted a permutation test that compares the grouping by vulnerability type with random groupings of the same defects. Specifically, if the difference of 34.0 points came only from some defects being harder than others, any two groups of 15 and 20 defects would often differ by as much. We therefore drew two groups of 15 and 20 defects at random from the 39 defects, 20,000 times, and computed the difference each time. The results show that only 1.5% of the random groupings give a difference of 34.0 points or more (). The grouping by vulnerability type gives a larger difference than almost all random groupings of the same defects, so the difficulty of the individual examples alone is unlikely to explain it.
Authors' Response to Reviewer 7D6t (1/4)
W. The paper is written in a way that is generally hard to parse and confusing. This makes understanding the paper and its contributions difficult, time-consuming, and mentally taxing. [...] these findings should be presented as patterns observed in this benchmark, rather than general features of agent performance. This limits the strength of the paper's conclusions, although it does not undermine the value of the benchmark itself.
Re:
1. Presentation. We thank the reviewer for this feedback. We removed the mid-sentence bold and edited the paper for readability. Bold now remains only in paragraph labels and for the best value in a table. Each research question opens with one plain sentence that states its result. The terms a reader needs are target, occurrence, source record, review scope, trust plane and claim validity. We define these terms in one place before the experiments (the Terms paragraph at the end of Section 4).
2. Strength of the conclusions. The paper draws two conclusions here, and we verified each of them as described below.
Conclusion 1: the vulnerability type matters more than the choice of model. The paper draws this conclusion from the file-scope tasks (the tasks that name the file to review). On these tasks, the share of planted defects missed ranges from 0% to 63% across the seven vulnerability types (the trust planes of the paper), and only from 19% to 33% across the six LLM agents that the paper evaluates (four models in five harnesses).
The reviewer notes that the examples of each type also differ in their code and difficulty. This cannot affect the comparison between the models, because all six LLM agents review the same examples: the same 39 planted defects. It can affect the comparison between the vulnerability types, because the examples of different types are different defects.
To verify whether the difference between the vulnerability types comes only from some examples being harder than others, we further conducted a permutation test that compares the grouping by vulnerability type with random groupings of the same defects.
Specifically, on the file-scope tasks, 15 of the 39 defects have the type INSTRUCTION, EXEC or RENDER, and the six LLM agents miss them 50.0% of the time. Another 20 defects have the type NETWORK, FILESYSTEM or IDENTITY_POLICY, and the six LLM agents miss them 16.0% of the time. The difference is 34.0 points. If this difference came only from some defects being harder than others, any two groups of 15 and 20 defects would often differ by as much. We therefore drew two groups of 15 and 20 defects at random from the 39 defects, 20,000 times, and computed the difference each time.
The results show that only 1.5% of the random groupings give a difference of 34.0 points or more (p=0.015). The grouping by vulnerability type gives a larger difference than almost all random groupings of the same defects, so the difficulty of the individual examples alone is unlikely to explain it.
To verify whether the difference depends on the particular set of examples, we further repeated the comparison in three ways. First, we left out each repository in turn: the difference stays between 30.0 and 40.4 points. Second, a fifth model, GLM-5.3-Flash, reviewed a larger set of 78 defects on file-scope tasks (the controlled experiment under Conclusion 2): it misses the defects of the first three types 45.0% of the time and those of the other three types 19.1% of the time, and the same test gives p=0.0043. Third, on all 170 tasks of the paper, which plant 95 defects, the four LLM agents that reviewed every task miss the defects of the two groups 57.4% and 26.7% of the time (p=0.0015). The results show that the difference appears in every one of these comparisons. Our reply to Q2 gives the full numbers.
Conclusion 2: the harness matters when the whole repository must be reviewed, and not when the task names the file. In the paper, DeepSeek V4.1 Flash shows this in three harnesses. We further conducted a controlled experiment with the fifth model, GLM-5.3-Flash, and verified that the conclusion holds for this model as well. Specifically, we planted each of 78 defects alone in three tasks: one task names the file, one names the module, and one asks for a review of the whole repository. GLM-5.3-Flash reviewed every task twice in Claude Code and twice in Pi. The results show that the two harnesses do not differ reliably when the task names the file (113 against 105 of 156 found, McNemar p=0.17), and that Claude Code finds more when the whole repository must be reviewed (31 against 18 of 156, p=0.015). The revised paper reports this experiment in Section 5 and Appendix N.
The paper provides both conclusions based on the experiments we conduct, and the revised paper says so explicitly (first paragraph of Section 5). We are glad that the reviewer finds that these points do not undermine the value of the benchmark itself.