acceptodds
Under review as a conference paper at ICLR 2027

HarnessSecGym: Can LLM Agents Find Real Vulnerabilities in Agent Harness Code?

Abstract

An LLM agent acts through its harness, the software that turns the model's text into actions. We argue that harness security is an emerging and critical problem that differs from both traditional software security and LLM security. Specifically, the harness treats the model as an authenticated caller even though any text the model reads can steer it, and because harness failures lie in code rather than in the model, aligning the model cannot repair them. Since testing that waits for a crash misses these failures and unit tests catch only the ones that someone anticipated, finding them calls for reviewing code, a task that LLM agents increasingly take on; we therefore ask whether they can find real harness vulnerabilities by reviewing harness source code. To answer this question, we collect 313 historical vulnerability records from 14 open-source harnesses, organize them by the protection they break, and build HarnessSecGym, which contains 170 review tasks at three review scopes, including unmodified control tasks. Evaluating six configurations that pair four frontier models with five harnesses, we find that the reviews of these LLM agents are systematically incomplete. Even when the task names the file, they miss 19% to 33% of the planted defects. They miss defects most often in two kinds of checks that guard the model's inputs and actions: checks on which commands and code the agent may run, where they miss 63% of the planted defects, and checks on which text the model receives as instructions, where they miss 45%. Moreover, the recall of one model on the same defects falls from 75% to 80% when the task names the file to 40% to 65% when the whole repository must be reviewed, depending on the reviewer's harness. Today's LLM agents therefore cannot yet be relied on to find vulnerabilities in agent harness code.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Agent Harness Security, Vulnerability Detection, Source Code Analysis, Security Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7D6tOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper argues for the importance of considering the security of agentic harnesses, and whether LLM agents themselves can find vulnerabilities in them. Its main contributions are:

  • Collating a large dataset of these vulnerabilities
  • Turning them into a benchmark, HarnessSecGym, which tests agents on whether they can find these vulnerabilities under different harness and prompting conditions
  • Testing frontier agents and harnesses on HarnessSecGym, and analysing the results

The main findings from this analysis are:

  • Agent's reviews have high precision but lower recall
  • The main variance in whether a vulnerability is found comes from the type of vulnerability, not the LLM conducting the review
  • There's evidence that the harness only matters for the quality of the review when the LLM is not told in which file the vulnerability is
Critical strengths

The paper explores a timely and important issue not covered (to the reviewers knowledge) by prior literature: how secure the harnesses used to turn LLMs into agents are, and if their vulnerabilities be found by agents autonomously. This is important due to increasing concerns around the safety of these agents, as well as reliance upon them for software engineering tasks and security work.

The paper is fairly comprehensive, as it aggregates a wide range of vulnerabilities, turns them into a useful benchmark with several types of review, and carefully evaluates many frontier models and harnesses with scientific rigour.

Critical weaknesses

The paper is written in a way that is generally hard to parse and confusing. This makes understanding the paper and its contributions difficult, time-consuming, and mentally taxing. Additionally, the excessive use of mid-sentence bolding is also bad style, and indicative that care has not been taken in some parts of the paper to lay out its prose such that these claims and points are naturally salient to the reader.

Some of the paper's main conclusions are over-stated compared to the empirical results. The experiments show that certain types of vulnerability are harder to detect, but do not actually establish that vulnerability type matters more than the choice of model as the examples in each category also differ in their code and difficulty. Similarly, the finding that the harness matters mainly for repository-wide review comes from testing one model across three harnesses, so it is unclear whether this finding generalises to other models. Whilst they are useful insights, these findings should be presented as patterns observed in this benchmark, rather than general features of agent performance. This limits the strength of the paper's conclusions, although it does not undermine the value of the benchmark itself.

Questions for authors
  1. Given the (acknowledged) use of AI in drafting of the paper, to what extent was it edited and checked by humans to make sure its contents made sense and properly reflected the views and understanding of the authors?
  2. What evidence supports the conclusion that vulnerability type matters more than model choice, beyond the difference in average detection rates? Can you distinguish this from differences in the repositories and difficulty of the examples?
  3. How were the 20 reports used to check the automated judge selected? Did they cover both missed planted defects and additional findings, including reports on control tasks?
6 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer 7D6t (1/4)

W. The paper is written in a way that is generally hard to parse and confusing. This makes understanding the paper and its contributions difficult, time-consuming, and mentally taxing. [...] these findings should be presented as patterns observed in this benchmark, rather than general features of agent performance. This limits the strength of the paper's conclusions, although it does not undermine the value of the benchmark itself.

Re:

1. Presentation. We thank the reviewer for this feedback. We removed the mid-sentence bold and edited the paper for readability. Bold now remains only in paragraph labels and for the best value in a table. Each research question opens with one plain sentence that states its result. The terms a reader needs are target, occurrence, source record, review scope, trust plane and claim validity. We define these terms in one place before the experiments (the Terms paragraph at the end of Section 4).

2. Strength of the conclusions. The paper draws two conclusions here, and we verified each of them as described below.

Conclusion 1: the vulnerability type matters more than the choice of model. The paper draws this conclusion from the file-scope tasks (the tasks that name the file to review). On these tasks, the share of planted defects missed ranges from 0% to 63% across the seven vulnerability types (the trust planes of the paper), and only from 19% to 33% across the six LLM agents that the paper evaluates (four models in five harnesses).

The reviewer notes that the examples of each type also differ in their code and difficulty. This cannot affect the comparison between the models, because all six LLM agents review the same examples: the same 39 planted defects. It can affect the comparison between the vulnerability types, because the examples of different types are different defects.

To verify whether the difference between the vulnerability types comes only from some examples being harder than others, we further conducted a permutation test that compares the grouping by vulnerability type with random groupings of the same defects.

Specifically, on the file-scope tasks, 15 of the 39 defects have the type INSTRUCTION, EXEC or RENDER, and the six LLM agents miss them 50.0% of the time. Another 20 defects have the type NETWORK, FILESYSTEM or IDENTITY_POLICY, and the six LLM agents miss them 16.0% of the time. The difference is 34.0 points. If this difference came only from some defects being harder than others, any two groups of 15 and 20 defects would often differ by as much. We therefore drew two groups of 15 and 20 defects at random from the 39 defects, 20,000 times, and computed the difference each time.

The results show that only 1.5% of the random groupings give a difference of 34.0 points or more (p=0.015p = 0.015). The grouping by vulnerability type gives a larger difference than almost all random groupings of the same defects, so the difficulty of the individual examples alone is unlikely to explain it.

To verify whether the difference depends on the particular set of examples, we further repeated the comparison in three ways. First, we left out each repository in turn: the difference stays between 30.0 and 40.4 points. Second, a fifth model, GLM-5.3-Flash, reviewed a larger set of 78 defects on file-scope tasks (the controlled experiment under Conclusion 2): it misses the defects of the first three types 45.0% of the time and those of the other three types 19.1% of the time, and the same test gives p=0.0043p = 0.0043. Third, on all 170 tasks of the paper, which plant 95 defects, the four LLM agents that reviewed every task miss the defects of the two groups 57.4% and 26.7% of the time (p=0.0015p = 0.0015). The results show that the difference appears in every one of these comparisons. Our reply to Q2 gives the full numbers.

Conclusion 2: the harness matters when the whole repository must be reviewed, and not when the task names the file. In the paper, DeepSeek V4.1 Flash shows this in three harnesses. We further conducted a controlled experiment with the fifth model, GLM-5.3-Flash, and verified that the conclusion holds for this model as well. Specifically, we planted each of 78 defects alone in three tasks: one task names the file, one names the module, and one asks for a review of the whole repository. GLM-5.3-Flash reviewed every task twice in Claude Code and twice in Pi. The results show that the two harnesses do not differ reliably when the task names the file (113 against 105 of 156 found, McNemar p=0.17p = 0.17), and that Claude Code finds more when the whole repository must be reviewed (31 against 18 of 156, p=0.015p = 0.015). The revised paper reports this experiment in Section 5 and Appendix N.

The paper provides both conclusions based on the experiments we conduct, and the revised paper says so explicitly (first paragraph of Section 5). We are glad that the reviewer finds that these points do not undermine the value of the benchmark itself.

AuthorsOpenReview ↗

Authors' Response to Reviewer 7D6t (2/4)

Q1. Given the (acknowledged) use of AI in drafting of the paper, to what extent was it edited and checked by humans to make sure its contents made sense and properly reflected the views and understanding of the authors?

Re: The authors determined the paper's claims, structure and experiments, and checked every statement and number in it. In fact, this work contains large human efforts in collecting, reviewing, and verifying the data at different difficulty levels, and also for reviewing the evaluation results. For the LLM tools, it did assist with drafting and editing under the authors' direction. However, as the AI use statement says, the authors have reviewed all the content that was edited by LLMs and take responsibility for the content. For experiments, every experimental number is computed from per-run results by human-written scripts.

AuthorsOpenReview ↗

Authors' Response to Reviewer 7D6t (3/4)

Q2. What evidence supports the conclusion that vulnerability type matters more than model choice, beyond the difference in average detection rates? Can you distinguish this from differences in the repositories and difficulty of the examples?

Re: Beyond the average detection rates, we ran three checks. They show that the vulnerability type matters more than the model, and that this result does not come from the repositories or from the difficulty of the examples. The revised paper reports the three checks in Section 5 and Appendix O.

All three checks use the file-scope tasks (the tasks that name the file to review). On these tasks, the six LLM agents of the paper (four models in five harnesses) review the same 39 defects. The vulnerability type of a defect is what the paper calls its trust plane. A target is a defect planted in one task, so a defect that is planted in two tasks counts as two targets.

As the paper does, the checks compare two groups of vulnerability types: the three types that guard the model's inputs, its actions and the operator's view (INSTRUCTION, EXEC, RENDER), and the three conventional access-control types (NETWORK, FILESYSTEM, IDENTITY_POLICY). The first group holds 15 of the 39 defects, and the second group holds 20. The six LLM agents miss 60 of the 120 targets in the first group (50.0%) and 23 of the 144 targets in the second group (16.0%), a difference of 34.0 points.

1. Does the result depend on the defect or on the model? For each of the 39 defects and each of the six LLM agents, we recorded whether the agent found the defect. We then split the variance of these results into three parts: the differences between defects, the differences between LLM agents, and the rest. The differences between defects make up 68.6% of the variance, the differences between LLM agents make up 1.2%, and the rest is 30.2%. So whether a defect is found depends mostly on the defect, and very little on which LLM agent reviews it. Each of the six LLM agents also shows the difference between the two groups on its own: each agent misses 35% to 70% of the targets in the first group and 8% to 25% in the second group. We repeated the split of the variance on new data, in which a fifth model, GLM-5.3-Flash, reviews 78 defects in two rounds in each of two harnesses (Claude Code and Pi): the differences between defects make up 66.5% of the variance, and the differences between the four rounds make up 0.6%.

2. Does the result come from particular repositories? No. We left out each repository in turn and computed the difference between the two groups again. The difference stays between 30.0 and 40.4 points, so no single repository produces it. Two single types show the same. EXEC stays the most-missed type on the file-scope tasks whichever repository is left out (60.4% to 72.2% missed). INSTRUCTION is missed 44.4% of the time without the Codex repository and 45.2% with the Codex repository.

3. Does the result come from the difficulty of the examples? No. Random groupings of the same defects rarely give a difference of this size. To verify whether the difference between the two groups comes only from some examples being harder than others, we further conducted a permutation test that compares the grouping by vulnerability type with random groupings of the same defects. Specifically, if the difference of 34.0 points came only from some defects being harder than others, any two groups of 15 and 20 defects would often differ by as much. We therefore drew two groups of 15 and 20 defects at random from the 39 defects, 20,000 times, and computed the difference each time. The results show that only 1.5% of the random groupings give a difference of 34.0 points or more (p=0.015p = 0.015). The grouping by vulnerability type gives a larger difference than almost all random groupings of the same defects, so the difficulty of the individual examples alone is unlikely to explain it.

Reviewer_d1GdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that the harness of an LLM agent is a distinct security boundary, differing from both classic software security and model-centric LLM security. The harness is the code that assembles the model's context, mediates its tool calls, holds settings and credentials, authenticates controllers and renders information to the operator. The paper makes this argument through eight "trust planes" and 21 case cards drawn from real fixes. It then builds HarnessSecGym which contains 313 historical vulnerability records from 14 open-source harnesses (312 eligible). These records are tiered by evidence: 76 advisory, 88 fix-convention and 148 keyword-only. Then a judge (Claude Opus 5 in Claude Code) grades the recall and claim validity which was also validated by human audit of 20 reports.

The experiments cover six agent configurations, pairing four models with five harnesses. All six review the 53 file-scope tasks. Four configurations review all 170 tasks, with DeepSeek V4.1 Flash run in three harnesses to vary the harness at fixed model. Results are as following:

  • RQ1: file-scope repository-macro recall is 76.8–88.3% with overlapping CIs, while claim validity is 97–100%. Over all tasks, recall falls to 55–67%.
  • RQ2: misses concentrate in EXEC (63% pooled) and INSTRUCTION (45%) and are absent in CONFIG_STATE (0%) at file scope; the paper concludes that "the blind spots follow the protection, not the model".
  • RQ3: at fixed model, the reviewer's harness hardly matters at file scope (75–80% on 20 shared records) but strongly at repository scope (40–65%).
Critical strengths
  1. A real-fix benchmark with verified ground truth, controls and leakage-proof packaging. Every record is backed by a reintroducing injector, a security test that passes on fixed and fails on injected code with unrelated tests still passing, and a reachability review. The conditions are demanding: in Hermes, "only 118 of 428 candidates with tests met them". Tasks keep the full repository at every scope, so scope changes only how much must be searched. The prompt never reveals how many defects exist. Controls are matched to injected files (each file-scope control "asks for a review of a file that also appears, with planted defects, in at least one injected file-scope task"). Packaging strips tests, history, changelogs, security notes and fix-keyed static-analysis rules, and the build fails if an advisory identifier survives (p. 20). This is the right level of care for a benchmark whose answers are public commits. Confidence intervals use a cluster bootstrap that resamples tasks sharing a source record or file together (p. 8). That correctly accounts for the fact that 205 occurrences come from only 95 records.

  2. Analyses that explain how reviews fail, not only how often. The tool-call logs classify each miss as opened, seen-only, never seen, or no report (Table 5). At repository scope the dominant failure differs by harness: 79% never-opened for Grok, 34% for DeepSeek in Claude Code. Appendix J identifies a shared failure mode: in all three universally missed defects, the check still exists but no longer covers every case. One example is the smolagents CVE-2025-5120 wrapper "defined but unused". The matched control comparison shows that unconfirmed claims concentrate on controls: 6.9% of control claims vs 1.0% on the injected versions of the same files.

Critical weaknesses
  1. The judge shares model and harness with two participants, is validated on 20 reports, and the data contain a harness-dependent false-alarm pattern that coincides with this overlap.

    • (a) Unexplained harness effect on false alarms at file scope. The paper states that the reviewer's harness matters at repository scope "not when the task names the file", but this holds only for recall. For the same model at file scope, unconfirmed claims on controls are:

      • 1/78 in Claude Code;
      • 11/94 in OpenCode (p=0.007p = 0.007, two-sided Fisher);
      • 6/44 in Pi (p=0.009p = 0.009).

      Both Claude Code configurations have 22/23 clean controls, every other configuration 14–18/23. Possible explanations:

      • Claude Code's prompting genuinely makes models more careful;
      • the judge, "Claude Opus 5 in Claude Code at maximum reasoning effort", is more charitable toward reports produced in its own harness's idiom.

      The current validation cannot distinguish the two. The 20-report audit (95% CI 83–100%) is not stratified by configuration or task type, and the paper does not say how those reports were chosen. With 20 reports, it cannot detect a bias of the size that would matter here. The limitation section acknowledges that "this agreement cannot guarantee that the judge is free of errors or biases" but does not test for this specific risk.

    • (b) What does "confirmed" mean? On injected tasks, OpenCode/DeepSeek has 889 of 923 claims confirmed while finding 130 of 205 planted targets. That implies roughly 760 confirmed non-planted weaknesses across 126 tasks, plus 352 confirmed claims on 44 unmodified controls; Grok and Claude Code/DeepSeek are similar. Either these pinned snapshots of 11 popular harnesses contain hundreds of real, unpatched weaknesses, or "really present" (Appendix E) is a lenient standard: premise true, mechanism possible, but not an exploitable vulnerability. If the former, recall on planted defects measures a small, arbitrary subset of what is there. If the latter, the 95–100% claim-validity figures overstate review quality, and the RQ1 framing "Reviews are valid but incomplete" needs qualification. The headline claim-validity numbers are also partly mechanical: "the claim rubric counts each such finding as one confirmed assertion", so 205 of the 596 file-scope injected-task claims are confirmed by construction, and controls are excluded from Table 2.

  2. The reviewer-harness conclusion (RQ3) and the abstract's "75% to 80% … 40% to 65%" rest on 20 records in single runs, and the full repository-scope data tell a somewhat different story.

    • (a) On the 20 records reviewed at both scopes, recall at file scope is 75.0/80.0/80.0% and at repository scope 65.0/40.0/47.5% (OpenCode/Pi/Claude Code). Differences of 5 percentage points are one record. The repository-scope spread corresponds to about 5 records. No interval or paired test is given, and each task was run once ("Each LLM agent reviews each of its tasks once"), so run-to-run variance of agentic reviews is not separated from the harness effect.

    • (b) Over all 32 injected repository tasks, the per-defect-count breakdowns in Appendix G give task recall of:

      • OpenCode 59%;
      • Pi 45%;
      • Claude Code 46%;
      • Grok 39%.

      Pi and Claude Code are indistinguishable. The harness effect reduces to "OpenCode is better than the other two", for which no mechanism is identified.

    • (c) Pi's repository-scope result is confounded with its time-outs: Pi has all 12 of its unusable reports from the 60-minute limit, and the harnesses spend different amounts of time ("The agent phase averaged 12.0 minutes in Pi and 17.3 minutes in Claude Code"). Without matched budgets, the harness effect is not separated from the time spent. The Grok 87.5% → 30.0% drop cannot be attributed to the harness, because Grok is only run in Grok Build.

Questions for authors
  1. Can you test the judge for same-family and same-harness bias? Concretely:

    • What explains the file-scope control result for DeepSeek V4.1 Flash, with 1/78 unconfirmed claims in Claude Code vs 11/94 in OpenCode and 6/44 in Pi, and 22/23 clean controls for both Claude Code configurations vs 14–18/23 for all others?
    • Would you re-grade a stratified subsample with a judge from a different model family, and extend the human audit beyond 20 reports, stratified by configuration?
    • How were the 20 audited reports selected?
  2. What fraction of the ~760 confirmed non-planted weaknesses (OpenCode/DeepSeek, injected tasks) and the 352 confirmed weaknesses on unmodified controls would a human security reviewer classify as exploitable vulnerabilities? If many are real, were they disclosed upstream? If few are, how should the 95–100% claim-validity figures be interpreted?

  3. How robust are the per-plane and per-harness conclusions to the unit of analysis? Specifically:

    • Can you report plane-level miss rates with intervals computed over targets (not runs) and a leave-one-repository-out version, in particular excluding Codex for INSTRUCTION?
    • Can you report RQ3 over all 32 injected repository tasks, with paired intervals and at least one repeated run per configuration?
    • How do the headline numbers change when restricted to advisory- and convention-tier targets, or when neutralizing injectors are excluded?
10 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer d1Gd (1/8)

Response to W1

Re: (a) Is the judge more charitable toward reports from its own harness? No. The judge is not more charitable toward these reports, and the fact that it shares its model and harness with two participants does not affect the paper's results. We tested this with a second judge and with a larger human audit.

A second judge. To test whether the grades depend on the paper's judge, we graded all 303 usable file-scope reports again with a judge from another model family in another harness (GPT-5.6-sol in Codex), and twice more with the paper's judge. The results show three things.

  1. Recall does not change. The four gradings give the same 274 verdicts on planted defects (found or missed).
  2. The difference between harnesses on controls is still there under the second judge. For DeepSeek V4.1 Flash, the second judge leaves 3 of 81 claims on controls unconfirmed in Claude Code (3.7%), 7 of 107 in OpenCode (6.5%) and 6 of 47 in Pi (12.8%). The paper's judge leaves 1 of 78 unconfirmed in Claude Code (1.3%), 11 of 94 in OpenCode (11.7%) and 6 of 44 in Pi (13.6%). The three harnesses are in the same order under both judges, so this difference does not come from the paper's judge. Both Claude Code configurations also remain among the configurations with the most clean controls under the second judge (20 of 23 each). The counts are small, 1 to 12 unconfirmed claims per configuration, so the two judges differ by a few claims.
  3. The unconfirmed claims on controls do not enter the paper's conclusions. The paper's primary measure is recall on planted defects. The ground truth of a planted defect is a test, and the judge only has to match a finding to the known defect. Claim validity on tasks with planted defects stays between 95.1% and 100% for every configuration under every grading. Under both judges, the unconfirmed claims are mostly on controls: the second judge leaves 4.7% of the claims on controls unconfirmed, against 1.6% on tasks with planted defects, and the paper's judge leaves 6.9% unconfirmed on controls, against 1.0%.

Unconfirmed claims on the 23 file-scope controls (unconfirmed / claims):

ConfigurationPaper's judgeSame judge, second gradingSame judge, third gradingGPT-5.6-sol in Codex
Claude Code, Claude Opus 51/70 (1.4%)2/70 (2.9%)3/69 (4.3%)4/78 (5.1%)
Codex, GPT-5.6-sol8/141 (5.7%)6/141 (4.3%)5/140 (3.6%)1/140 (0.7%)
Grok Build, Grok 4.610/107 (9.3%)12/106 (11.3%)7/106 (6.6%)5/106 (4.7%)
OpenCode, DeepSeek V4.1 Flash11/94 (11.7%)10/98 (10.2%)10/91 (11.0%)7/107 (6.5%)
Pi, DeepSeek V4.1 Flash6/44 (13.6%)6/44 (13.6%)6/44 (13.6%)6/47 (12.8%)
Claude Code, DeepSeek V4.1 Flash1/78 (1.3%)1/78 (1.3%)2/78 (2.6%)3/81 (3.7%)
Controls with no unconfirmed claim, for the six configurations from top to bottom22, 14, 18, 17, 14, 2221, 14, 17, 17, 13, 2220, 16, 19, 16, 13, 2120, 19, 20, 17, 14, 20

Codex has 20 usable control reports and Pi has 19. The other configurations have 23.

A larger human audit. The audit in the paper uses a random sample of reports, stratified by configuration and by review scope. Its 20 reports are 8 file-scope, 8 module-scope and 4 repository-scope reports, spread evenly over the configurations within each scope, and the auditors agreed with the judge on all 20 reports. Following the reviewer's suggestion, we further extended this audit. Three researchers with experience in building LLM agents and a background in security and safety audited 50 further reports that the first audit does not cover (20 file-scope, 19 module-scope and 11 repository-scope reports). The two configurations that are evaluated only at file scope contribute 4 reports each, and the other four configurations contribute 10 or 11 each. For each report, each auditor checked whether the report finds each planted defect and whether it contains unconfirmed claims. On these 50 reports, the judge gave 59 verdicts on planted defects (39 found, 20 missed) and 50 verdicts on unconfirmed claims, one per report (9 reports contain unconfirmed claims). Fourteen of the 50 reports are on control tasks. The majority of the three auditors agrees with the judge on all 50 reports. Over both audits, the auditors agree with the judge on all 70 reports, which gives an exact 95% confidence interval of 94.9% to 100% for the agreement rate (83% to 100% for the first 20 reports).

The revised paper describes both audits in Appendix M (Table 8) and reports the controls under both judges in Appendix M (Table 10).

AuthorsOpenReview ↗

Authors' Response to Reviewer d1Gd (2/8)

Response to W1

Re: (b) "Confirmed" means that the weakness that a report claims is really present in the code: the premise of the claim matches the code, and the mechanism that it describes can occur. This is the standard of a security code review, which reports weaknesses. It does not mean that an exploit was demonstrated. The revised paper states this definition next to the measure (Section 4, Grading). For a detection task, we consider it reasonable that a reviewer raises potential vulnerabilities, even if some of them turn out not to be exploitable or not to be real. In practice, such an agent serves as an early warning, and what it raises is followed up. What matters most is whether the agent finds the vulnerability that is there, because a miss can have severe consequences. This is what the paper measures first, with recall on the planted defects. The paper uses claim validity in RQ1 to show that the reports are not padded with false statements. We answer the reviewer's four points in turn.

  • "Recall on planted defects measures a small, arbitrary subset." The planted defects are not an arbitrary subset. Each planted defect reverts a fix that the project's maintainers made for a security reason, and it comes with a test that fails when the fix is reverted and with a review that confirms that an attacker can reach the defect (a reachability review). Other weaknesses in these snapshots have neither a test nor a reachability review, so the paper does not count them toward recall.
  • "'Really present' is a lenient standard." The judge rejects a claim when its premise misstates the code, when its mechanism cannot occur, or when the code does not support it. The two rejected examples in Appendix J (an unescaped template value and a shell command) describe the cited lines correctly, but the judge rejected both claims because attacker input cannot reach those lines. The second judge, from another model family, confirms almost the same share of claims (96.9%, against 96.2% for the paper's judge).
  • The additional findings. The findings beyond the planted defects point at 1,607 distinct locations in source files that hold 677,356 lines, which is beyond what can be verified by hand. We will therefore release all participant reports publicly with the benchmark, so that the community can investigate these findings further (see Q2).
  • "The claim-validity numbers are partly mechanical, and controls are excluded from Table 2." Appendix J already gives file-scope claim validity without the 205 findings of planted defects. Only 6 of the other 391 claims are unconfirmed (98.5% confirmed), against 37 of 534 claims on controls. Over all 170 tasks, claim validity changes by at most 1.2 points when the findings of planted defects are removed:
ConfigurationClaim validity (all claims on tasks with planted defects)Other claims confirmed (without the findings of planted defects)Claims on controls confirmed
OpenCode, DeepSeek V4.1 Flash889/923 (96.3%)759/793 (95.7%)352/377 (93.4%)
Grok Build, Grok 4.6935/988 (94.6%)828/881 (94.0%)285/313 (91.1%)
Pi, DeepSeek V4.1 Flash379/392 (96.7%)273/286 (95.5%)148/164 (90.2%)
Claude Code, DeepSeek V4.1 Flash1,008/1,041 (96.8%)886/919 (96.4%)341/352 (96.9%)

The revised paper gives these columns in Table 7 (Appendix J) and adds the corresponding two columns at file scope to Table 2. What the reports claim is almost always present in the code, yet the reports miss a large share of the planted defects. This is what RQ1 says: the claims of a review are valid, and the review is still incomplete.

AuthorsOpenReview ↗

Authors' Response to Reviewer d1Gd (3/8)

Response to W2

Re: The conclusion of RQ3 does not rest on the 20 records alone. We further conducted a controlled experiment on 78 defects, with paired tests and a repeated run, and verified both parts of the conclusion: the harness makes no reliable difference when the task names the file, and it makes a difference when the whole repository must be reviewed. The full repository-scope data of the paper show the same: the harnesses differ at repository scope and not reliably at file scope. The revised paper adds this experiment to RQ3 (Section 5; Appendix N, Table 11).

The experiment. For each of 78 defects of the benchmark, we planted the defect alone in three tasks: one task names the file, one names the module, and one asks for a review of the whole repository (234 tasks). Every defect is thus reviewed at all three scopes, so a difference between scopes cannot come from different defects. GLM-5.3-Flash, a fifth model that is not among the paper's configurations, reviewed the 234 tasks in Claude Code and in Pi. To measure the run-to-run variance, GLM-5.3-Flash reviewed every task a second time in each harness. We call one complete review of the 234 tasks in one harness a round.

(a) Intervals, paired tests and run-to-run variance. The table gives, for each harness and round, how many of the 78 defects GLM-5.3-Flash finds at each scope.

GLM-5.3-Flash inFileModuleRepositoryChange from file to repository scope
Claude Code, first round54/78 (69.2%)27/78 (34.6%)15/78 (19.2%)-50.0 points (95% CI -62.8 to -37.2), p=8.7×10−10p = 8.7 \times 10^{-10}
Pi, first round53/78 (67.9%)33/78 (42.3%)6/78 (7.7%)-60.3 points (95% CI -70.5 to -50.0), p=1.4×10−14p = 1.4 \times 10^{-14}
Claude Code, second round59/78 (75.6%)25/78 (32.1%)16/78 (20.5%)-55.1 points (95% CI -66.7 to -43.6), p=2.6×10−12p = 2.6 \times 10^{-12}
Pi, second round52/78 (66.7%)27/78 (34.6%)12/78 (15.4%)-51.3 points (95% CI -62.8 to -39.7), p=2.0×10−11p = 2.0 \times 10^{-11}
Claude Code against Pi, first round (defects found by only one of the two harnesses, McNemar test)8 vs 7, p=1.0p = 1.05 vs 11, p=0.21p = 0.2110 vs 1, p=0.012p = 0.012
Claude Code against Pi, second round9 vs 2, p=0.065p = 0.0658 vs 10, p=0.82p = 0.829 vs 5, p=0.42p = 0.42
Claude Code against Pi, both rounds17 vs 9, p=0.17p = 0.1713 vs 21, p=0.23p = 0.2319 vs 6, p=0.015p = 0.015

The results show three things.

  1. Recall drops with the scope in every round. The same 78 defects are found 50.0 to 60.3 points less often at repository scope than at file scope (p<10−9p < 10^{-9} in each of the four rounds).
  2. The two harnesses differ at repository scope and not at file scope. Over both rounds, each harness reviews each defect twice at each scope, which gives 156 runs. At file scope, Claude Code finds the defect in 113 of the 156 runs and Pi in 105, a difference of 5.1 points (95% CI -1.3 to 12.2, p=0.17p = 0.17). At module scope, the difference is -5.1 points (-12.2 to 1.3, p=0.23p = 0.23). At repository scope, Claude Code finds the defect in 31 runs and Pi in 18, a difference of 8.3 points (1.9 to 15.4, p=0.015p = 0.015). In the first round alone, the two harnesses find 54 and 53 of the 78 defects at file scope (p=1.0p = 1.0), and 15 and 6 at repository scope (p=0.012p = 0.012).
  3. The run-to-run variance is small next to the effect of the scope. The second round gives the same result (found or missed) as the first round for 74% to 87% of the defects at every scope: for 61, 58 and 67 of the 78 defects at file, module and repository scope in Claude Code, and for 65, 66 and 68 in Pi. Between the two rounds, recall changes by at most 7.7 points at any scope. From file scope to module and repository scope, recall drops by 26 to 60 points in both rounds.

At file scope, GLM-5.3-Flash finds 67% to 76% of the defects. This is comparable with DeepSeek V4.1 Flash on the paper's file-scope tasks (67% to 73%).

Reviewer_mYCzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HarnessSecGym studies whether LLM agents can find security flaws in agent harness code. It consists of 170 tasks based on historically grounded defects and unmodified controls. The experiments provide credible evidence that the tested configurations leave many seeded vulnerabilities unreported. In particular, the evaluation covers six configurations combining four models with five harnesses. Agents miss 19-33% of the seeded defects even when given the relevant file. The most frequently missed defects at file scope concern execution controls and instruction handling.

Critical strengths

The paper provides a systematic way to evaluate security review at the software boundary that governs an agent’s authority. The benchmark spans different harness repositories with different designs and use cases covering a broader agentic use cases. Further, the seeded defects undergo differential testing, attacker-reachability checks, and human review. Overall, this combination of validation and diversity strengthens the benchmark’s credibility and usefulness in the community.

Critical weaknesses
  • The analysis does not clearly distinguish search failures from recognition failures. The paper treats a file path appearing in tool-call arguments as evidence that the agent opened the file. This does not establish that the agent saw the defective lines or the context needed to understand them. The command could read another region, might fail, or return truncated output.
  • The main results depend on a single LLM judge, but human validation covers only 20 reports. Further, the paper does not explain how these reports were selected or how many individual grading decisions were checked.
Questions for authors
  • How were the 20 audited reports selected, how many independent grading decisions did they contain?
  • Among the missed defects classified as “opened,” how often did the agent actually receive the defective lines and the context needed to assess them?
5 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer mYCz (1/4)

W1. The analysis does not clearly distinguish search failures from recognition failures. The paper treats a file path appearing in tool-call arguments as evidence that the agent opened the file. This does not establish that the agent saw the defective lines or the context needed to understand them. The command could read another region, might fail, or return truncated output.

Re: Thanks for your valuable feedback! We argue that the paper does distinguish search failures from recognition failures. For every missed defect, Table 5 (Appendix I) records whether the agent opened the defect's file, only saw it in a tool output, or never saw it. At file scope, the agent opened the file in every miss, so every file-scope miss is a recognition failure. At repository scope, 34% to 79% of the misses lie in files that the agent never opened and are search failures (Appendix H). We thank the reviewer for the suggestion to look inside the opened files. Additionally, for every miss, we checked whether the defective lines themselves were returned to the agent. The revised paper includes this analysis (Appendix P, Table 16; summarized in Section 5 and Appendix I).

1. At file scope, the defective lines were shown to the agent in every miss, which confirms that file-scope misses are recognition failures. Specifically, the logs of the runs record the text that every tool call returned to the agent, which is exactly what the agent saw. For each miss with an opened file, we tested whether the defective lines appear in that text. If a command reads another region, fails, or returns truncated output, the defective lines do not appear in that text, so the test rules out all three cases that the reviewer names. The results show that the defective lines were shown to the agent in all 69 misses with an opened file at file scope, 70 of the 95 at module scope, and 30 of the 58 at repository scope.

ScopeConfigurationMisses with the file opened (Table 5)Defective lines shown to the agent
Fileall six6969
ModuleGrok Build, Grok 4.61815
ModuleOpenCode, DeepSeek V4.1 Flash1612
ModulePi, DeepSeek V4.1 Flash3123
ModuleClaude Code, DeepSeek V4.1 Flash3020
RepositoryGrok Build, Grok 4.696
RepositoryOpenCode, DeepSeek V4.1 Flash125
RepositoryPi, DeepSeek V4.1 Flash128
RepositoryClaude Code, DeepSeek V4.1 Flash2511

At module and repository scope, an agent that opens a file does not always read the region that holds the defect, because it may search within the file or read only part of the file. In such a miss, the defective lines were not shown to the agent. This confirms the paper's conclusion that the larger the scope, the more misses are search failures. As a check of the test itself, the tool output shows all defective lines for 525 of the 531 defects that the agents did find, and part of the defective lines for the other 6.

2. We further conducted a controlled experiment on 78 defects at three review scopes, and verified that about two thirds of the repository-scope misses are search failures. Specifically, for each of 78 source records, we planted its defect alone in three tasks (234 tasks): one task names the file, one names the module, and one asks for a review of the whole repository. The three tasks have the same code and the same defect, and only the scope differs. One model (GLM-5.3-Flash) reviewed every task in Claude Code and in Pi. The results show that 42 of the 63 defects that Claude Code misses at repository scope, and 47 of the 72 that Pi misses, are found by the same agent once the task names the file. The agent recognizes these defects when it is pointed to the file, so these misses are search failures. The agents miss the remaining 21 defects in Claude Code and 25 in Pi even when the task names the file. The revised paper reports this experiment in Section 5 and Appendix N (Table 11).

HarnessFileModuleRepositoryRepository-scope misses that the same agent finds when the task names the file
Claude Code54/78 (69.2%)27/78 (34.6%)15/78 (19.2%)42 of 63
Pi53/78 (67.9%)33/78 (42.3%)6/78 (7.7%)47 of 72

Claude Code finds 54 of the 78 defects when the task names the file, and 15 of the same 78 when the user is asking to review the entire code repo. Pi finds 53 and 6.

AuthorsOpenReview ↗

Authors' Response to Reviewer mYCz (2/4)

W2. The main results depend on a single LLM judge, but human validation covers only 20 reports. Further, the paper does not explain how these reports were selected or how many individual grading decisions were checked.

Re: Thanks for rising this question. To solve your concern, we re-graded all 303 usable file-scope reports three more times: once with an independent judge from another model family in another harness (GPT-5.6-sol in Codex) and twice with the original judge. The four gradings agree on all 274 verdicts on planted defects. For the human validation, the 20 reports in the paper are a stratified random sample over the three review scopes and the six configurations. For each report, the auditors agreed with the judge on both of its outcomes: the planted defects that the report identifies, and whether it contains unconfirmed claims. Our reply to Q1 gives the details. Following the reviewer's suggestion, we also extended the human audit to 70 reports, and the auditors agree with the judge on all 70 reports.

1. Single judge: Recall is the paper's main measure, and the independent judge reproduces every file-scope recall number in the paper (Table 2, Figure 5; Appendix M, Table 9 of the revised paper). This follows from the design. For each planted defect, a test that passes on the fixed code and fails on the injected code fixes the ground truth. The judge only decides whether a finding names that defect by mechanism and location.

GradingVerdicts on planted defects equal to the paper'sClaims countedConfirmed
Paper: Claude Opus 5 in Claude Code(reference)1,13096.2%
Same judge, second grading274/2741,13696.3%
Same judge, third grading274/2741,12796.5%
GPT-5.6-sol in Codex274/2741,17996.9%

Claim validity on tasks with planted defects stays between 95.1% and 100% for every configuration under every grading. The share of unconfirmed claims on controls is 6.9% under the paper's judge and 4.7% under the independent judge. The revised paper reports the control results under both judges (Appendix M, Table 10; details in our reply to Reviewer d1Gd, W1a).

2. Human validation: The auditors agree with the judge on all 20 reports in the paper and on all 50 reports in the extended audit. The 20 audited reports are a stratified random sample over the three review scopes and the six configurations. The sample has 8 file-scope, 8 module-scope and 4 repository-scope reports, in proportion to the runs at each scope. Within each scope, the reports are spread evenly over the configurations. Our reply to Q1 gives the distribution. For each report, the auditors checked the judge's two outcomes against the reviewed code: whether the report identifies each planted defect, and whether the report contains unconfirmed claims. These 20 reports hold 21 verdicts on planted defects and 20 verdicts on unconfirmed claims, one per report. Following the reviewer's suggestion, we extended the audit beyond 20 reports. Specifically, three researchers with experience in building LLM agents and a background in security and safety audited 50 further reports that the first audit does not cover. The 50 new reports are stratified in the same way (20 file-scope, 19 module-scope and 11 repository-scope reports) and hold 59 verdicts on planted defects and 50 verdicts on unconfirmed claims, one per report. All three auditors judged each report, and their majority judgment agrees with the judge on all 50 reports. Over both audits, the auditors agree with the judge on all 70 reports, which gives an exact 95% confidence interval of 94.9% to 100% for the agreement rate. The revised paper describes both audits in Section 4 and Appendix M (Table 8).

AuthorsOpenReview ↗

Authors' Response to Reviewer mYCz (3/4)

Q1. How were the 20 audited reports selected, how many independent grading decisions did they contain?

Re: The 20 reports are a stratified random sample over the three review scopes and the six configurations. For each report, the auditors checked the judge's two outcomes against the reviewed code: whether the report identifies each planted defect, and whether the report contains unconfirmed claims. The 20 reports hold 21 verdicts on planted defects and 20 verdicts on unconfirmed claims, one per report. The auditors agreed with the judge on all 20 reports. The sample has 8 file-scope, 8 module-scope and 4 repository-scope reports, in proportion to the runs at each scope (318, 296 and 172 of 786). Within each scope, the reports are spread evenly over the configurations:

ConfigurationFileModuleRepositoryTotal
Claude Code, Claude Opus 52--2
Codex, GPT-5.6-sol2--2
Grok Build, Grok 4.61214
OpenCode, DeepSeek V4.1 Flash1214
Pi, DeepSeek V4.1 Flash1214
Claude Code, DeepSeek V4.1 Flash1214
Total88420

Claude Code with Claude Opus 5 and Codex with GPT-5.6-sol are evaluated at file scope only. Of the 21 verdicts on planted defects, 12 are "found" and 9 are "missed". Five of the 20 reports contain unconfirmed claims and 15 do not. Twelve of the reports are on tasks with planted defects and eight are on control tasks. We also extended the audit by 50 reports, and the auditors agreed with the judge on all 50 (our reply to W2). Table 8 in Appendix M of the revised paper gives the composition of both samples.

Additionally, we re-graded all 303 usable file-scope reports with an independent judge (our reply to W2). These 303 reports hold 274 verdicts on planted defects, and the independent judge agrees with the paper's judge on all 274. They also hold more than 1,100 verdicts on claims, and the two human judges confirm 96.2% and 96.9% of them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank the reviewers for their careful and constructive reviews. We summarize the strengths that they identify, answer the two concerns that more than one reviewer raises, and list the changes in the revised paper.

Strengths that the reviewers identify.

  • A timely problem that prior work does not cover. Reviewer 7D6t writes that the paper "explores a timely and important issue not covered (to the reviewers knowledge) by prior literature". Reviewer mYCz writes that the paper "provides a systematic way to evaluate security review at the software boundary that governs an agent's authority".
  • Verified ground truth and careful construction. Reviewer d1Gd describes "a real-fix benchmark with verified ground truth, controls and leakage-proof packaging" and calls this "the right level of care for a benchmark whose answers are public commits". Reviewer mYCz notes that the planted defects "undergo differential testing, attacker-reachability checks, and human review", which "strengthens the benchmark's credibility and usefulness in the community".
  • Breadth. The benchmark "spans different harness repositories with different designs and use cases" (mYCz), and the paper "carefully evaluates many frontier models and harnesses with scientific rigour" (7D6t).
  • Analyses that explain failures. Reviewer d1Gd values "analyses that explain how reviews fail, not only how often", the matched controls, and the cluster bootstrap that "correctly accounts for" repeated source records. Reviewer 7D6t adds that the concerns about two conclusions do "not undermine the value of the benchmark itself".

Concerns that more than one reviewer raises.

1. Do the main results depend on a single LLM judge, and is the human validation limited to 20 reports? (mYCz, d1Gd, 7D6t) No. We graded all 303 usable reports of the file-scope tasks (the tasks that name the file to review) again with an independent judge from another model family (GPT-5.6-sol in Codex). The two judges give the same 274 verdicts on planted defects (205 found, 69 missed), so every recall number is unchanged. A test fixes the ground truth of each planted defect, so the judge only checks whether a finding matches that defect. We further extended the human audit from 20 to 70 reports, drawn at random within each review scope and each LLM agent. The auditors agree with the judge on all 70 reports (95% CI: 94.9% to 100%).

2. Are the conclusions on vulnerability types and on the harness supported strongly enough? (d1Gd, 7D6t) Yes.

  • Conclusion 1: the vulnerability type matters more than the choice of model. All six LLM agents review the same 39 defects on the file-scope tasks, so the comparison between the models does not depend on the examples. To verify whether the difference between the vulnerability types comes only from some examples being harder than others, we further conducted a permutation test. Specifically, the LLM agents miss 50.0% of the planted defects of the types INSTRUCTION, EXEC and RENDER (15 defects) and 16.0% of those of NETWORK, FILESYSTEM and IDENTITY_POLICY (20 defects), a difference of 34.0 points. We drew two groups of 15 and 20 defects at random from the 39 defects, 20,000 times. The results show that only 1.5% of the random groupings give a difference of 34.0 points or more (p=0.015p = 0.015), so the difficulty of the examples alone is unlikely to explain it. A fifth model, GLM-5.3-Flash, shows the same difference on 78 defects (45.0% against 19.1% missed, p=0.0043p = 0.0043).
  • Conclusion 2: the harness matters when the whole repository must be reviewed, and not when the task names the file. To verify whether this conclusion holds for another model, we further conducted a controlled experiment with GLM-5.3-Flash. Specifically, we planted each of 78 defects alone in three tasks: one names the file, one names the module, and one asks for a review of the whole repository. GLM-5.3-Flash reviewed every task twice in Claude Code and twice in Pi. The results show that the two harnesses do not differ reliably when the task names the file (113 against 105 of 156 found, p=0.17p = 0.17), and that they differ when the whole repository must be reviewed (31 against 18 of 156, p=0.015p = 0.015).

Changes in the revised paper. All changes are in blue.

  • A second judge and a human audit of 70 reports: Section 4; Appendix M, Tables 8 to 10.
  • The definition of a confirmed claim, and claim validity without the findings of planted defects: Section 4; Tables 2 and 7.
  • The controlled experiment with a fifth model: Section 5; Appendix N, Table 11.
  • The analyses of the vulnerability types and of the harnesses per source record: Section 5; Appendix O, Tables 12 to 15.
  • The check that the agent saw the lines of a planted defect that it missed: Appendices I and P, Table 16.
  • Presentation: no bold inside sentences; terms defined in Section 4.
  • All participant reports will be released with the benchmark (reproducibility statement).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.