acceptodds
Under review as a conference paper at ICLR 2027

VulVerifyBench: Can and How LLM Agents Verify Vulnerability Reports?

Abstract

As AI agents scale vulnerability discovery, plausible reports can be produced faster than human experts can review them. Verifying these reports requires not only reproducing the reported behavior but deciding whether the evidence justifies accepting the claim. Existing vulnerability benchmarks begin with predefined labels, objectives, or confirmed vulnerabilities, so they evaluate detection or exploitation rather than whether a submitted report should be upheld. We introduce VulVerifyBench, a novel benchmark that evaluates whether and how agents verify vulnerability reports. It contains 747 real-world reports across smart contracts, traditional software, and web applications, including 220 invalid reports. These include 45 hard negatives, in which real behavior does not justify acceptance, and 88 counterfactual negatives that pair unchanged reports with patched code and have labels fixed by construction. Given a scrubbed report and, where available, a repository, an agent independently investigates the claim and returns an evidence-supported verdict. VulVerifyBench measures verdict accuracy, investigation depth, and grounding in the recorded trajectory. Across two frontier and two open-weight backend models, the best accuracies in the three domains are 84.5%, 87.5%, and 78.4%, respectively, but no backend exceeds 77.8% recall on hard negatives. Trajectory audits find verdicts unsupported by gathered facts in 6.3% of runs and action accounts inconsistent with the trajectory in 2.9%. These results show that current agents can verify many reports, but their central weakness lies not in producing evidence but in deciding what that evidence warrants.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, benchmark, vulnerability verification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zuDFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VulVerifyBench, a benchmark for evaluating whether LLM agents can verify vulnerability reports, i.e., decide whether a submitted report should be upheld, rather than detect, reproduce, or exploit a vulnerability. The dataset contains 747 real-world reports (239 from Solodit and 508 from HackerOne) spanning smart contracts, traditional software, and web applications, of which 527 are valid and 220 are invalid. The invalid reports consist of clear negatives (87), hard negatives (45), and counterfactual negatives (88) whose labels are fixed by construction. For evaluation, an agent works in a sandbox for at most 50 turns and outputs valid / invalid / uncertain along with a self-reported evidence layer. The paper evaluates Opus 4.8, GPT-5.6, DeepSeek-V4-Pro, and GLM-5.2; the best accuracies in the three domains are 84.5%, 87.5%, and 78.4%, and no model exceeds 77.8% recall on hard negatives. The authors' central conclusion is that the main weakness of agents lies not in producing evidence but in deciding whether the evidence justifies upholding a report.

Critical strengths
  • A novel task that complements existing benchmarks. Existing benchmarks mostly start from confirmed vulnerabilities or targets and lack "reports that should be rejected." This paper takes the report as the unit of evaluation, retains genuinely rejected submissions, and constructs counterfactual negatives whose labels are fixed by construction, thereby separating "can reproduce" from "should be upheld." As AI-generated vulnerability reports become more common, this problem is of practical relevance.

  • A careful evaluation design with honest disclosure of limitations. Beyond accuracy, the paper recovers investigation depth and audits whether verdicts are supported by the evidence, and it surfaces some thought-provoking observations. For example, runs that execute a PoC tend to be more accurate on valid reports but less accurate on invalid ones. The authors manually reviewed the auditor, performed contamination checks, and explicitly state that labels reflect adjudication outcomes rather than absolute ground truth and that GPT-5.6 uses a different scaffold, which increases the credibility of the results.

Critical weaknesses
  • The central conclusion rests on a small and noisy hard-negative subset. There are only 45 hard negatives (31 software, 14 web), and the paper's main conclusion that "the frontier advantage does not transfer to hard negatives" comes from this subset. Opus 4.8's advantage over the other backends is supported by exact McNemar tests (p ≤ 0.035) without multiple-comparison correction, and the samples may be concentrated in a few projects and thus not independent. In addition, GPT-5.6 uses a different scaffold, so part of the difference may stem from the scaffold. More importantly, according to Appendix A, at least 9 of these 45 samples were closed for reasons such as "already known, out of scope, or bounty policy," and 13 depend on project-specific threat models that the agent cannot see. The metric therefore mixes reasoning ability with guessing project policy. As a result, the conclusions may lack generalizability.

  • Key baselines and controls are missing. Only 19 of the 227 web reports are invalid, so a trivial "always answer valid" baseline would reach about 91.6% accuracy (computed from the figures above), higher than the best model accuracy of 78.4%. Although the paper reports balanced accuracy, it provides no trivial or weak baselines (always-valid, single-pass reasoning without tools, or human triage), so readers cannot tell how much benefit the agents' investigation actually provides.

Questions for authors

See the Weaknesses section above.

3 comments

We thank you for your detailed comments. Our responses to the weaknesses (W1, W2) are presented below. We agree that the hard-negative subset deserves additional sensitivity analyses because it is relatively small and contains heterogeneous closure reasons. We therefore examine the effect of removing non-technical closures, correct the multiple-comparison analysis, test sensitivity to the dominant project, and add a scaffold-swap control. We also add trivial and report-only baselines to clarify what the reported accuracy reflects.

zuDF-W1: The central conclusion rests on a small and noisy hard-negative subset.

Thank you for examining the hard-negative subset closely. Removing the 9 non-technical closures strengthens rather than weakens the hard-negative pattern: Opus 4.8’s lead widens, and all three Holm-corrected comparisons are significant on the remaining 36 cases. Below, we further examine closure reasons, project concentration, and scaffold sensitivity.

1. Sensitivity to hard-negative closure reasons

Our closure-reason audit labels all 45 hard negatives: 20 were closed because the reported behavior was intended, 13 were judged not to constitute vulnerabilities under the program’s threat model, 3 were fixed without a stated position, and 9 were closed for non-technical reasons (4 already known, 3 out of scope, and 2 bounty-policy decisions). Rebuttal Table 1 reports hard-negative recall after separating these groups.

Rebuttal Table 1: Hard-negative recall (%) by closure reason.

Subset (n)Opus 4.8GPT-5.6DeepSeek-V4-ProGLM-5.2
All (45)77.857.853.357.8
Excl. 9 non-technical (36)86.161.158.358.3
Intended behavior only (20)95.055.065.060.0
Threat-model-dependent (13)76.969.246.253.8

Removing the 9 explicitly non-technical closures strengthens the main qualitative pattern: Opus 4.8 increases from 77.8% to 86.1% recall, while GPT-5.6 remains close to the two open-weight backends. The clearest technical subset—the 20 intended-behavior cases—shows an even stronger separation for Opus 4.8 (95.0%), whereas GPT-5.6 remains at 55.0%.

The 9 non-technical closures are difficult for all backends (33–56% recall). We retain them in the benchmark because our labels reflect actual program adjudications rather than our own technical reinterpretation, but we will distinguish them explicitly in the revised analysis.

The 13 threat-model-dependent cases show a smaller gap between Opus 4.8 and GPT-5.6 (76.9% vs. 69.2%), suggesting that this subset is more sensitive to program-specific security assumptions. We therefore report these cases separately from the intended-behavior subset.

2. Multiple-comparison correction

We thank you for pointing out that the original pairwise McNemar tests were reported without correction for multiple comparisons. We therefore apply Holm correction across the three comparisons between Opus 4.8 and the other backends, using exact two-sided McNemar tests.

Rebuttal Table 2: Exact two-sided McNemar tests of Opus 4.8 against each backend on hard negatives, with Holm correction across the three comparisons.

SubsetComparisonOnly Opus 4.8 correctOnly other correctRaw pHolm p
All (45)Opus 4.8 vs. DeepSeek-V4-Pro1210.00340.010
Opus 4.8 vs. GPT-5.61230.03520.070
Opus 4.8 vs. GLM-5.21230.03520.070
Excl. 9 non-technical (36)Opus 4.8 vs. DeepSeek-V4-Pro1110.00630.019
Opus 4.8 vs. GLM-5.21110.00630.019
Opus 4.8 vs. GPT-5.61120.02250.023

On all 45 hard negatives, the Opus 4.8–DeepSeek-V4-Pro comparison remains significant after Holm correction (adjusted p = 0.010), whereas the comparisons with GPT-5.6 and GLM-5.2 become marginal (adjusted p = 0.070). After excluding the 9 non-technical closures, all three comparisons are significant after correction (adjusted p ≤ 0.023). After Holm correction, Opus 4.8 remains significantly stronger on the 36-case subset excluding non-technical closures.

3. Project concentration and dependence

You also raise a valid concern that reports from the same program may not be statistically independent. The 45 hard negatives span 13 programs, with curl contributing the largest cluster (19 reports, 42%). As a sensitivity check, we recompute recall separately within curl and across the remaining 26 reports in Rebuttal Table 3.

Rebuttal Table 3: Hard-negative recall (%) within and outside the largest program.

Subset (n)Opus 4.8GPT-5.6DeepSeek-V4-ProGLM-5.2
curl only (19)84.257.952.657.9
Non-curl (26)73.157.753.857.7

The same qualitative pattern appears both within the dominant curl cluster and outside it: Opus 4.8 remains substantially ahead, while the other three backends remain closely grouped. This check does not eliminate within-program dependence, but it shows that the observed pattern is not driven solely by the largest project. We will clarify this distinction and retain project clustering as a limitation of the hard-negative analysis.

4. Scaffold-swap control

You also note that GPT-5.6 uses a different agent scaffold, so part of the observed difference could arise from the harness rather than the backend itself. To examine this, we reran DeepSeek-V4-Pro—which is evaluated under Claude Code in the main paper—on all 747 reports using OpenAI Codex, the scaffold used by GPT-5.6. We hold the container image, prepared workspace, system and task prompts, reasoning effort, and 50-turn budget fixed.

Rebuttal Table 4: DeepSeek-V4-Pro under Claude Code and OpenAI Codex (%).

DomainScaffoldCoverageOverall (95% CI)BalancedValidInvalid
Smart contractClaude Code201/23965.7 [59.8, 71.5]65.865.665.9
OpenAI Codex207/23967.8 [61.9, 73.6]68.864.972.7
Traditional softwareClaude Code259/28176.2 [71.5, 80.8]75.081.069.0
OpenAI Codex249/28178.6 [74.0, 83.3]77.782.772.6
Web applicationClaude Code205/22762.1 [55.5, 68.7]65.061.568.4
OpenAI Codex186/22768.3 [61.7, 74.4]63.669.257.9

Rebuttal Table 5: Investigation behavior of DeepSeek-V4-Pro under each scaffold. Correct reports are out of 747; the other columns cover the 738 reports with token usage logged for both runs.

ScaffoldCorrect reportsPoC executed (%)Mean output tokensMean tool calls
Claude Code5126611.7k23
OpenAI Codex5389315.8k43

The scaffold clearly changes investigation behavior. Under Codex, DeepSeek-V4-Pro executes PoCs much more often, uses more tool calls, and produces longer outputs. The two scaffolds also solve different individual reports: 432 reports are correct under both, 80 only under Claude Code, and 106 only under Codex.

However, these behavioral differences do not produce a statistically significant change in aggregate verdict accuracy. Accuracy increases from 68.5% to 72.0% overall, but the paired difference is not significant (exact McNemar, p = 0.067), and no domain-level difference is significant. DeepSeek-V4-Pro also remains below both frontier backends in every domain. We therefore do not claim that scaffold choice is irrelevant. Rather, this control shows that the scaffold materially affects how the agent investigates, while the main aggregate performance pattern remains robust in this controlled scaffold swap.

Revision. We will add the closure-reason sensitivity analysis, Holm-corrected comparisons, project-concentration analysis, and scaffold-swap control to the revised paper and will state the remaining limitations of the hard-negative subset more explicitly.

zuDF-W2: Key baselines and controls are missing.

We agree that additional baselines are important, particularly because raw accuracy can be misleading in the highly imbalanced web subset. We therefore add both a trivial majority-class baseline and a report-only baseline that removes repository and tool access.

1. Trivial majority-class baseline

Because valid reports are the majority class in every domain, the majority-class baseline is equivalent to always predicting valid.

Rebuttal Table 6: Accuracy and balanced accuracy (%) of trivial baselines per domain.

DomainValidInvalidAlways-valid acc.Always-valid bal.Always-invalid acc.Always-invalid bal.
Smart contract1518863.250.036.850.0
Traditional software16811359.850.040.250.0
Web2081991.650.08.450.0

As you note, the always-valid baseline reaches 91.6% raw accuracy on the web subset because valid reports dominate this subset. However, this apparent performance is entirely driven by class imbalance: the same baseline has only 50.0% balanced accuracy and zero recall on invalid reports. By contrast, the evaluated backends achieve 59.0–87.6% balanced accuracy across domains. An always-valid classifier therefore fails precisely on the invalid reports that a verification or triage system must identify.

Revision. We will therefore foreground balanced accuracy and invalid-report recall more clearly when discussing cross-domain results.

2. Report-only baseline

We also ran the report-only baseline on all 747 reports: each backend receives the same scrubbed report in a single call, without repository access or tools. This directly measures what agentic code investigation contributes beyond reasoning from the report itself.

Rebuttal Table 7: Accuracy (%) of the tool-enabled verification agent and the report-only baseline on all 747 reports, by label type. † marks the significantly higher setting (exact McNemar, p < 0.05).

Label type (n)SettingOpus 4.8GPT-5.6DeepSeek-V4-ProGLM-5.2
Valid (527)Tool-enabled79.782.768.962.6
Report-only79.579.985.2†88.6†
Clear negative (87)Tool-enabled92.089.777.083.9†
Report-only96.694.378.263.2
Hard negative (45)Tool-enabled77.857.853.357.8
Report-only75.660.057.840.0
Counterfactual negative (88)Tool-enabled95.5†88.6†65.9†71.6†
Report-only23.926.113.611.4
All (747)Tool-enabled82.9†82.7†68.565.9
Report-only74.774.074.3†73.6†

Across all 747 reports, the two frontier tool-enabled agents significantly outperform their report-only baselines. The largest gains occur on the 88 counterfactual negatives: every backend improves by 52–72 percentage points. These cases keep the original report unchanged while replacing the vulnerable code with a patched version, so the report alone cannot reveal the correct verdict. This provides a controlled demonstration that code-level investigation contributes information beyond the report.

Outside the counterfactual negatives, the two frontier agents perform similarly to report-only, whereas the open-weight agents perform worse, largely because many tool-enabled runs terminate without a valid verdict. Thus, the baseline reveals a more nuanced result: agentic code investigation is essential when the answer depends on the underlying code state, but current agents do not yet convert additional investigation into uniformly better adjudication across naturally occurring reports.

Revision. We will add the trivial and report-only baselines to the revised paper and foreground balanced accuracy and invalid-report recall for class-imbalanced domains.

Reviewer_kFACOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this paper introduces VulVerify, a new benchmark for verifying vulnerability reports, where the agent is judged on classification accuracy and providing effective evidence in its decision

Critical strengths
  • the paper has clear motivation and goals and executes well
  • the topic is relevant and suitable for the ICLR conference
  • the paper contributes a new benchmark that hasn't been discussed or produced yet by the community, and is a critical capability that needs to be evaluated by current LLMs
Critical weaknesses

N/A

Questions for authors
  • The protocol collects an "uncertain" option and a per-verdict confidence, but both are folded into accuracy (uncertain = incorrect) and never reported. Please report, per model and domain: (i) the abstention rate and the committed-error rate (wrong valid/invalid verdicts) separately; (ii) an abstention-aware score, e.g. +1 correct / 0 uncertain / −1 wrong, which rewards committing when right and abstaining when unsure; and (iii) calibration of the reported confidence (ECE, or a risk–coverage curve of accuracy vs. fraction answered). In triage, a confidently wrong "valid" (wasted engineering effort) or "invalid" (missed vulnerability) is costlier than an "uncertain" that gets escalated to a human, so accuracy alone may misrank models. Since the prompt explicitly discourages "uncertain", does that bias the comparison, and how do results change without that instruction?
  • §3.1 notes that most of Opus 4.8's 12 missing verdicts are safety refusals, and the auditor (Sonnet 5) also refused on some runs. Please report refusal counts and rates separately from other non-verdicts (turn exhaustion, unparseable output), per model, domain, and vulnerability class. Are refusals concentrated in particular domains (e.g. web exploitation) or classes? Since refusals count as incorrect, how much do they move accuracy and the model ranking? Do the 45 unlabeled auditor runs (or the Sonnet 4.6 fallback) bias the RQ2/RQ3 analyses?
3 comments

We thank you for your careful reading and constructive questions (Q1, Q2). Below, we report the requested abstention-aware and calibration analyses, and separately examine the effects of safety refusals and auditor failures.

kFAC-Q1: Abstention, committed errors, abstention-aware scoring, and calibration are not reported.

Thank you for this suggestion. We agree that accuracy alone does not distinguish between abstaining, committing to an incorrect verdict, and failing to return a usable verdict. We therefore decompose these outcomes, evaluate an abstention-aware score, and analyze confidence calibration and risk–coverage.

1. Abstention versus committed errors

We classify each run into five outcomes: correct verdict, abstention (uncertain), committed error (an incorrect valid/invalid verdict), safety refusal, or other missing verdict (e.g., turn-budget exhaustion or non-conforming output). Rebuttal Table 1 aggregates these outcomes across domains, while Rebuttal Table 2 reports them by model and domain.

Rebuttal Table 1: Outcome decomposition per model, aggregated over domains (747 runs each). Accuracy, abstention, and committed error are percentages of runs; refusals and other missing verdicts are counts.

ModelAccuracyAbstentionCommitted errorRefusalsOther missing
Opus 4.882.90.015.5121
GPT-5.682.71.715.302
DeepSeek-V4-Pro68.53.217.3082
GLM-5.265.91.118.30110

Rebuttal Table 2: Outcome decomposition per model and domain. Units as in Rebuttal Table 1.

DomainModelAccuracyAbstentionCommitted errorRefusalsOther missing
Smart contract (239)Opus 4.882.80.016.701
GPT-5.684.50.414.202
DeepSeek-V4-Pro65.70.018.4038
GLM-5.265.71.319.2033
Traditional software (281)Opus 4.887.50.08.9110
GPT-5.684.71.813.500
DeepSeek-V4-Pro76.21.114.9022
GLM-5.270.80.415.7037
Web (227)Opus 4.877.10.022.510
GPT-5.678.43.118.500
DeepSeek-V4-Pro62.19.318.9022
GLM-5.259.91.820.7040

Abstention is uncommon for every backend: it never exceeds 3.2% when aggregated across domains, with the largest domain-specific rate being 9.3% for DeepSeek-V4-Pro on web reports. By comparison, committed errors account for 15.3–18.3% of runs overall. Opus 4.8 never returns uncertain. Thus, most errors in the current evaluation arise from committed incorrect verdicts or, for the open-weight backends, missing verdicts rather than explicit abstention.

2. Abstention-aware scoring (+1 / 0 / −1)

Following your suggestion, we assign +1 to a correct verdict, 0 to uncertain, and −1 to an incorrect committed verdict. We report two variants for refusals and other missing verdicts: U0 assigns them 0, corresponding to escalation to a human, whereas U− assigns them −1, corresponding to producing no usable answer.

Rebuttal Table 3: Abstention-aware score per model, aggregated over domains.

ModelU0U−
Opus 4.80.6730.657
GPT-5.60.6750.672
DeepSeek-V4-Pro0.5130.403
GLM-5.20.4750.328

The abstention-aware scores preserve the main separation between the two frontier and two open-weight backends. They do slightly change the numerical ordering of the top two models: GPT-5.6 scores marginally above Opus 4.8 under both variants, whereas Opus 4.8 has slightly higher raw accuracy (82.9% vs. 82.7%). The differences are very small, however, and the two frontier backends remain effectively tied, consistent with their statistically indistinguishable verdict accuracy (McNemar’s test).

3. Confidence calibration and risk–coverage

For committed verdicts, Rebuttal Table 4 reports expected calibration error (ECE) and mean confidence separately for correct and incorrect predictions. Rebuttal Table 5 reports selective accuracy when retaining only the most confident predictions at fixed coverage levels.

Rebuttal Table 4: Calibration of committed verdicts per model.

ModelECEMean confidence, correctMean confidence, incorrect
Opus 4.80.0350.8460.764
GPT-5.60.1140.8510.856
DeepSeek-V4-Pro0.1010.8840.860
GLM-5.20.0760.8570.811

Rebuttal Table 5: Selective accuracy (%) at fixed coverage of all reports, answering the most confident verdicts first. “–” indicates coverage above the backend's committed fraction.

Model50%70%80%90%Committed (coverage)
Opus 4.890.690.088.285.684.3 (98.3)
GPT-5.684.083.483.383.884.4 (98.0)
DeepSeek-V4-Pro82.881.380.3–79.9 (85.8)
GLM-5.282.680.479.0–78.2 (84.2)

These analyses reveal a difference that raw accuracy obscures. Opus 4.8 is substantially better calibrated (ECE = 0.035), and its confidence meaningfully separates correct from incorrect verdicts (0.846 vs. 0.764). GPT-5.6, despite nearly identical overall accuracy, has the highest ECE (0.114) and is slightly more confident when incorrect than when correct (0.856 vs. 0.851). The risk–coverage results show the practical consequence: on the 50% of reports for which each backend is most confident, Opus 4.8 reaches 90.6% accuracy, whereas GPT-5.6 remains at 84.0%. Thus, confidence provides substantially more useful triage information for Opus 4.8 than for GPT-5.6.

Regarding the prompt discouraging uncertain, we clarify that the instruction is identical across all backends, so all models are evaluated under the same prompting condition. Under this prompt, abstention is low (at most 3.2% overall), and explicitly accounting for abstention preserves the broad performance pattern: the two frontier backends remain effectively tied and well ahead of the two open-weight backends. The large separation between the frontier and open-weight backends also remains under abstention-aware scoring. We acknowledge, however, that the absolute abstention rates are conditional on this prompt design and should not be interpreted as prompt-independent model behavior.

Revision. We will add the outcome decomposition, abstention-aware scoring, calibration, and risk–coverage analyses to the revised paper, and clarify that the observed abstention rates are conditional on the evaluation prompt.

kFAC-Q2: Refusals are not separated from other non-verdicts, and auditor refusals may bias RQ2/RQ3.

Thank you for raising this issue. We separately analyze (1) safety refusals in the evaluated agents, (2) the 45 runs left unlabeled by the auditor, and (3) the use of Sonnet 4.6 as a fallback when Sonnet 5 refuses. These analyses show that safety refusals modestly depress Opus 4.8's reported accuracy, while the auditor failures have limited influence on the RQ2/RQ3 conclusions.

(1) Safety refusals

Our per-run records contain a separate safety-refusal flag, allowing refusals to be distinguished from turn-budget exhaustion and non-conforming output. We observe 12 safety refusals among 2,988 runs, all from Opus 4.8. Eleven occur in traditional software and one in web; none occur in smart contracts. By vulnerability class, 11 of the 12 involve Memory Safety and one involves Injection & Untrusted Input.

Rebuttal Table 6: Opus 4.8 safety refusals by vulnerability class and domain.

Vulnerability classSmart contractTraditional softwareWebTotal
Memory Safety010111
Injection & Untrusted Input0101
Total011112

For Opus 4.8, this corresponds to an overall refusal rate of 1.6% (12/747), concentrated in traditional software (3.9%, 11/281) and especially in memory-safety tasks. This pattern is consistent with safety restrictions around constructing memory-corruption exploits. These refusals actually lower the model's accuracy. If the 12 refusal runs are treated as cases escalated to human review and excluded, Opus 4.8's accuracy increases from 82.9% to 84.1%, while the accuracy of the other backends remains unchanged.

Rebuttal Table 7: Accuracy (%) with and without refusal runs.

ModelFull accuracyAccuracy excluding refusals
Opus 4.882.984.1
GPT-5.682.782.7
DeepSeek-V4-Pro68.568.5
GLM-5.265.965.9

Thus, excluding refusal cases does not change the model ranking and slightly widens Opus 4.8’s lead.

(2) The 45 unlabeled auditor runs

For RQ2, the auditor is needed only when the agent-reported and rule-derived evidence layers disagree. Among the 45 runs for which the auditor failed to return a label, 37 enter the evidence-layer analysis, and the first two signals already agree on 35 of them. Thus, only 2 runs require an unavailable auditor tie-break, corresponding to 0.07% of all 2,988 runs. The missing auditor labels therefore have negligible influence on the RQ2 layer analysis.

For RQ3, the 45 auditor failures are excluded from the 1,553 runs receiving a final grounding label. The reported grounded rate among these labeled runs is 93.5%. Under a conservative worst-case sensitivity analysis in which all 45 unlabeled runs are treated as ungrounded, the grounded fraction over the full 1,598-run audited sample becomes 90.9% (1,452/1,598). Thus, even this extreme assignment preserves the conclusion that the large majority of audited verdicts are grounded.

(3) Sonnet 4.6 fallback

Sonnet 4.6 is used only when Sonnet 5 refuses. For the RQ2 evidence-layer judgment, the two auditor models agree exactly in 74.4% of cases and are within one ordinal evidence level in 94.2%, indicating that fallback-induced differences are typically small.

For RQ3, auditor-flagged faithfulness judgments are manually reviewed, and the paper additionally reports a blind human review of 100 runs labeled grounded with no reversals. Moreover, in disagreements between the two auditor models, Sonnet 4.6 more often assigns the stricter label—55 disagreements in which it assigns ungrounded or partially grounded, compared with 6 in the opposite direction. This pattern provides no indication that use of the fallback artificially inflates the grounded rate.

Overall, safety refusals, auditor failures, and fallback use affect only a small fraction of the evaluation and do not change the substantive RQ2/RQ3 conclusions.

Revision. We will separately report agent safety refusals, distinguish them from other missing verdicts, add the refusal sensitivity analysis, and clarify how unlabeled auditor runs and the Sonnet 4.6 fallback enter the RQ2/RQ3 analyses.

Reviewer_uZbjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for deciding whether submitted vulnerability reports should be upheld. It contains 747 reports from smart contract audits and bug bounty programs. The paper provides four configuration evaluated on all reports and measures whether the agent's conclusions are supported by its recorded actions. The results show that agents often struggle with reports whose behavior is reproducible but does not justify accepting the vulnerability claim.

Critical strengths
  • Important benchmark as more and more LLMs are used for vulnerabitliy analysis. The verifiaciton of related report is important for the next step of LLM-enabled security because the verification is also time-consuming.

  • The evaluation on benchmark provides interesting findings. The paper shows that successful reproduction can still lead to wrong verdict which is useful for future relevant harness design and post-tranining algorihtm.

Critical weaknesses
  • The benchmark seems easily saturated. As reported in the evaluation, mainstream LLMs achieve at least around 60% accuracy and considering the rapid progress of frontier LLMs' ability on coding, the benchmark can be saturated in a short time. Even worst, there is no signs that such report verification tasks is difficult enough for the LLM to evolve for a long period.

  • The evaluation lacks a simple report-only control and more varied harness envionrment. Without a baseline that reads the same scrubbed report but cannot inspect code or run tools, it is hard to measure what independent agent investigation adds over reasoning from the report itself. Also, it is also improtant to further measure whether better system or user prompt can help with verification.

Questions for authors
  1. Can the authors report the results separately for cases with and without a repository, and compare the agents against baselines on the same cases?

  2. How much do results on the difficult negative cases change when restricted to reports rejected for a clear technical security reason rather than program policy or scope?

4 comments

We appreciate your thoughtful comments. Our responses to the weaknesses (W1, W2) and questions (Q1, Q2) are presented below.

uZbj-W1: Mainstream LLMs achieve at least around 60% accuracy and considering the rapid progress of frontier LLMs' ability on coding, the benchmark can be saturated in a short time.

We agree that benchmark saturation deserves careful consideration. To examine whether improvements in coding ability alone are likely to saturate VulVerifyBench, we followed your suggestion and introduced a report-only baseline. Each backend reads the same scrubbed report in a single call, without repository access or tool use. We compare this baseline with our standard tool-enabled verification agent, which can inspect code and execute tools under the VulVerifyBench protocol. The same report-only setup is also used in our response to W2 & Q1.

Rebuttal Table 1: Hard-negative recall (%) of the tool-enabled agent and the report-only baseline (45 cases).

Hard-negative recall (45 cases)Opus 4.8GPT-5.6DeepSeek-V4-ProGLM-5.2
Tool-enabled agent77.857.853.357.8
Report-only75.660.057.840.0

a) The bottleneck is not simply coding ability, but security judgment. On hard negatives, the tool-enabled agent does not significantly outperform the report-only baseline for any backend (exact McNemar, p ≥ 0.077), despite being able to inspect code and execute experiments. The same pattern holds on the HackerOne reports (Rebuttal Table 3). Hard-negative recall also remains limited, at 77.8% for the strongest backend and 53.3–57.8% for the others. These results suggest that improving coding ability alone is unlikely to resolve the main difficulty exposed by these cases: deciding whether observed behavior actually warrants accepting a vulnerability claim.

b) Reliable verification requires more than a correct verdict (i.e., high accuracy mentioned here). Verdict accuracy captures only whether the final adjudication is correct. RQ3 therefore captures a dimension that verdict accuracy alone does not: whether the verdict is actually supported by the trajectory evidence. Although most audited verdicts are grounded, the remaining failures show why accuracy alone is insufficient to characterize reliable verification.

c) The contribution of VulVerifyBench extends beyond the difficulty of the current fixed snapshot. We agree that any fixed set of benchmark instances may become easier as models improve. VulVerifyBench, however, is intended to evaluate a capability that existing vulnerability benchmarks do not capture: whether an agent can independently adjudicate a submitted vulnerability report and justify that verdict with evidence. Accordingly, the benchmark contribution includes not only the current 747 reports, but also the report-level verification task, challenging hard and counterfactual negatives, and trajectory-based evaluation of investigation depth and verdict grounding. The report-only results further suggest that progress in coding ability alone does not imply that this verification capability has been solved.

uZbj-W2 & uZbj-Q1: The evaluation lacks a simple report-only control and more varied harness environment. Without a baseline that reads the same scrubbed report but cannot inspect code or run tools, it is hard to measure what independent agent investigation adds over reasoning from the report itself.

We thank you for this suggestion. We added two controls to directly address these concerns. First, we evaluate a report-only baseline that receives exactly the same scrubbed report but has no repository access or tools, allowing us to isolate the value of agentic code investigation. Second, we rerun the same backend under two different agent scaffolds while holding the task, workspace, prompts, reasoning effort, and turn budget fixed. The new results show that agentic code investigation provides a large advantage when the correct verdict depends on the underlying code state, but this advantage does not extend uniformly to all report types. The scaffold substantially changes how the agent investigates, while aggregate verdict accuracy remains statistically similar in our controlled scaffold swap.

1. What does independent agent investigation (i.e., agentic code investigation in our setting) add beyond reasoning from the report?

We ran the report-only baseline on all 747 reports. Each backend receives the same scrubbed report as in our main evaluation, but in a single model call with no repository access or tools. We compare this baseline against our standard tool-enabled verification agent, which can inspect code and execute experiments under the VulVerifyBench protocol.

Rebuttal Table 2: Accuracy (%) of the tool-enabled agent and the report-only baseline on all 747 reports, by label type. † marks the higher setting when the difference is significant (exact McNemar, p < 0.05).

Label type (n)SettingOpus 4.8GPT-5.6DeepSeek-V4-ProGLM-5.2
Valid (527)Tool-enabled79.782.768.962.6
Report-only79.579.985.2†88.6†
Clear negative (87)Tool-enabled92.089.777.083.9†
Report-only96.694.378.263.2
Hard negative (45)Tool-enabled77.857.853.357.8
Report-only75.660.057.840.0
Counterfactual negative (88)Tool-enabled95.5†88.6†65.9†71.6†
Report-only23.926.113.611.4
All (747)Tool-enabled82.9†82.7†68.565.9
Report-only74.774.074.3†73.6†

a) Agentic code investigation matters most when the report alone cannot reveal the answer. The clearest evidence comes from the 88 counterfactual negatives. Each counterfactual keeps the original vulnerability report unchanged but replaces the vulnerable code with a patched version. The report alone therefore continues to describe the original vulnerability, whereas inspecting the actual code state can reveal that the reported issue has already been fixed. On these cases, the tool-enabled agent improves accuracy by 52.3–71.6 percentage points over report-only across all four backends. This provides a controlled demonstration that agentic code investigation can contribute decisive information beyond the report itself.

b) More code investigation does not automatically produce a better verdict. After excluding counterfactual negatives, the two frontier backends perform similarly to their report-only baselines on the remaining HackerOne reports, regardless of repository availability. In other words, access to code and tools does not consistently improve final adjudication when the report already contains substantial information. This is consistent with our main finding that deeper investigation or successful reproduction does not necessarily determine whether the evidence ultimately warrants accepting the vulnerability claim. For the open-weight backends, the tool-enabled setting can perform worse than report-only. A substantial part of this gap comes from agent-execution failures: DeepSeek-V4-Pro and GLM-5.2 frequently exhaust the turn budget or fail to return the required structured verdict. Thus, the report-only comparison also exposes a distinct agentic difficulty that is invisible in a single-pass reasoning baseline.

Rebuttal Table 3: Accuracy (%) of the tool-enabled agent and the report-only baseline by repository availability, excluding counterfactual negatives. † as in Rebuttal Table 2.

Subset (n, valid / invalid)SettingOpus 4.8GPT-5.6DeepSeek-V4-ProGLM-5.2
HackerOne, repository (131, 106 / 25)Tool-enabled80.989.364.948.9
Report-only78.685.580.2†85.5†
HackerOne, no repository (377, 270 / 107)Tool-enabled83.679.371.671.9
Report-only82.280.482.2†79.3†
Smart contract, repository (151, 151 / 0)Tool-enabled75.582.165.662.3
Report-only82.176.284.8†85.4†

Revision. We will add the report-only control to the Appendix, including its prompt and two corresponding analyses: accuracy by label type (Rebuttal Table 2), and accuracy by repository availability after excluding counterfactual negatives (Rebuttal Table 3), covering HackerOne reports with and without repository access as well as the smart-contract cases.

2. How sensitive are the results to the agent harness?

To examine harness sensitivity, we reran DeepSeek-V4-Pro—which is evaluated under Claude Code in the main paper—on all 747 reports using OpenAI Codex, the scaffold used by GPT-5.6. We keep the container image, prepared workspace, system and task prompts, reasoning effort, and 50-turn budget unchanged, so the primary change is the agent scaffold.

Rebuttal Table 4: DeepSeek-V4-Pro under Claude Code and OpenAI Codex (%).

DomainScaffoldCoverageOverall (95% CI)BalancedValidInvalid
Smart contractClaude Code201/23965.7 [59.8, 71.5]65.865.665.9
OpenAI Codex207/23967.8 [61.9, 73.6]68.864.972.7
Traditional softwareClaude Code259/28176.2 [71.5, 80.8]75.081.069.0
OpenAI Codex249/28178.6 [74.0, 83.3]77.782.772.6
Web applicationClaude Code205/22762.1 [55.5, 68.7]65.061.568.4
OpenAI Codex186/22768.3 [61.7, 74.4]63.669.257.9

The scaffold-swap control yields two complementary findings.

a) The scaffold swap does not significantly change aggregate verdict accuracy, and the model ordering is preserved. Moving DeepSeek-V4-Pro from Claude Code to OpenAI Codex increases accuracy by 2.1, 2.5, and 6.2 percentage points across the three domains, and from 68.5% to 72.0% overall. None of these differences is statistically significant under exact McNemar tests (p = 0.61, 0.44, and 0.10 by domain; p = 0.067 overall). Moreover, under either scaffold, DeepSeek-V4-Pro remains below both frontier backends in every domain. Thus, while the scaffold materially changes the investigation process, the aggregate accuracy differences are not statistically significant and the relative model ordering remains unchanged in this controlled comparison.

b) The scaffold substantially changes investigation behavior. Under OpenAI Codex, DeepSeek-V4-Pro executes PoCs more often (93% vs. 66%), uses more tool calls on average (43 vs. 23), and produces longer outputs (15.8k vs. 11.7k tokens). The two scaffolds also solve different individual reports: they are both correct on 432 reports, while 80 are correct only under Claude Code and 106 only under Codex. Thus, the scaffold clearly affects how the backend investigates and which cases it solves.

Revision. We will add the scaffold-swap control and its behavioral comparison to the Scaffold Differences appendix.

uZbj-Q2: How much do results on the difficult negative cases change when restricted to reports rejected for a clear technical security reason rather than program policy or scope?

We thank you for this comment. To examine whether policy- or scope-related closures drive our results, we first exclude the 9 explicitly non-technical cases and then restrict the analysis to the 33 cases with an explicit security judgment. The results show that removing these cases improves recall for most backends while preserving the main qualitative finding on hard negatives.

Our closure-reason audit (Appendix, hard-negative label review) assigns each of the 45 hard negatives a closure reason: 20 intended behavior, 13 were judged not to constitute vulnerabilities under the program’s threat model, 3 fixed without a stated position, and 9 non-technical (4 already known, 3 out of scope, and 2 bounty-policy decisions). We recompute hard-negative recall on subsets defined by these reasons (Rebuttal Table 6).

Rebuttal Table 6: Hard-negative recall (%) by closure reason; tool-enabled agent, with report-only results in parentheses.

SubsetnOpus 4.8GPT-5.6DeepSeek-V4-ProGLM-5.2
All4577.8 (75.6)57.8 (60.0)53.3 (57.8)57.8 (40.0)
Excl. 9 non-technical3686.1 (83.3)61.1 (66.7)58.3 (63.9)58.3 (41.7)
Security-judgment subset (intended + threat-model)3387.9 (84.8)60.6 (66.7)57.6 (63.6)57.6 (42.4)
Non-technical only944.444.433.355.6

Excluding the 9 explicitly non-technical closures raises recall for all four backends. Restricting further to the 33 cases closed on an intended-behavior or threat-model judgment yields the same qualitative pattern: Opus 4.8 remains substantially stronger, while GPT-5.6 remains close to the two open-weight backends. The Opus 4.8–GPT-5.6 gap increases from 20.0 points on all 45 hard negatives to 27.3 points on this 33-case subset.

The statistical results are consistent with this pattern. With Holm correction across the three pairwise comparisons, Opus 4.8 significantly outperforms each of the other backends on both the 36-case subset and the 33-case subset (Holm-adjusted p ≤ 0.023), whereas GPT-5.6 remains statistically indistinguishable from either open-weight backend (p = 1.0). By contrast, performance on the 9 non-technical closures is low for all backends (33.3–55.6%), indicating that policy- and scope-related adjudications introduce a distinct source of difficulty.

Overall, excluding non-technical closures does not weaken our main hard-negative finding. Opus 4.8 remains the strongest backend, while the frontier advantage observed on clear and counterfactual negatives still does not transfer consistently to hard negatives. We will clarify this distinction in the revised paper and report the closure-reason sensitivity analysis in the Appendix.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Our Paper and Discussion

We are grateful to the reviewers for their insightful comments and to the ACs, SACs, and PCs for their thoughtful evaluation of our submission.

Summary of our contribution:

  • We introduced VulVerifyBench, to our knowledge the first benchmark that asks agents to adjudicate vulnerability reports in a working environment, shifting evaluation from producing evidence to deciding whether that evidence justifies upholding a claim.
  • We curated 747 real-world reports across three domains, retaining 220 rejected submissions, including 45 hard negatives and 88 counterfactual negatives.
  • We defined an ordered evidence-layer taxonomy to recover each run's investigation depth and audit whether its verdict is grounded in the recorded trajectory.
  • We evaluated four agent backends and showed that deeper investigation or successful reproduction does not necessarily yield a correct verdict; the new report-only and scaffold-swap controls confirm this pattern.

Overall, VulVerifyBench shifts vulnerability evaluation from whether agents can produce evidence to whether they can judge what that evidence warrants.

We are grateful that reviewers recognized the practical importance of our benchmark (uZbj, kFAC, zuDF), the novelty of the report-level task with rejected and counterfactual negatives (kFAC, zuDF), the careful evaluation of investigation depth and evidential grounding (kFAC, zuDF), and the insightful findings on reproduction and verdicts (uZbj, zuDF).

How we address reviewers' concerns:

We address all points raised with new experiments and analyses:

  • Report-only control (uZbj, zuDF): We ran a report-only baseline on all 747 reports, in which each backend receives the same scrubbed report in a single call without repository access or tools. The results showed that every backend's agent gains 52–72 points on the 88 counterfactual negatives, while outside them the two frontier agents perform similarly to report-only. This confirms that agentic code investigation is essential when the answer depends on the underlying code state, but current agents do not yet convert additional investigation into uniformly better adjudication.
  • Scaffold-swap control (uZbj, zuDF): We reran DeepSeek-V4-Pro on all 747 reports under OpenAI Codex, the scaffold used by GPT-5.6. The results showed that the scaffold changes investigation behavior (PoC execution rises from 66% to 93%), while aggregate accuracy does not change significantly (p = 0.067) and DeepSeek-V4-Pro still trails both frontier backends. This confirms that the scaffold materially affects how the agent investigates, while the main aggregate performance pattern remains robust.
  • Closure-reason sensitivity analysis (uZbj, zuDF): We recomputed hard-negative recall after excluding the 9 non-technical closures identified by our closure-reason audit. The results showed that recall rises for every backend, and Opus 4.8 significantly outperforms each other backend after Holm correction (adjusted p ≤ 0.023). This confirms that excluding non-technical closures does not weaken our main hard-negative finding.
  • Abstention-aware scoring and calibration (kFAC): We decomposed each run into correct verdicts, abstentions, committed errors, and missing verdicts, and computed abstention-aware scores, ECE, and risk–coverage. The results showed that abstention stays at or below 3.2%, that abstention-aware scoring keeps the two frontier backends well ahead of the two open-weight backends, and that Opus 4.8 is substantially better calibrated than GPT-5.6 (ECE 0.035 vs. 0.114). This confirms that explicitly accounting for abstention preserves the broad performance pattern, while confidence provides substantially more useful triage information for Opus 4.8 than for GPT-5.6.
  • Separating refusals from other non-verdicts (kFAC): We separated safety refusals using the per-run safety-refusal flag and examined the 45 unlabeled auditor runs and the Sonnet 4.6 fallback. The results showed that all 12 refusals come from Opus 4.8 and that excluding them raises its accuracy from 82.9% to 84.1%, while treating all unlabeled auditor runs as ungrounded still leaves 90.9% of audited verdicts grounded, and the Sonnet 4.6 fallback more often assigns the stricter label. This confirms that safety refusals, auditor failures, and fallback use affect only a small fraction of the evaluation and do not change the substantive RQ2/RQ3 conclusions.

Summary of manuscript revisions:

We have updated the manuscript in response to the concerns raised during the rebuttal, and the key revisions are summarized below.

  • We added a report-only control in Appendix D, with its prompt in Appendix H.6, comparing the full agent with a single-call baseline that has no repository access or tools, by label type and by repository availability.
  • We added a scaffold-swap control in Appendix C that runs DeepSeek-V4-Pro under OpenAI Codex, reporting accuracy, per-report overlap, and investigation behavior, and referenced it in Sections 3 and 3.1.
  • We added hard-negative recall by closure reason, Holm-corrected comparisons, and a project-concentration check in Appendix E.
  • We added trivial majority-class baselines in Appendix E.
  • We added the outcome decomposition, abstention-aware scores, calibration, and selective accuracy per model–scaffold pair in Appendix E.
  • We separated safety refusals from other missing verdicts and clarified how the unlabeled auditor runs and the Sonnet 4.6 fallback enter RQ2 and RQ3 in Appendix E.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.