- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes VulVerifyBench, a benchmark for evaluating whether LLM agents can verify vulnerability reports, i.e., decide whether a submitted report should be upheld, rather than detect, reproduce, or exploit a vulnerability. The dataset contains 747 real-world reports (239 from Solodit and 508 from HackerOne) spanning smart contracts, traditional software, and web applications, of which 527 are valid and 220 are invalid. The invalid reports consist of clear negatives (87), hard negatives (45), and counterfactual negatives (88) whose labels are fixed by construction. For evaluation, an agent works in a sandbox for at most 50 turns and outputs valid / invalid / uncertain along with a self-reported evidence layer. The paper evaluates Opus 4.8, GPT-5.6, DeepSeek-V4-Pro, and GLM-5.2; the best accuracies in the three domains are 84.5%, 87.5%, and 78.4%, and no model exceeds 77.8% recall on hard negatives. The authors' central conclusion is that the main weakness of agents lies not in producing evidence but in deciding whether the evidence justifies upholding a report.
Critical strengths
-
A novel task that complements existing benchmarks. Existing benchmarks mostly start from confirmed vulnerabilities or targets and lack "reports that should be rejected." This paper takes the report as the unit of evaluation, retains genuinely rejected submissions, and constructs counterfactual negatives whose labels are fixed by construction, thereby separating "can reproduce" from "should be upheld." As AI-generated vulnerability reports become more common, this problem is of practical relevance.
-
A careful evaluation design with honest disclosure of limitations. Beyond accuracy, the paper recovers investigation depth and audits whether verdicts are supported by the evidence, and it surfaces some thought-provoking observations. For example, runs that execute a PoC tend to be more accurate on valid reports but less accurate on invalid ones. The authors manually reviewed the auditor, performed contamination checks, and explicitly state that labels reflect adjudication outcomes rather than absolute ground truth and that GPT-5.6 uses a different scaffold, which increases the credibility of the results.
Critical weaknesses
-
The central conclusion rests on a small and noisy hard-negative subset. There are only 45 hard negatives (31 software, 14 web), and the paper's main conclusion that "the frontier advantage does not transfer to hard negatives" comes from this subset. Opus 4.8's advantage over the other backends is supported by exact McNemar tests (p ≤ 0.035) without multiple-comparison correction, and the samples may be concentrated in a few projects and thus not independent. In addition, GPT-5.6 uses a different scaffold, so part of the difference may stem from the scaffold. More importantly, according to Appendix A, at least 9 of these 45 samples were closed for reasons such as "already known, out of scope, or bounty policy," and 13 depend on project-specific threat models that the agent cannot see. The metric therefore mixes reasoning ability with guessing project policy. As a result, the conclusions may lack generalizability.
-
Key baselines and controls are missing. Only 19 of the 227 web reports are invalid, so a trivial "always answer valid" baseline would reach about 91.6% accuracy (computed from the figures above), higher than the best model accuracy of 78.4%. Although the paper reports balanced accuracy, it provides no trivial or weak baselines (always-valid, single-pass reasoning without tools, or human triage), so readers cannot tell how much benefit the agents' investigation actually provides.
Questions for authors
See the Weaknesses section above.
3 comments
3. Project concentration and dependence
You also raise a valid concern that reports from the same program may not be statistically independent. The 45 hard negatives span 13 programs, with curl contributing the largest cluster (19 reports, 42%). As a sensitivity check, we recompute recall separately within curl and across the remaining 26 reports in Rebuttal Table 3.
Rebuttal Table 3: Hard-negative recall (%) within and outside the largest program.
| Subset (n) | Opus 4.8 | GPT-5.6 | DeepSeek-V4-Pro | GLM-5.2 |
|---|---|---|---|---|
| curl only (19) | 84.2 | 57.9 | 52.6 | 57.9 |
| Non-curl (26) | 73.1 | 57.7 | 53.8 | 57.7 |
The same qualitative pattern appears both within the dominant curl cluster and outside it: Opus 4.8 remains substantially ahead, while the other three backends remain closely grouped. This check does not eliminate within-program dependence, but it shows that the observed pattern is not driven solely by the largest project. We will clarify this distinction and retain project clustering as a limitation of the hard-negative analysis.
4. Scaffold-swap control
You also note that GPT-5.6 uses a different agent scaffold, so part of the observed difference could arise from the harness rather than the backend itself. To examine this, we reran DeepSeek-V4-Pro—which is evaluated under Claude Code in the main paper—on all 747 reports using OpenAI Codex, the scaffold used by GPT-5.6. We hold the container image, prepared workspace, system and task prompts, reasoning effort, and 50-turn budget fixed.
Rebuttal Table 4: DeepSeek-V4-Pro under Claude Code and OpenAI Codex (%).
| Domain | Scaffold | Coverage | Overall (95% CI) | Balanced | Valid | Invalid |
|---|---|---|---|---|---|---|
| Smart contract | Claude Code | 201/239 | 65.7 [59.8, 71.5] | 65.8 | 65.6 | 65.9 |
| OpenAI Codex | 207/239 | 67.8 [61.9, 73.6] | 68.8 | 64.9 | 72.7 | |
| Traditional software | Claude Code | 259/281 | 76.2 [71.5, 80.8] | 75.0 | 81.0 | 69.0 |
| OpenAI Codex | 249/281 | 78.6 [74.0, 83.3] | 77.7 | 82.7 | 72.6 | |
| Web application | Claude Code | 205/227 | 62.1 [55.5, 68.7] | 65.0 | 61.5 | 68.4 |
| OpenAI Codex | 186/227 | 68.3 [61.7, 74.4] | 63.6 | 69.2 | 57.9 |
Rebuttal Table 5: Investigation behavior of DeepSeek-V4-Pro under each scaffold. Correct reports are out of 747; the other columns cover the 738 reports with token usage logged for both runs.
| Scaffold | Correct reports | PoC executed (%) | Mean output tokens | Mean tool calls |
|---|---|---|---|---|
| Claude Code | 512 | 66 | 11.7k | 23 |
| OpenAI Codex | 538 | 93 | 15.8k | 43 |
The scaffold clearly changes investigation behavior. Under Codex, DeepSeek-V4-Pro executes PoCs much more often, uses more tool calls, and produces longer outputs. The two scaffolds also solve different individual reports: 432 reports are correct under both, 80 only under Claude Code, and 106 only under Codex.
However, these behavioral differences do not produce a statistically significant change in aggregate verdict accuracy. Accuracy increases from 68.5% to 72.0% overall, but the paired difference is not significant (exact McNemar, p = 0.067), and no domain-level difference is significant. DeepSeek-V4-Pro also remains below both frontier backends in every domain. We therefore do not claim that scaffold choice is irrelevant. Rather, this control shows that the scaffold materially affects how the agent investigates, while the main aggregate performance pattern remains robust in this controlled scaffold swap.
Revision. We will add the closure-reason sensitivity analysis, Holm-corrected comparisons, project-concentration analysis, and scaffold-swap control to the revised paper and will state the remaining limitations of the hard-negative subset more explicitly.
zuDF-W2: Key baselines and controls are missing.
We agree that additional baselines are important, particularly because raw accuracy can be misleading in the highly imbalanced web subset. We therefore add both a trivial majority-class baseline and a report-only baseline that removes repository and tool access.
1. Trivial majority-class baseline
Because valid reports are the majority class in every domain, the majority-class baseline is equivalent to always predicting valid.
Rebuttal Table 6: Accuracy and balanced accuracy (%) of trivial baselines per domain.
| Domain | Valid | Invalid | Always-valid acc. | Always-valid bal. | Always-invalid acc. | Always-invalid bal. |
|---|---|---|---|---|---|---|
| Smart contract | 151 | 88 | 63.2 | 50.0 | 36.8 | 50.0 |
| Traditional software | 168 | 113 | 59.8 | 50.0 | 40.2 | 50.0 |
| Web | 208 | 19 | 91.6 | 50.0 | 8.4 | 50.0 |
As you note, the always-valid baseline reaches 91.6% raw accuracy on the web subset because valid reports dominate this subset. However, this apparent performance is entirely driven by class imbalance: the same baseline has only 50.0% balanced accuracy and zero recall on invalid reports. By contrast, the evaluated backends achieve 59.0–87.6% balanced accuracy across domains. An always-valid classifier therefore fails precisely on the invalid reports that a verification or triage system must identify.
Revision. We will therefore foreground balanced accuracy and invalid-report recall more clearly when discussing cross-domain results.
2. Report-only baseline
We also ran the report-only baseline on all 747 reports: each backend receives the same scrubbed report in a single call, without repository access or tools. This directly measures what agentic code investigation contributes beyond reasoning from the report itself.
Rebuttal Table 7: Accuracy (%) of the tool-enabled verification agent and the report-only baseline on all 747 reports, by label type. † marks the significantly higher setting (exact McNemar, p < 0.05).
| Label type (n) | Setting | Opus 4.8 | GPT-5.6 | DeepSeek-V4-Pro | GLM-5.2 |
|---|---|---|---|---|---|
| Valid (527) | Tool-enabled | 79.7 | 82.7 | 68.9 | 62.6 |
| Report-only | 79.5 | 79.9 | 85.2† | 88.6† | |
| Clear negative (87) | Tool-enabled | 92.0 | 89.7 | 77.0 | 83.9† |
| Report-only | 96.6 | 94.3 | 78.2 | 63.2 | |
| Hard negative (45) | Tool-enabled | 77.8 | 57.8 | 53.3 | 57.8 |
| Report-only | 75.6 | 60.0 | 57.8 | 40.0 | |
| Counterfactual negative (88) | Tool-enabled | 95.5† | 88.6† | 65.9† | 71.6† |
| Report-only | 23.9 | 26.1 | 13.6 | 11.4 | |
| All (747) | Tool-enabled | 82.9† | 82.7† | 68.5 | 65.9 |
| Report-only | 74.7 | 74.0 | 74.3† | 73.6† |
Across all 747 reports, the two frontier tool-enabled agents significantly outperform their report-only baselines. The largest gains occur on the 88 counterfactual negatives: every backend improves by 52–72 percentage points. These cases keep the original report unchanged while replacing the vulnerable code with a patched version, so the report alone cannot reveal the correct verdict. This provides a controlled demonstration that code-level investigation contributes information beyond the report.
Outside the counterfactual negatives, the two frontier agents perform similarly to report-only, whereas the open-weight agents perform worse, largely because many tool-enabled runs terminate without a valid verdict. Thus, the baseline reveals a more nuanced result: agentic code investigation is essential when the answer depends on the underlying code state, but current agents do not yet convert additional investigation into uniformly better adjudication across naturally occurring reports.
Revision. We will add the trivial and report-only baselines to the revised paper and foreground balanced accuracy and invalid-report recall for class-imbalanced domains.
We thank you for your detailed comments. Our responses to the weaknesses (W1, W2) are presented below. We agree that the hard-negative subset deserves additional sensitivity analyses because it is relatively small and contains heterogeneous closure reasons. We therefore examine the effect of removing non-technical closures, correct the multiple-comparison analysis, test sensitivity to the dominant project, and add a scaffold-swap control. We also add trivial and report-only baselines to clarify what the reported accuracy reflects.
Thank you for examining the hard-negative subset closely. Removing the 9 non-technical closures strengthens rather than weakens the hard-negative pattern: Opus 4.8’s lead widens, and all three Holm-corrected comparisons are significant on the remaining 36 cases. Below, we further examine closure reasons, project concentration, and scaffold sensitivity.
1. Sensitivity to hard-negative closure reasons
Our closure-reason audit labels all 45 hard negatives: 20 were closed because the reported behavior was intended, 13 were judged not to constitute vulnerabilities under the program’s threat model, 3 were fixed without a stated position, and 9 were closed for non-technical reasons (4 already known, 3 out of scope, and 2 bounty-policy decisions). Rebuttal Table 1 reports hard-negative recall after separating these groups.
Rebuttal Table 1: Hard-negative recall (%) by closure reason.
Removing the 9 explicitly non-technical closures strengthens the main qualitative pattern: Opus 4.8 increases from 77.8% to 86.1% recall, while GPT-5.6 remains close to the two open-weight backends. The clearest technical subset—the 20 intended-behavior cases—shows an even stronger separation for Opus 4.8 (95.0%), whereas GPT-5.6 remains at 55.0%.
The 9 non-technical closures are difficult for all backends (33–56% recall). We retain them in the benchmark because our labels reflect actual program adjudications rather than our own technical reinterpretation, but we will distinguish them explicitly in the revised analysis.
The 13 threat-model-dependent cases show a smaller gap between Opus 4.8 and GPT-5.6 (76.9% vs. 69.2%), suggesting that this subset is more sensitive to program-specific security assumptions. We therefore report these cases separately from the intended-behavior subset.
2. Multiple-comparison correction
We thank you for pointing out that the original pairwise McNemar tests were reported without correction for multiple comparisons. We therefore apply Holm correction across the three comparisons between Opus 4.8 and the other backends, using exact two-sided McNemar tests.
Rebuttal Table 2: Exact two-sided McNemar tests of Opus 4.8 against each backend on hard negatives, with Holm correction across the three comparisons.
On all 45 hard negatives, the Opus 4.8–DeepSeek-V4-Pro comparison remains significant after Holm correction (adjusted p = 0.010), whereas the comparisons with GPT-5.6 and GLM-5.2 become marginal (adjusted p = 0.070). After excluding the 9 non-technical closures, all three comparisons are significant after correction (adjusted p ≤ 0.023). After Holm correction, Opus 4.8 remains significantly stronger on the 36-case subset excluding non-technical closures.