- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors contribute a benchmark of 20 complicated tasks to test the ability of LLM-powered agents to identify and exploit vulnerabilities in running webservers. The tasks range in difficulty, with the most complicated ones requiring chaining multiple vulnerabilities together. In addition to the benchmark, the authors describe a pipeline for finding vulnerabilities to use in evaluations, isolating the exact vulnerabilities to be tested on, and validating tasks are logical. The pipeline requires some manual steps (like manually validating tasks are solvable) but, overall, reduces the amount of programming work needed to patch repositories to ensure they only have the exact vulnerabilities tested. The authors show that their vulnerability isolation pipeline eliminates false discovery rates of vulnerabilities when using GPT 5.6 and Opus 5 when they were previously at 41% and 45.5%, respectively.
Critical strengths
- The paper shows the importance of vulnerability isolation, as shown in table 6. This is a gap in previous cybersecurity benchmarking research, so it is helpful to a) have empirical evidence that this is a real problem and b) have a roadmap provided by the authors for solving the issue.
- Realistic execution environment: using Kali Linux (which real cybersecurity professionals use) along with 20 complex tasks lends credence to the fact that this a rigorous study. Also, the 58/59% pass rates from frontier models suggest the benchmark is not saturated, which means this paper will continue to be relevant into the future.
Critical weaknesses
None
Questions for authors
- Would this benchmark be difficult to scale? The paper mentions that manual work is involved in reviewing and validating the tasks.
- A very high 76 percent of potential tasks were dropped after manual review by experts (5). What were the most common causes of tasks being dropped? Would it be a good idea to include an example of a task dropped and the reasoning behind that decision in the appendix? The reason I ask is because 76 percent seems like a quite high proportion of dropped tasks.
We thank the reviewer for the positive feedback and helpful questions.
The manual work in building CVE-Bench Hard has three parts. We think that some of them are feasible to scale.
Reviewing CVEs for task selection. We think that this part is easy to scale because we can define explicit selection rules to guide human annotators or LLM judges.
Reproducing exploits given the vulnerability description and source code. As AI agents become increasingly capable of exploiting a known vulnerability given source code, we think that this step may soon be automated with little human intervention.
Reviewing the patches and transcripts for validity. Our adversarial testing already uses agents to find alternative vulnerabilities, while our version selection reduces the disclosed alternative vulnerabilities requiring patches from 672 to 18 (Section 3.2). However, experts still need to review the transcripts and write and review the patches. Without these manual measures, 42–46% of successful attempts did not exploit the target vulnerability (Table 6).
We find that low intellectual difficulty and limited reproducibility are the two most common causes. This is because our task filtering relies on severity-oriented CVSS metadata, while vulnerability disclosures do not always establish reproducibility. Following the suggestion, we added two examples of excluded candidates to Appendix A.1.