acceptodds
Under review as a conference paper at ICLR 2027

CVE-Bench Hard: Rigorous Evaluation of AI Agents for Identifying and Exploiting Complex Web Application Vulnerabilities

Abstract

AI agents have already compromised real-world systems through both human directed misuse and evaluation-driven reward hacking, making rigorous assessment of their offensive cybersecurity capabilities an urgent priority. However, existing benchmarks fail to adequately assess attacks against web applications in the wild due to their limitations in task difficulty, coverage of security outcomes, or evaluation rigor. To address these gaps, we introduce CVE-Bench Hard, a set of 20 tasks of complex web-application vulnerability exploitation, each manually selected, reproduced, and verified. Evaluating AI agents’ ability to exploit a target vulnerability requires isolating the target vulnerability from alternative vulnerabilities affecting the same application. To address this challenge efficiently, we develop a pipeline that optimizes application version selection, removes alternative vulnerabilities, and restores the target vulnerability. Our pipeline reduces the number of alternative vulnerabilities requiring manual patches from 672 to 18. We evaluated GPT-5.6 Sol and Opus 5 on CVE-Bench Hard. These models achieved pass@1 rates of 59% and 58%, respectively, suggesting that current models are proficient in applying generic exploits but struggle to compose a chain of weaknesses into a successful exploit given limited observable intermediate feedback.

Primary area:
datasets and benchmarks
Keywords:
AI agent, evaluation, cybersecurity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1DT3OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors contribute a benchmark of 20 complicated tasks to test the ability of LLM-powered agents to identify and exploit vulnerabilities in running webservers. The tasks range in difficulty, with the most complicated ones requiring chaining multiple vulnerabilities together. In addition to the benchmark, the authors describe a pipeline for finding vulnerabilities to use in evaluations, isolating the exact vulnerabilities to be tested on, and validating tasks are logical. The pipeline requires some manual steps (like manually validating tasks are solvable) but, overall, reduces the amount of programming work needed to patch repositories to ensure they only have the exact vulnerabilities tested. The authors show that their vulnerability isolation pipeline eliminates false discovery rates of vulnerabilities when using GPT 5.6 and Opus 5 when they were previously at 41% and 45.5%, respectively.

Critical strengths
  1. The paper shows the importance of vulnerability isolation, as shown in table 6. This is a gap in previous cybersecurity benchmarking research, so it is helpful to a) have empirical evidence that this is a real problem and b) have a roadmap provided by the authors for solving the issue.
  2. Realistic execution environment: using Kali Linux (which real cybersecurity professionals use) along with 20 complex tasks lends credence to the fact that this a rigorous study. Also, the 58/59% pass rates from frontier models suggest the benchmark is not saturated, which means this paper will continue to be relevant into the future.
Critical weaknesses

None

Questions for authors
  • Would this benchmark be difficult to scale? The paper mentions that manual work is involved in reviewing and validating the tasks.
    • A very high 76 percent of potential tasks were dropped after manual review by experts (5). What were the most common causes of tasks being dropped? Would it be a good idea to include an example of a task dropped and the reasoning behind that decision in the appendix? The reason I ask is because 76 percent seems like a quite high proportion of dropped tasks.
1 comment

We thank the reviewer for the positive feedback and helpful questions.

Q1: Would this benchmark be difficult to scale? The paper mentions that manual work is involved in reviewing and validating the tasks.

The manual work in building CVE-Bench Hard has three parts. We think that some of them are feasible to scale.

  1. Reviewing CVEs for task selection. We think that this part is easy to scale because we can define explicit selection rules to guide human annotators or LLM judges.

  2. Reproducing exploits given the vulnerability description and source code. As AI agents become increasingly capable of exploiting a known vulnerability given source code, we think that this step may soon be automated with little human intervention.

  3. Reviewing the patches and transcripts for validity. Our adversarial testing already uses agents to find alternative vulnerabilities, while our version selection reduces the disclosed alternative vulnerabilities requiring patches from 672 to 18 (Section 3.2). However, experts still need to review the transcripts and write and review the patches. Without these manual measures, 42–46% of successful attempts did not exploit the target vulnerability (Table 6).

Q2: A very high 76 percent of potential tasks were dropped after manual review by experts (5). What were the most common causes of tasks being dropped? Would it be a good idea to include an example of a task dropped and the reasoning behind that decision in the appendix? The reason I ask is because 76 percent seems like a quite high proportion of dropped tasks.

We find that low intellectual difficulty and limited reproducibility are the two most common causes. This is because our task filtering relies on severity-oriented CVSS metadata, while vulnerability disclosures do not always establish reproducibility. Following the suggestion, we added two examples of excluded candidates to Appendix A.1.

Reviewer_dhukOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CVE-Bench Hard, a challenging benchmark for evaluating the offensive cybersecurity capabilities of AI agents. The paper curates 20 tasks based on real-world vulnerabilities through a semi-automated pipeline and evaluates two state-of-the-art coding agents on the benchmark. Subsequent analysis of agent trajectories reveals the limitations of current models in identifying and exploiting complex vulnerabilities. Overall, the presentation of the paper is easy to follow, and the proposed contribution is clear.

Critical strengths
  1. Building a harder benchmark, given that the existing CVE-Bench has become nearly saturated with the advancement of foundation models, seems to be a sound motivation. The benchmark curation process is also reasonable.
  2. The benchmark isolates intended vulnerabilities, which shows potential for providing a more accurate way to evaluate the corresponding agents.
  3. Focusing on reward hacking and validating whether agents actually exploit the intended vulnerabilities during evaluation addresses an important research gap.
Critical weaknesses
  1. Table 6 provides the main quantitative evidence supporting the paper's contribution to evaluation rigor. The authors report that, with all evaluation safeguards enabled, the FDR for both models drops to 0%. However, this result relies on human auditing, and no concrete examples are provided to illustrate what evidence the human auditors examined or how their decisions were made. A more detailed explanation of the auditing process, ideally with representative cases, would strengthen the credibility of this result.
  2. While the authors make great efforts to isolate the intended vulnerabilities, the effectiveness of this process is not sufficiently evaluated at the benchmark level. Table 6 reports an FDR of 0% based on the observed agent trajectories, but this does not necessarily indicate that all alternative exploitation paths have been eliminated. A more systematic task-level evaluation, such as testing whether agents can still achieve the attack goals after removing the target vulnerabilities, would strengthen the validity of the benchmark.
  3. The experiments only evaluate proprietary models, which could be somewhat limiting for a benchmark paper. Including open-weight models such as those from the DeepSeek or Qwen series would strengthen the conclusions.
  4. The benchmark contains only 20 tasks, compared with 40 in the original CVE-Bench, which is relatively small. This may limit the statistical reliability and generalizability of the evaluation results.
Questions for authors
  1. For Table 6, could the authors provide more details about the other baseline settings, e.g., how many alternative exploitation paths are available for each task? This would provide a more straightforward comparison of how isolating intended vulnerabilities affects the evaluation.
  2. Since the tasks involve performing cyberattacks, are there any cases where proprietary models refuse to complete the tasks due to their safety alignment?
3 comments

We thank the reviewer for their insightful feedback. We concisely addressed them below and accordingly revised the paper with changes highlighted in blue.

W1: FDR results rely on human auditing. A more detailed explanation of the auditing process, ideally with representative cases, would strengthen the credibility of this result.

In the revised paper, we included an explanation of our auditing process and representative cases in Appendix G. Here, we summarize the key steps of our auditing.

  1. Locate the credited success. For each attempt that passed the grader, we identify the agent actions that earned the score.

  2. Attribute the mechanism. We compare the components of agent actions with the root cause of the target vulnerability. If the credited request does not exploit the target, we count the attempt as a false discovery.

  3. Check rule compliance. In CVE-Bench Hard, we disallow brute-force password cracking, volumetric DoS, and attacking the benchmark infrastructure. We check whether the agent benefited from disallowed action.

W2: While the authors make great efforts to isolate the intended vulnerabilities, the effectiveness of this process is not sufficiently evaluated at the benchmark level. Testing whether agents can still achieve the attack goals after removing the target vulnerabilities would strengthen the validity of the benchmark.

Following the suggestion, we ran Codex with GPT 5.6 Sol once on each of the 20 tasks with the official fixes of target vulnerabilities in place. We find that none of the 20 attempts achieved an enabled security outcome.This provides additional empirical evidence supporting our vulnerability isolation. We added this result to Section 5.3 and discussed the remaining limitation in Section 6.

W3: Include open-weight models such as those from DeepSeek or Qwen series would strengthen the conclusions.

We ran an additional experiment to evaluate Claude Code with DeepSeek V4 Pro on all tasks. With other settings consistent with other agents and models evaluated in Section 5, we show that DeepSeek achieved a pass@1 of 24%, underperforming Codex with GPT 5.6 Sol by 35%. We defer detailed per-task results to Section 5.2 of the revised paper.

W4: The benchmark contains only 20 tasks, compared with 40 in the original CVE-Bench, which is relatively small. This may limit the statistical reliability and generalizability of the evaluation results.

The size of CVE-Bench Hard is consistent with prior widely used benchmarks that have no more than 20 tasks [6, 7]. While we acknowledge that a benchmark of 20 tasks is relatively small, we focus on improving the evaluation rigor of CVE-Bench Hard. In the revised paper, we added a Limitations section to discuss it. We summarize the key points below.

Benchmark size and evaluation rigor. We find that a large benchmark size does not always demonstrate high evaluation rigor. Many widely used large benchmarks exhibit substantial validity issues. For example, EpochAI identified issues in 20.3% of DeepSWE tasks that lead to false negative results [8]. Proximal found issues in 15.1% of CyberGym tasks that lead to false positives and false negatives [9]. In contrast, we prioritize evaluation validity by manually selecting, reproducing, isolating, and adversarially testing each task in CVE-Bench Hard. Without these measures, we show that 42-46% of successful attempts did not exploit the target vulnerability (Table 6).

Diversity and representativeness. We cover 18 diverse functional categories (Table 11), including AI applications, e-commerce, and web infrastructure. Our evaluation also covers all eight security outcomes of CVE-Bench v2.1 [4, 5]. In an exploratory analysis of 100 randomly sampled web-application CVE records published between October 1 and November 11, 2026, we find that 88% could reach at least one security outcome we evaluated, demonstrating the representativeness of our evaluation.

Generalization. Our construction pipeline and the failure mechanism we identified generalize beyond these 20 tasks. First, the construction pipeline and validation checks in Section 3 are reusable to build other rigorous cybersecurity benchmarks. Second, because we define the difficulty dimensions (Table 3) independent of individual tasks, they apply to other complex vulnerabilities.

Q1: For Table 6, how many alternative exploitation paths are available for each task?

For each baseline setting in Table 6, we added the number of identified alternative exploitation paths in Section 5.3 of the revised paper and defer the per-task counts to Appendix B (Table 10). In summary, the configurations in Table 6 differ in three measures:

  1. Vulnerability isolation, which patches publicly disclosed alternative vulnerabilities
  2. Adversarial testing, which identifies additional alternative vulnerabilities
  3. Ingress control, which prevents generic DoS
MeasuresNumber of identified alternative exploitation paths
1 + 2 + 30
1 + 216
125
N/A697

Q2: Since the tasks involve performing cyberattacks, are there any cases where proprietary models refuse to complete the tasks due to their safety alignment?

No. Since we leveraged the Daybreak program of OpenAI and the Cyber Verification Program of Anthropic to run evaluation, we observed no refusals by any of the evaluated proprietary models.


[1] Zhu, Yuxuan, et al. "CVE-bench: A benchmark for AI agents' ability to exploit real-world web application vulnerabilities." ICML, 2025.

[2] Wang, Zhun, et al. "Exploitgym: Can ai agents turn security vulnerabilities into real attacks?." arXiv:2605.11086 (2026).

[3] Alexander Barry. “Brief notes on the OpenAI/Hugging Face incident.” Substack blog post, 2026.

[4] Zhu, Yuxuan, et al. "CVE-bench: A benchmark for AI agents' ability to exploit real-world web application vulnerabilities." ICML (2025).

[5] CVE-Bench Teams. “CVE-Bench v2.1” https://github.com/uiuc-kang-lab/cve-bench

[6] Wijk, Hjalmar, et al. "Re-bench: Evaluating frontier AI R&D capabilities of language model agents against human experts." ICML (2025).

[7] Starace, Giulio, et al. "PaperBench: evaluating AI's ability to replicate AI research." arXiv:2504.01848 (2025).

[8] Epoch AI. “DeepSWE v1.1 – Benchmark review” https://epoch.ai/benchmarks/deepswe/review

[9] Proximal. “Robustly Evaluating Cyber Capabilities” https://www.proximal.ai/blog/cybergym-cyberrl

Dear Reviewer dhuk,

Thank you again for your feedback. In the revision, we added representative audit cases and per-task counts of alternative paths. We also ran two new experiments: GPT 5.6 Sol achieved no scored outcomes across the 20 patched tasks, and DeepSeek V4 Pro achieved 24% pass@1 on the benchmark. We also added a discussion of the limitations of the 20-task benchmark in Section 6.

Please let us know if you have any remaining questions. If any concerns remain, we would be happy to provide further clarification.

Best,

The Authors

Reviewer_j4UMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CVE-Bench Hard, a benchmark containing 20 challenging real-world web vulnerability exploitation tasks. It introduces a vulnerability isolation pipeline and improved evaluation infrastructure to ensure that agent success reflects exploitation of the intended vulnerability.

Critical strengths

The vulnerability isolation pipeline is a valuable contribution that significantly reduces alternative exploitation paths and improves evaluation validity.

The experiments include useful ablation studies and failure analyses, providing insights into current agents' limitations in complex exploitation.

Critical weaknesses

The benchmark contains only 20 tasks, which may limit its diversity and representativeness.

Restoring target vulnerabilities and patching alternative vulnerabilities may alter the original application environment, potentially affecting real-world realism.

Questions for authors

How do the authors ensure that patching alternative vulnerabilities and restoring target vulnerabilities do not introduce unintended changes to the original application behavior?

Given the relatively small benchmark size of 20 tasks, how well do the results generalize to other complex real-world vulnerabilities?

5 comments

We thank the reviewer for their comments. We addressed them below and revised the paper with changes highlighted in blue.

W1 / Q2: The benchmark contains only 20 tasks, which may limit its diversity and representativeness. How well do the results generalize to other complex real-world vulnerabilities?

The size of CVE-Bench Hard is consistent with prior widely used benchmarks that have no more than 20 tasks [9, 10]. While we acknowledge that a benchmark of 20 tasks is relatively small, we focus on improving the evaluation rigor of CVE-Bench Hard. In the revised paper, we added a Limitations section to discuss it. We summarize the key points below.

Benchmark size and evaluation rigor. We find that a large benchmark size does not always demonstrate high evaluation rigor. Many widely used large benchmarks exhibit substantial validity issues. For example, EpochAI identified issues in 20.3% of DeepSWE tasks that lead to false negative results [11]. Proximal found issues in 15.1% of CyberGym tasks that lead to false positives and false negatives [12]. In contrast, we prioritize evaluation validity by manually selecting, reproducing, isolating, and adversarially testing each task in CVE-Bench Hard. Without these measures, we show that 42-46% of successful attempts did not exploit the target vulnerability (Table 6).

Diversity and representativeness. We cover 18 diverse functional categories (Table 11), including AI applications, e-commerce, and web infrastructure. Our evaluation also covers all eight security outcomes of CVE-Bench v2.1 [7, 8]. In an exploratory analysis of 100 randomly sampled web-application CVE records published between October 1 and November 11, 2026, we find that 88% could reach at least one security outcome we evaluated, demonstrating the representativeness of our evaluation.

Generalization. Our construction pipeline and the failure mechanism we identified generalize beyond these 20 tasks. First, the construction pipeline and validation checks in Section 3 are reusable to build other rigorous cybersecurity benchmarks. Second, because we define the difficulty dimensions (Table 3) independent of individual tasks, they apply to other complex vulnerabilities.

W2 / Q1: How do the authors ensure that patching alternative vulnerabilities and restoring target vulnerabilities do not introduce unintended changes to the original application behavior?

When restoring target vulnerabilities and patching alternative vulnerabilities, we ensured realism by keeping each application close to its official release and validating every change. Our code changes affect application behaviors only for requests that exercise the target or a patched alternative vulnerability, while all other requests go through the official code. In the paper, we revised Section 3.2 and Appendix A.3 to describe our patching and restoration methods. We summarize the key points here.

Restoring target vulnerability. We first tried to restore each target vulnerability by reverting its official patch directly. This worked for 17 vulnerabilities. For 3 others, since the affected code was rewritten after the official fix, reverting the patch does not reintroduce the vulnerability. We follow the official code before the rewrite to re-implement only the code path taken by the requests that trigger target vulnerabilities.

Patching alternative vulnerabilities. We first tried to backport the official fixes of alternative vulnerabilities. When backporting failed or official fixes were unavailable, we patched in the following order:

  1. For a generic DoS vulnerability, we bounded the resource use following established practice [1].

  2. For a vulnerability that impacts only an optional feature off the exploitation path of the target vulnerability, we disabled the feature. This follows real-world established practice recommended by cybersecurity professionals [2].

  3. Otherwise, we implemented a custom minimal fix of one of the three established types: sanitizing untrusted input [3, 4], adding a permission check [5], or serializing a race [6].

Validation for avoiding unintended changes. We verify every task with the checks in Table 2: the reference solution succeeds in both the original and the isolated environments and fails without restoration, and each identified alternative route succeeds without its fix and fails with it.

[1] OWASP Team. “API4:2023 Unrestricted Resource Consumption” https://api-security.owasp.org/editions/2023/en/0xa4-unrestricted-resource-consumption

[2] Scarfone, Karen, Wayne Jansen, and Miles Tracy. "Guide to general server security." NIST Special Publication 800-123 (2008).

[3] Balzarotti, Davide, et al. "Saner: Composing static and dynamic analysis to validate sanitization in web applications." IEEE Symposium on Security and Privacy (2008).

[4] Maffeis, Sergio, John C. Mitchell, and Ankur Taly. "Isolating JavaScript with filters, rewriting, and wrappers." European Symposium on Research in Computer Security (2009).

[5] Sun, Fangqi, Liang Xu, and Zhendong Su. "Static detection of access control vulnerabilities in web applications." USENIX Security Symposium (2011).

[6] MySQL Team. “MySQL 8.4 Reference Manual: Locking Reads.” https://dev.mysql.com/doc/refman/8.4/en/innodb-locking-reads.html

[7] Zhu, Yuxuan, et al. "CVE-bench: A benchmark for AI agents' ability to exploit real-world web application vulnerabilities." ICML (2025).

[8] CVE-Bench Teams. “CVE-Bench v2.1” https://github.com/uiuc-kang-lab/cve-bench

[9] Wijk, Hjalmar, et al. "Re-bench: Evaluating frontier AI R&D capabilities of language model agents against human experts." ICML (2025).

[10] Starace, Giulio, et al. "PaperBench: evaluating AI's ability to replicate AI research." arXiv:2504.01848 (2025).

[11] Epoch AI. “DeepSWE v1.1 – Benchmark review” https://epoch.ai/benchmarks/deepswe/review

[12] Proximal. “Robustly Evaluating Cyber Capabilities” https://www.proximal.ai/blog/cybergym-cyberrl

Dear Reviewer j4UM,

Thank you again for your feedback. We added a Limitations section on the generalizability of the 20-task benchmark and clarified how we verified that isolation preserved the intended application behavior in Section 3.2 and Appendix A.3 of the revised paper.

Please let us know whether these updates address your concerns. We would be happy to address any remaining questions.

Best, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.