acceptodds
Under review as a conference paper at ICLR 2027

AgentCyberRange: Evaluating Frontier AI Systems in Realistic Cyber Ranges

Abstract

Frontier AI agents can increasingly discover and exploit real-world vulnerabilities, but their ability to sustain realistic cyberattacks remains poorly understood. Existing benchmarks largely evaluate bounded tasks around individual vulnerabilities or targets, whereas real intrusions are open-ended and stateful: agents must discover attack surfaces, establish footholds, accumulate information and privileges, and reuse them as network reachability evolves. In this paper, we introduce AgentCyberRange, an open and reproducible cyber-range benchmark for evaluating frontier AI agents under such conditions. AgentCyberRange contains 110 vulnerabilities across 15 real web applications and eight enterprise-like multi-host cyber ranges with 156 hosts, spanning web exploitation and post-exploitation. Its state-coupled attack chains require progress to carry across stages, while effect-based oracles and PoC-aware verification enable reliable evaluation despite open-ended attack paths. Across seven frontier AI systems evaluated under matched prompts and budgets, GPT-5.6-Sol with Codex achieves the strongest performance, solving 36.1% of web-exploitation tasks and 78.1% of post-exploitation tasks. Agents also discover previously unknown vulnerabilities and adapt payloads to bypass host defenses. However, performance degrades with attack depth and state dependencies: agents frequently miss hidden attack surfaces, fail to reuse information acquired earlier, and exhibit substantial run-to-run instability, i.e., reliably sustaining multi-stage intrusions remains a significant challenge for frontier AI agents.

Primary area:
datasets and benchmarks
Keywords:
Cybersecurity, AI, Agents

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_rBSkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AgentCyberRange, a benchmark for evaluating autonomous cyber agents on web exploitation and multi-host post-exploitation. It contains 110 vulnerabilities across 15 applications and eight segmented cyber ranges with 156 hosts. The evaluation combines effect-based verification, PoC-aware vulnerability attribution, and host-compromise markers. Experiments with seven agent-model configurations show substantial exploitation capabilities, alongside limitations in attack-surface exploration and progress across dependent stages. The authors further examine these limitations through hint-level experiments and analysis of execution traces.

Critical strengths
  1. The benchmark evaluates dependencies that are difficult to capture with isolated vulnerability tasks. Progress through the segmented ranges requires agents to recover credentials, establish pivots, and use information obtained from earlier compromises. Decoy services and restricted network reachability add practical challenges to target selection and exploration. These features make the benchmark useful for studying multi-stage agent behavior.

  2. The verification procedure provides concrete evidence of attack outcomes. Observable security effects are combined with PoC-aware attribution to distinguish vulnerabilities that produce similar effects. Appendix B.2 also reports human validation of 297 sampled vulnerability reports. The hint-level experiments complement the aggregate scores by showing how endpoint information and topology information affect performance differently across the two tracks.

Critical weaknesses
  1. The main post-exploitation metric measures partial compromise more directly than reliable completion of an intrusion. Table 8 reports performance over 41 attack-chain targets; the headline 78.05% therefore does not represent the probability of completing an entire range. Target-level progress is a useful metric, but the paper's emphasis on sustained multi-stage intrusions also calls for explicit completion rates. Figure 11 shows progress within individual ranges, yet per-run counts of completed attack chains are not reported. This limits the quantitative support for conclusions about end-to-end reliability.

  2. The treatment of interrupted runs requires further clarification. Section 4.3 reports scores over completed runs and excludes 12 safety-related refusal trials for Claude-Opus-4.7. The unit of an excluded trial and the distribution of exclusions across ranges and hint levels are unclear. Performance conditional on completion should be distinguished from performance over all attempted runs, with refusals and infrastructure failures reported separately. This distinction matters when interpreting the results as measures of operational reliability.

Questions for authors
  1. For the existing Level-0 evaluations, how often does each system complete all required attack-chain targets in a range? Please provide per-run completion counts and clarify how user-level and root-level markers contribute to the 41-target metric.

  2. What constitutes one of the 12 excluded refusal trials? Please report attempted, completed, refused, and infrastructure-failed run counts by system and hint level, and indicate how the scores change when refusals are counted as unsuccessful attempts.

Reviewer_PtT3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a cybersecurity benchmark (AgentCyberRange), containing 110 vulnerabilities for 15 real web applications. It evaluates on 7 frontier systems on web exploitation and post exploitation. GPT-5.6-Sol with Codex performs strongest.

Critical strengths

Moving cybersecurity tasks toward more realistic settings is important.

Critical weaknesses

The independent effects of model vs. scaffold are not possible to disentangle.

It is unclear how zero days are verified.

56 of the vulnerabilities were previously disclosed. What evidence of task leakage, retrieval of CVE etc. exists? I would suggest reporting results separately.

The paper should tests for statistical significance.

While the web verifiers seem meaningful, the post exploitation verifier (Appendix A3) seems much more general and weaker. This really affects the interpretation of the differential success rate for web vs. post exploitation.

Questions for authors

Can you address the questions above?

Reviewer_cEmdOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper creates two tracks of "cyber ranges", which test an AI agent's ability to carry out long horizon cyber attacks in realistic environments. The first track if "Web Exploitation", which involves finding exploits in real world web apps to gain an initial foodhold in a remote environment. There are 15 webapps, and the environments are constructed with known and synthetic vulnerabilities. The second track is "post exploitation", which involves steps after the web exploitation phase that is performed after gaining access to an internal environment, which includes privilege escalation, getting private data or credentials, etc. This part of the benchmark involves 8 cyber ranges, each involving a number of distinct hosts. The paper evaluates a range of models in this environment, across various difficulty levels (where easier levels mean giving the model more background info/hints about the environment/task). The findings are that frontier models such as GPT-5.6-Sol perform noticeably better than non frontier models. For Web Exploitation GPT 5.6 Sol achieves around 35%, and for Post Exploitation it achieves around 80%. Meanwhile the worst model Kimi 2.6 only achieves about 3% and 12% respectively. The paper observes that a major failure mode is that the models fail to do sufficient exploration of attack surfaces, and waste too much tokens on useless hosts. They also observe that models discovered vulnerabilities the authors were not previously aware of. In summary, the author's contributions are the production of realistic cyber ranges, and experiments on running various LLMs on these cyber ranges.

Critical strengths
  • Large and comprehensive set of applications and vulnerabilities, designed by cybersecurity experts
  • Insightful results about state of cyber capability and failure modes of current systems
Critical weaknesses

None

Questions for authors
  • It is stated that a fixed step budget is used. This is minor, but are there numbers for how many steps on average it took agents to complete the task for successes? (Might be more informative than token count)
  • How were the different cyber ranges and web environments chosen? In particular, how did you make sure they are sufficiently different from each other?
  • Were agents allowed to access the internet (e.g. to search for things) for the evaluations?
  • If an agent stops before the task is completed, what happens? Is an additional prompt sent telling it to continue?
  • (Suggestion) Did agents ever demonstrate evaluation awareness (e.g. saying things that suggest it realizes it is in an evaluation and not a real environment?)
  • (Suggestion) Perhaps analysis for whether ranges involving older/more well known vulnerabilities were easier would be useful
  • (Suggestion) It would be cool to see how well professional humans perform within these cyber ranges.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.