- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AgentCyberRange, a benchmark for evaluating autonomous cyber agents on web exploitation and multi-host post-exploitation. It contains 110 vulnerabilities across 15 applications and eight segmented cyber ranges with 156 hosts. The evaluation combines effect-based verification, PoC-aware vulnerability attribution, and host-compromise markers. Experiments with seven agent-model configurations show substantial exploitation capabilities, alongside limitations in attack-surface exploration and progress across dependent stages. The authors further examine these limitations through hint-level experiments and analysis of execution traces.
Critical strengths
-
The benchmark evaluates dependencies that are difficult to capture with isolated vulnerability tasks. Progress through the segmented ranges requires agents to recover credentials, establish pivots, and use information obtained from earlier compromises. Decoy services and restricted network reachability add practical challenges to target selection and exploration. These features make the benchmark useful for studying multi-stage agent behavior.
-
The verification procedure provides concrete evidence of attack outcomes. Observable security effects are combined with PoC-aware attribution to distinguish vulnerabilities that produce similar effects. Appendix B.2 also reports human validation of 297 sampled vulnerability reports. The hint-level experiments complement the aggregate scores by showing how endpoint information and topology information affect performance differently across the two tracks.
Critical weaknesses
-
The main post-exploitation metric measures partial compromise more directly than reliable completion of an intrusion. Table 8 reports performance over 41 attack-chain targets; the headline 78.05% therefore does not represent the probability of completing an entire range. Target-level progress is a useful metric, but the paper's emphasis on sustained multi-stage intrusions also calls for explicit completion rates. Figure 11 shows progress within individual ranges, yet per-run counts of completed attack chains are not reported. This limits the quantitative support for conclusions about end-to-end reliability.
-
The treatment of interrupted runs requires further clarification. Section 4.3 reports scores over completed runs and excludes 12 safety-related refusal trials for Claude-Opus-4.7. The unit of an excluded trial and the distribution of exclusions across ranges and hint levels are unclear. Performance conditional on completion should be distinguished from performance over all attempted runs, with refusals and infrastructure failures reported separately. This distinction matters when interpreting the results as measures of operational reliability.
Questions for authors
-
For the existing Level-0 evaluations, how often does each system complete all required attack-chain targets in a range? Please provide per-run completion counts and clarify how user-level and root-level markers contribute to the 41-target metric.
-
What constitutes one of the 12 excluded refusal trials? Please report attempted, completed, refused, and infrastructure-failed run counts by system and hint level, and indicate how the scores change when refusals are counted as unsuccessful attempts.