- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CyberArena, a benchmark for evaluating cybersecurity agents through online attack-and-defense competitions. The main gap identified by the authors in this area is that benchmarks mainly evaluate agents against static targets. To close this gap CyberArena pairs two agents that simultaneously attack each other's services while defending their own across 3 rounds. Both agents have the same services. CyberArena includes a multi-round evaluation protocol, a statistical model for estimating offensive and defensive capabilities and an adaptive game-selection strategy, because exhaustive testing across all possible settings is not feasible. To support extendibility the paper also proposes an agentic pipeline for generating and validating new challenges. The authors evaluate 11 models (each has a single harness) on 40 challenges (compromised services) and 386 matches in total. From these experiment several behavioral patterns are pointed out, including that models tend to generate incomplete vulnerability patches on the defensive side, and persistent backdoor together with adaptation to the opponent actions on the attacker's side.
Critical strengths
-
Important problem and solution approach: The paper addresses an important limitation of existing cybersecurity agentic benchmarks, which mainly evaluate offensive or defensive capabilities independently against static targets. Introducing online adversarial interactions provides a more realistic evaluation setting that can expose behaviors and capabilities not captured by existing benchmarks. The experimental results and insights support the relevance of this evaluation paradigm.
-
Substantial implementation and experimental evaluation: The authors present a comprehensive benchmark infrastructure supporting multi-round adversarial interactions, detailed execution monitoring, and automated challenge creation. The evaluation covers a diverse set of vulnerabilities and frontier agents (spanning both SOTA proprietary agents from the leading companies in the industry and the leading open source models), and the behavioral analysis reveals interesting strategies and failure modes.
-
Adaptive and resource-efficient evaluation strategy: The main algorithmic novelty in the paper is an interesting adaptive game-selection strategy that leverages expected information gain (based on item response theory) and results from previously evaluated matches to prioritize informative matchups. This is especially relevant given the high computational cost of multi-round adversarial evaluations, and provides a principled approach to orchestrate the different evaluation options.
Critical weaknesses
1. Presentation and clarity: The paper would benefit from substantial improvements in writing and the way the main sections are organized. Sections 3 and 4 are difficult to follow, making it hard to understand the details of the proposed framework, its implementation, and mainly motivation behind different design choices (for example why are there 3 rounds of 20 minutes per round and not 5 rounds of 12 minutes?). There are also general writing issues throughout the paper, such as repeated nearly word-to-word statements and sentences that are out of context (for example this sentence in line 86 that starts a new paragraph - "Realizing this paradigm requires a principled way to evaluate which side holds the advantage.").
2. Insufficient justification and lack of motivation of the proposed methodology: The main methodological choices (the latent-ability ranking model, adaptive game-selection strategy, and automated challenge-curation pipeline) are not sufficiently justified or empirically validated through comparisons with alternative approaches or appropriate baselines.
3. Limited scientific depth and novelty: The primary contribution of the paper is the development of a substantial benchmarking infrastructure, mainly its engineering and integration of different techniques. While the framework is relevant and potentially valuable to the agentic security community, the paper offers relatively limited novel methodological or scientific contribution beyond the infrastructure itself. The experimental analysis provides several interesting observations about agent behavior, but these findings are mostly descriptive, and their broader implications or usefulness remain unclear.
Questions for authors
-
Could the authors revise the main text, especially Sections 3 and 4, to improve clarity and readability? Maybe the additional page allowed in the rebuttal period could provide room to better explain and motivate the key design choices?
-
What are the practical advantages of the proposed latent-ability ranking model over simpler approaches, including deterministic scoring methods? Could the authors discuss the trade-offs in terms of ranking reliability, interpretability and computational cost?
3.How does the proposed expected-information-gain-based selection strategy compare with uniform random matchmaking or balanced sampling under the same evaluation budget? How does this affect the results in terms of final scores and rankings and in terms of the different insights?
-
Could the authors add complementary experiment to Figure 7? An experiment against static defenders could help demonstrate and clarify the specific benefits of live adversarial evaluation.
-
How reliable is the automated challenge-curation pipeline? Can the authors quantify the percentage of candidates successfully validated without human intervention, the frequency of errors detected during human review, and the agreement between automated validation and human assessment?
5 comments
Thank you for the detailed response and revisions. The new comparisons and explanations are helpful. I still have two main concerns:
- Ranking and game selection - The similar results with Bradley-Terry leave the practical benefit and cost of the more complex ranking model unclear. Additionally, the game-selection evaluation depends on the same model assumptions used to guide selection, making it unclear whether the reported savings would generalize to real games. I understand that a full-scale comparison using real games is not possible during the rebuttal period, could a small-scale comparison be feasible?
- Static defender - I meant a comparison with a "fixed" opponent, for example, an agent with a fixed defensive approach that cannot change based on experience between rounds. This could help isolate the benefit of interacting with an adaptive opponent.
Overall, I am currently borderline. I recognize the relevance of this work and appreciate the authors’ efforts during the rebuttal. A clearer justification of the ranking model’s value and a comparison with a non-adaptive defense are the main points that could shift my assessment to the positive side.
Thank you for the follow-up and for recognizing the relevance of our work. We address your remaining concerns below.
F1: Benefit of the ranking model
The primary benefit of our model is interpretability. It measures attack, defense, and availability skills separately, whereas Bradley–Terry assigns a single strength value that mixes all three. This separation enables the analysis in Section 5 (Figure 3b), which shows that GPT-5.6 Sol leads mainly through defense, consistent with our observation that it defended its flag in 87% of rounds, against 47–64% for the other entrants.
F1: EIG savings in real games
We agree with your concern. Since true standings are unknown on real games, we compared the two strategies against public security benchmarks as a reference for cybersecurity skills.
To resolve this, we just performed extended experiments compositing two branches of rankings (random and EIG based). Both branches share the 46 random pilot games (batches 1–3, already there). The random branch adds 45 random games from our newly launched experiments (new runs, we will release them also), and the EIG branch reuses the 48 games our strategy selected in batches 4–5 (already there). We fit our ranking model on each branch, derive an early stage ranking with totally approximate ~90 games. We further compare this early stage ranking with CyberGym (Wang et al., 2026a) and ExploitGym (Wang et al., 2026b). We took scores from each benchmark's leaderboard, or else from a model developer's report, and excluded entrants without a score.
A general claim here is benchmarks that measure security skills should broadly agree, as these two do (Kendall's τ = 0.60 on the 5 shared entrants between CyberGym and ExploitGym). Our final ranking from all 386 games also yields τ = 0.43 on CyberGym (8 shared entrants) and 0.80 on ExploitGym (5 shared entrants).
Come back to our extended experiments. The EIG-sample based ranking agrees comparably (0.57 and 0.80), whereas the random-sample ranking agrees way less less (0.29 and 0.20). This suggests that, when at a very limited number of games (5batch, ~90games), EIG sampling helps the ranking algorithm converges faster to benchmark-consistent ranking than the random sampling.
We hope this 5 batches mini experiment justify the effectiveness of EIG for helps ranking converge faster, which saves us money on token costs. Yet we admit that quantifying the exact savings requires a larger experiment.
F3: Fixed versus live opponent
Thank you for the clarification. Defenders that cannot adapt to their attacker are already evaluated by prior benchmarks. In CyberGym-E2E (Shi et al., 2026), an agent's patch is checked against a proof of concept (PoC) it has never seen. Our arena instead measures whether a defender notices its opponent and adapts. This matters in practice, where security engineers act on a vulnerability report with a PoC, run penetration tests, and investigate compromised systems.
Our trajectories reveal such adaptation. In 60 of 108 cases where a defender patched in at least two rounds, the opponent drove the second patch. For example, defenders closed the route the opponent had used, traced a stolen flag to the opponent's commands, or adopted the opponent's fix after their own attack was blocked. A defender blind to its attacker can instead let the same exploit through every round. MiniMax M3, for instance, patched two username-spoofing routes but left case-insensitive name matching open, and the attacker exploited it in all three rounds. This is consistent with CyberGym-E2E's finding that agents often patch a valid vulnerability other than the targeted one.
References
- Shi, T., et al. (2026). CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities. In ICML 2026.
- Wang, Z., et al. (2026a). CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale. In ICLR 2026.
- Wang, Z., et al. (2026b). ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? arXiv:2605.11086.
We thank the reviewer for the feedback and have revised the paper to make the motivation of our methods clear.
W1 & Q1: Presentation and motivation
We agree, and our revision focuses on the introduction and Sections 3 and 4. In the introduction, the new third paragraph drops the ambiguous opening and now explains that a task once solved stays solved in a static benchmark but becomes harder against a live adversary, and motivates adversarial environments with precedents from self-play and frontier AI labs. We would welcome further suggestions on any statements that remain unclear.
W1: Round length
We chose 20-minute rounds so that each round leaves enough time to triage, exploit, and patch. In round 1 of the 386 games, 25% of first patches occur after minute 12, whereas 96% of first captures occur by minute 16. Across the three rounds, the capture rate declines with diminishing returns, from 62% to 38% and then 30%.
W2 & Q2: Ranking model
We have revised Section 3.2 to make the motivation clear. Playing every pair of our 11 entrants on all 40 challenges would incur a prohibitive cost of about $32k in model tokens. Following Rashid et al. (2021), we instead fit a model that predicts how each entrant would perform against every opponent on every challenge from a subset of games. Our model is inspired by the attack-minus-defense structure of paired sports models and by item response theory, and it uses attack, defense, and availability round by round rather than only wins and losses.
In 5-fold cross-validation on the 386 ranking games, repeated 10 times, the model predicts held-out winners correctly in 74.6% of cases, compared with 72.7% for Bradley–Terry. The two rankings also agree at Kendall τ = 0.93 (Section 5), with only two adjacent pairs switching places, neither significantly.
W2 & Q3: Adaptive game selection
We have revised Section 3.3 to state the motivation: to obtain accurate estimates with fewer games, we choose games by expected information gain (EIG), adapting the method of Rashid et al. (2021).
Since true standings cannot be observed, we validated it in simulation, drawing true abilities and challenge parameters from the model's prior, fixed before the campaign. Using EIG, uniform, and balanced sampling, we replayed a 386-game campaign in 32 simulations. EIG recovers the true standings slightly more accurately, with a Kendall τ of 0.92 against 0.88 for both uniform and balanced sampling and a lower score error, and it reaches the accuracy of 386 uniformly sampled games with about 30% fewer games.
W2 & Q5: Challenge curation pipeline
We have revised Section 4 to state the motivation: building a challenge is labor-intensive, so we automate it and audit the results. Of the 61 candidate projects that entered the pipeline, 32 were reproduced and proposed as challenges. Team members raised problems on 14 of them, and 6 were rejected for license issues, impossible exploits, or trivial defense routes.
27 challenges received comprehensive code review over multiple iterations, and the rest, mostly CTF services, received a lighter check. In our experience, agent reviewers complemented human reviewers well, particularly in exposing cheating possibilities.
W3: Scientific depth and novelty
We believe this study adds to previous studies by evaluating 11 frontier agents against live opponents at scale in attack–defense CTF. It also shows that a live defender makes solved challenges harder (see Q4).
W3: Descriptive findings
Going beyond anecdotal description, we measured the frequency of emergent behaviors, including failures. For example, defenders hide the flag instead of fixing the flaw in 6% of games, and agents declare repairs working without testing them in 18%. This is in line with OpenAI (2026), which reports such behaviors as qualitative examples, including agents attacking evaluation infrastructure in its security evaluations and attempting to tamper with logs to mislead the grader during training.
Q4: Static-defender comparison
Figure 7 shows that a challenge solvable without any defender becomes harder in the presence of an active defender. On the 30 challenges that GPT-5.6 Sol, Claude Opus 4.8, and GLM-5.3 solve alone, their average capture rate falls from 89% in round 1 to 42% in round 3 once a defender acts. This can benefit evaluation and training by maintaining reward headroom on tasks that agents can already solve. Could you clarify what a static defender might be in our setting? We will address your concern.
References