acceptodds
Under review as a conference paper at ICLR 2027

CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat

10 views

Abstract

AI agents are demonstrating increasingly strong offensive and defensive cybersecurity capabilities in real world. Current benchmarks mostly evaluate agents in isolation against static targets. Real cyber operations, however, are dynamic and coupled: security postures constantly change as attackers and defenders adapt to each other in real time. Static benchmarks quickly saturate and miss an agent's ability to attack and defend simultaneously under live pressure. To address this gap, we present CyberArena, a dynamic, symmetric adversarial benchmark built around multi-round attack-and-defense combat. In isolated sandboxes, two agents compete under identical rules, acting concurrently as both attackers and defenders over continuously changing services. This converts static targets into evolving, semi-open challenges, directly testing an agent's speed and precision in discovering, exploiting, and patching vulnerabilities. To sustain benchmark's longevity, an automated agentic pipeline continuously generates, reviews, and validates new environments. Furthermore, because pairwise contests are notoriously hard to quantify, we introduce a dynamical model that maps match dynamics into unified, multidimensional scores across offensive, defensive, and domain-specific capabilities. Evaluating 11 frontier LLM agents across 40 challenges in 386 matches, CyberArena separates model capabilities where single-agent baselines hit a ceiling. The trajectories read like a fight rather than an exam: defenders patch one hole and miss the next and certify repairs on stale evidence, while capable agents think a move ahead, planting backdoors for later rounds and, in one case, writing their own packet sniffer to watch the opponent. Beyond a benchmark, CyberArena serves as an adaptable, scalable experimental platform for future research on cybersecurity agents.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, Coding Agents, cybersecurity, benchmark, attack-and-defense, CTF, adversarial evaluation, multi-agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_R2KkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CyberArena, a benchmark for evaluating cybersecurity agents through online attack-and-defense competitions. The main gap identified by the authors in this area is that benchmarks mainly evaluate agents against static targets. To close this gap CyberArena pairs two agents that simultaneously attack each other's services while defending their own across 3 rounds. Both agents have the same services. CyberArena includes a multi-round evaluation protocol, a statistical model for estimating offensive and defensive capabilities and an adaptive game-selection strategy, because exhaustive testing across all possible settings is not feasible. To support extendibility the paper also proposes an agentic pipeline for generating and validating new challenges. The authors evaluate 11 models (each has a single harness) on 40 challenges (compromised services) and 386 matches in total. From these experiment several behavioral patterns are pointed out, including that models tend to generate incomplete vulnerability patches on the defensive side, and persistent backdoor together with adaptation to the opponent actions on the attacker's side.

Critical strengths
  1. Important problem and solution approach: The paper addresses an important limitation of existing cybersecurity agentic benchmarks, which mainly evaluate offensive or defensive capabilities independently against static targets. Introducing online adversarial interactions provides a more realistic evaluation setting that can expose behaviors and capabilities not captured by existing benchmarks. The experimental results and insights support the relevance of this evaluation paradigm.

  2. Substantial implementation and experimental evaluation: The authors present a comprehensive benchmark infrastructure supporting multi-round adversarial interactions, detailed execution monitoring, and automated challenge creation. The evaluation covers a diverse set of vulnerabilities and frontier agents (spanning both SOTA proprietary agents from the leading companies in the industry and the leading open source models), and the behavioral analysis reveals interesting strategies and failure modes.

  3. Adaptive and resource-efficient evaluation strategy: The main algorithmic novelty in the paper is an interesting adaptive game-selection strategy that leverages expected information gain (based on item response theory) and results from previously evaluated matches to prioritize informative matchups. This is especially relevant given the high computational cost of multi-round adversarial evaluations, and provides a principled approach to orchestrate the different evaluation options.

Critical weaknesses

1. Presentation and clarity: The paper would benefit from substantial improvements in writing and the way the main sections are organized. Sections 3 and 4 are difficult to follow, making it hard to understand the details of the proposed framework, its implementation, and mainly motivation behind different design choices (for example why are there 3 rounds of 20 minutes per round and not 5 rounds of 12 minutes?). There are also general writing issues throughout the paper, such as repeated nearly word-to-word statements and sentences that are out of context (for example this sentence in line 86 that starts a new paragraph - "Realizing this paradigm requires a principled way to evaluate which side holds the advantage.").

2. Insufficient justification and lack of motivation of the proposed methodology: The main methodological choices (the latent-ability ranking model, adaptive game-selection strategy, and automated challenge-curation pipeline) are not sufficiently justified or empirically validated through comparisons with alternative approaches or appropriate baselines.

3. Limited scientific depth and novelty: The primary contribution of the paper is the development of a substantial benchmarking infrastructure, mainly its engineering and integration of different techniques. While the framework is relevant and potentially valuable to the agentic security community, the paper offers relatively limited novel methodological or scientific contribution beyond the infrastructure itself. The experimental analysis provides several interesting observations about agent behavior, but these findings are mostly descriptive, and their broader implications or usefulness remain unclear.

Questions for authors
  1. Could the authors revise the main text, especially Sections 3 and 4, to improve clarity and readability? Maybe the additional page allowed in the rebuttal period could provide room to better explain and motivate the key design choices?

  2. What are the practical advantages of the proposed latent-ability ranking model over simpler approaches, including deterministic scoring methods? Could the authors discuss the trade-offs in terms of ranking reliability, interpretability and computational cost?

3.How does the proposed expected-information-gain-based selection strategy compare with uniform random matchmaking or balanced sampling under the same evaluation budget? How does this affect the results in terms of final scores and rankings and in terms of the different insights?

  1. Could the authors add complementary experiment to Figure 7? An experiment against static defenders could help demonstrate and clarify the specific benefits of live adversarial evaluation.

  2. How reliable is the automated challenge-curation pipeline? Can the authors quantify the percentage of candidates successfully validated without human intervention, the frequency of errors detected during human review, and the agreement between automated validation and human assessment?

5 comments

We thank the reviewer for the feedback and have revised the paper to make the motivation of our methods clear.

W1 & Q1: Presentation and motivation

We agree, and our revision focuses on the introduction and Sections 3 and 4. In the introduction, the new third paragraph drops the ambiguous opening and now explains that a task once solved stays solved in a static benchmark but becomes harder against a live adversary, and motivates adversarial environments with precedents from self-play and frontier AI labs. We would welcome further suggestions on any statements that remain unclear.

W1: Round length

We chose 20-minute rounds so that each round leaves enough time to triage, exploit, and patch. In round 1 of the 386 games, 25% of first patches occur after minute 12, whereas 96% of first captures occur by minute 16. Across the three rounds, the capture rate declines with diminishing returns, from 62% to 38% and then 30%.

W2 & Q2: Ranking model

We have revised Section 3.2 to make the motivation clear. Playing every pair of our 11 entrants on all 40 challenges would incur a prohibitive cost of about $32k in model tokens. Following Rashid et al. (2021), we instead fit a model that predicts how each entrant would perform against every opponent on every challenge from a subset of games. Our model is inspired by the attack-minus-defense structure of paired sports models and by item response theory, and it uses attack, defense, and availability round by round rather than only wins and losses.

In 5-fold cross-validation on the 386 ranking games, repeated 10 times, the model predicts held-out winners correctly in 74.6% of cases, compared with 72.7% for Bradley–Terry. The two rankings also agree at Kendall τ = 0.93 (Section 5), with only two adjacent pairs switching places, neither significantly.

W2 & Q3: Adaptive game selection

We have revised Section 3.3 to state the motivation: to obtain accurate estimates with fewer games, we choose games by expected information gain (EIG), adapting the method of Rashid et al. (2021).

Since true standings cannot be observed, we validated it in simulation, drawing true abilities and challenge parameters from the model's prior, fixed before the campaign. Using EIG, uniform, and balanced sampling, we replayed a 386-game campaign in 32 simulations. EIG recovers the true standings slightly more accurately, with a Kendall τ of 0.92 against 0.88 for both uniform and balanced sampling and a lower score error, and it reaches the accuracy of 386 uniformly sampled games with about 30% fewer games.

W2 & Q5: Challenge curation pipeline

We have revised Section 4 to state the motivation: building a challenge is labor-intensive, so we automate it and audit the results. Of the 61 candidate projects that entered the pipeline, 32 were reproduced and proposed as challenges. Team members raised problems on 14 of them, and 6 were rejected for license issues, impossible exploits, or trivial defense routes.

27 challenges received comprehensive code review over multiple iterations, and the rest, mostly CTF services, received a lighter check. In our experience, agent reviewers complemented human reviewers well, particularly in exposing cheating possibilities.

W3: Scientific depth and novelty

We believe this study adds to previous studies by evaluating 11 frontier agents against live opponents at scale in attack–defense CTF. It also shows that a live defender makes solved challenges harder (see Q4).

W3: Descriptive findings

Going beyond anecdotal description, we measured the frequency of emergent behaviors, including failures. For example, defenders hide the flag instead of fixing the flaw in 6% of games, and agents declare repairs working without testing them in 18%. This is in line with OpenAI (2026), which reports such behaviors as qualitative examples, including agents attacking evaluation infrastructure in its security evaluations and attempting to tamper with logs to mislead the grader during training.

Q4: Static-defender comparison

Figure 7 shows that a challenge solvable without any defender becomes harder in the presence of an active defender. On the 30 challenges that GPT-5.6 Sol, Claude Opus 4.8, and GLM-5.3 solve alone, their average capture rate falls from 89% in round 1 to 42% in round 3 once a defender acts. This can benefit evaluation and training by maintaining reward headroom on tasks that agents can already solve. Could you clarify what a static defender might be in our setting? We will address your concern.

References

Thank you for the detailed response and revisions. The new comparisons and explanations are helpful. I still have two main concerns:

  1. Ranking and game selection - The similar results with Bradley-Terry leave the practical benefit and cost of the more complex ranking model unclear. Additionally, the game-selection evaluation depends on the same model assumptions used to guide selection, making it unclear whether the reported savings would generalize to real games. I understand that a full-scale comparison using real games is not possible during the rebuttal period, could a small-scale comparison be feasible?
  2. Static defender - I meant a comparison with a "fixed" opponent, for example, an agent with a fixed defensive approach that cannot change based on experience between rounds. This could help isolate the benefit of interacting with an adaptive opponent.

Overall, I am currently borderline. I recognize the relevance of this work and appreciate the authors’ efforts during the rebuttal. A clearer justification of the ranking model’s value and a comparison with a non-adaptive defense are the main points that could shift my assessment to the positive side.

Thank you for the follow-up and for recognizing the relevance of our work. We address your remaining concerns below.

F1: Benefit of the ranking model

The primary benefit of our model is interpretability. It measures attack, defense, and availability skills separately, whereas Bradley–Terry assigns a single strength value that mixes all three. This separation enables the analysis in Section 5 (Figure 3b), which shows that GPT-5.6 Sol leads mainly through defense, consistent with our observation that it defended its flag in 87% of rounds, against 47–64% for the other entrants.

F1: EIG savings in real games

We agree with your concern. Since true standings are unknown on real games, we compared the two strategies against public security benchmarks as a reference for cybersecurity skills.

To resolve this, we just performed extended experiments compositing two branches of rankings (random and EIG based). Both branches share the 46 random pilot games (batches 1–3, already there). The random branch adds 45 random games from our newly launched experiments (new runs, we will release them also), and the EIG branch reuses the 48 games our strategy selected in batches 4–5 (already there). We fit our ranking model on each branch, derive an early stage ranking with totally approximate ~90 games. We further compare this early stage ranking with CyberGym (Wang et al., 2026a) and ExploitGym (Wang et al., 2026b). We took scores from each benchmark's leaderboard, or else from a model developer's report, and excluded entrants without a score.

A general claim here is benchmarks that measure security skills should broadly agree, as these two do (Kendall's τ = 0.60 on the 5 shared entrants between CyberGym and ExploitGym). Our final ranking from all 386 games also yields τ = 0.43 on CyberGym (8 shared entrants) and 0.80 on ExploitGym (5 shared entrants).

Come back to our extended experiments. The EIG-sample based ranking agrees comparably (0.57 and 0.80), whereas the random-sample ranking agrees way less less (0.29 and 0.20). This suggests that, when at a very limited number of games (5batch, ~90games), EIG sampling helps the ranking algorithm converges faster to benchmark-consistent ranking than the random sampling.

We hope this 5 batches mini experiment justify the effectiveness of EIG for helps ranking converge faster, which saves us money on token costs. Yet we admit that quantifying the exact savings requires a larger experiment.

F3: Fixed versus live opponent

Thank you for the clarification. Defenders that cannot adapt to their attacker are already evaluated by prior benchmarks. In CyberGym-E2E (Shi et al., 2026), an agent's patch is checked against a proof of concept (PoC) it has never seen. Our arena instead measures whether a defender notices its opponent and adapts. This matters in practice, where security engineers act on a vulnerability report with a PoC, run penetration tests, and investigate compromised systems.

Our trajectories reveal such adaptation. In 60 of 108 cases where a defender patched in at least two rounds, the opponent drove the second patch. For example, defenders closed the route the opponent had used, traced a stolen flag to the opponent's commands, or adopted the opponent's fix after their own attack was blocked. A defender blind to its attacker can instead let the same exploit through every round. MiniMax M3, for instance, patched two username-spoofing routes but left case-insensitive name matching open, and the attacker exploited it in all three rounds. This is consistent with CyberGym-E2E's finding that agents often patch a valid vulnerability other than the targeted one.

References

  • Shi, T., et al. (2026). CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities. In ICML 2026.
  • Wang, Z., et al. (2026a). CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale. In ICLR 2026.
  • Wang, Z., et al. (2026b). ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? arXiv:2605.11086.
Reviewer_FSs6OpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CyberArena, in which AI agents are pitted against each other in 1vs1 capture the flag (CTF) arenas. The paper introduces "recursive adversarial improvement," or the improvement of AI systems through adversarial game playing with other AI models. In this particular setting, the authors examine offensive vs defensive cybersecurity capabilities. They find that frontier AI agents are far ahead of open-weight and non-frontier models and that offense and defense capabilities are strongly correlated, in addition to running further experimentation on the challenge environments and agentic failure modes.

Critical strengths

As the authors rightly point out, much of the literature around benchmarking AI agents for cybersecurity (offense and/or defense) does so in static environments. Even works, such as CyberGym E2E, that contain aspects of both offense and defense don't do so in a live environment. However, in real deployment scenarios, incident response is likely to happen in realtime while the possibility remains of agentic AI systems still attacking a target. This paper is the first of its kind that explores attack vs defense in a live simulation. In addition to presenting results, the authors also perform useful further analysis, including an adversarial test of the arena's integrity, which is incredibly valuable given the prevalence of reward hacking in general, but also specifically in cyber evals.

Critical weaknesses

The cybersecurity evaluation space started with Cybench, which is a benchmark of 40 CTF tasks, and one of the primary criticisms from the community was that it lacked realism. Since then, we have moved towards more open-ended environments, including those with real codebases, for evaluating offensive and defensive capabilities. While I understand why the arena-style simulation lends itself well to CTFs, I believe returning the focus to CTF-based evaluations limits the utility of this benchmark, given that CTFs are broadly a test of compute / efficiency and not necessarily latent capabilities (though I would concede that efficiency can be seen as a measurement of latent capabilities, more broadly I would have liked to see more realistic environments).

Questions for authors
  1. When I have considered this experiment on my own time, I have often found it interesting to evaluate the effects of compute combined with speed on outcome, given that this is very often a critical trade-off that practitioners have to make. For example, is a frontier model with low reasoning and fast inference speed more effective than a sub-frontier model with maximum reasoning? I would love to see experiments to this degree as I think it would be incredibly useful.
  2. I'm curious if you considered scenarios where the defender has to do something much simpler to succeed, such as taking a service offline completely? I can imagine a world in which, if you are mid-breach, it is a "win" to take everything offline instead of rushing to an immediate fix, and I think that would be a useful scenario to examine here as well.
3 comments

We thank the reviewer for the positive review. We address the comments and questions below.

W1: Realism of CTF-based evaluation

We agree that CTF challenges do not reflect real-world cybersecurity tasks, and we adopt the format because its verifiable scoring makes a live contest between two agents measurable. To narrow the realism gap, 26 of our 40 challenges are built from real open-source vulnerabilities, including five known to be exploited in the wild. We also agree that fixed objectives limit the agent's freedom beyond the declared target, and we have revised the Limitations section to include this limit as one that future work should address with open-ended objectives and verifiable outcomes.

Q1: Reasoning effort and speed

We conducted a complementary experiment and found that reasoning effort has a small impact on agent performance in CyberArena. Because an agent's speed cannot be set directly, we set different reasoning efforts, which trade thinking per step for speed. We evaluate Claude Opus 4.8 against itself at high and low effort over 12 games, because a single-model setup isolates reasoning effort, whereas pairing a frontier model with a weaker one would confound it with capability. At low effort, Opus 4.8 is evenly matched with its high-effort counterpart (4 wins, 5 draws, and 3 losses for high effort). Low effort reaches the first capture and first patch slightly earlier (medians of 2.2 vs 3.0 and 3.2 vs 4.8 minutes), but neither difference is significant. At both levels, the median first capture and first patch come within 5 minutes, leaving ample time in each 20-minute round.

Q2: Taking the service offline

We agree that taking a service offline is a valid mitigation in practice, but it is easy and blocks every attack for good, so it would not measure an agent's ability to patch a real vulnerability. CyberArena therefore asks defenders to patch the vulnerability without breaking the service's functionality. This follows CyberGym-E2E (Shi et al., 2026), where a patch must also pass the project's functionality tests. To enforce this, the health checker simulates benign users and is designed to reject defenses that remove required features or return canned responses.

References

  • Shi, T., et al. (2026). CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities. In ICML 2026.

Thank you for your responses to my comments and questions. Ultimately I believe that this paper is a far better paper than the majority of the works submitted to this conference and should be accepted.

Dear reviewer FSs6, thank you so much for the recognition of our contribution and, most importantly, the resonance with our vision behind this work! Your constructive suggestions help us further sharpen the work, and we sincerely appreciate the time and effort you devoted to the review.

Reviewer_MntROpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper introduces a dynamic cybersecurity benchmark where two agents attack each other's systems while defending their own. The authors run 11 models against each other in 40 challenges. They find that attack and defense capabilities strongly correlate and that they exhibit interesting behaviors (like performing only partial patches).

Critical strengths
  1. The area is definitely important, especially as AI agents take on more security-related roles. More research is definitely needed
Critical weaknesses
  • The authors position themselves as the first to study dynamic setups between AIs in cybersecurity, ignoring existing work like [1] and [2]. This significantly undermines the novelty of the paper
  • Some of the used CVEs (~10) are before all the models' cutoff dates, some (~10) are within some cutoff dates and not others, while the remaining (~10) are after all cutoff dates. It's unclear how much of the measured performance is shaped by the fact that some models are genuinely unaware of the CVEs, while others are
  • In Appendix E, the authors state that 169 (~30%) of the matches were excluded due to failure, cancellation and other issues. Several details are unclear about the exclusion process, and it's likely that the process biased the results (due to the fact that not all models' matches were excluded at the same rate)

[1] Balassone, Francesco, et al. "Cybersecurity ai: Evaluating agentic cybersecurity in attack/defense ctfs." arXiv preprint arXiv:2510.17521 (2025).

[2] Mayoral-Vilches, Víctor, et al. "Dynamic cyber ranges." arXiv preprint arXiv:2604.24184 (2026).

Update: after rebuttal, I have decided to raise my score to 2 (Weak Reject).

Questions for authors
  1. How much of the performance decline is due to adaptive defense rather than the differences between evaluation protocols (e.g. solo success is measured with two attempts, while contest success is measured per round)?
  2. How do the 169 excluded runs affect the reliability and potential bias of the rankings?
  3. To what extent did knowledge of publicly disclosed vulnerabilities influence the findings?
  4. How robust are the reported scores to the benchmark integrity issues listed in the Appendix?
  5. Can the authors provide the source code to reproduce the experiments?
5 comments

We thank the reviewer for the constructive advice and address the integrity concerns below.

W1: Prior work on live attack–defense evaluation

Thank you and we agree that the two works are relevant precedents in adversarial cybersecurity evaluation. CAIBench (Sanz-Gómez et al., 2025) also includes an attack–defense category. We have revised the related work section to credit them.

We believe our work contributes a large-scale evaluation in which 11 frontier agents attack and defend simultaneously on 40 challenges, 26 of them from real open-source vulnerabilities and five listed by CISA (2026) as exploited in the wild. In comparison, Balassone et al. (2025) play one model against itself, Mayoral-Vilches et al. (2026) fix attacker and defender roles, and CAIBench pits models against its authors' agent on internally developed challenges.

W2 & Q3: Knowledge cutoff and contamination

We conducted a knowledge-cutoff analysis on the three models whose developers state a single knowledge cutoff and found no significant impact on the ranking.

ModelKnowledge cutoffSource
GPT-5.6 SolFebruary 16, 2026OpenAI (2026)
Claude Opus 4.8January 2026Anthropic (2026)
MiniMax M3January 2026MiniMax (2026)

Using the earliest public record of each vulnerability, we split the 14 played open-source challenges disclosed after Feb. 16, 2026 from earlier ones, excluding CTF challenges, which are all older. When we refit the ranking model separately on the two groups, the rankings of all 11 entrants agree with Kendall τ = 0.71, in line with random splits of the same challenges (median τ = 0.67). From the earlier to the later challenges, the scores of GPT-5.6 Sol, Claude Opus 4.8, and MiniMax M3 change by −1.8, −0.3, and +3.5 points, none significantly (all |z| < 1.5).

W3 & Q2: Excluded runs

We agree the details were unclear and clarify our error-handling rule. We count a game only if both agents pass the readiness check, play all three rounds to the end without an infrastructure failure, and leave a record that can be scored. A failed game is retried in the same configuration, up to three attempts in total.

The excluded runs include 42 refusal aborts by Opus 4.8, and no other entrant has this issue. The other 127 excluded runs have the following causes.

CauseRuns
Flag service failed to plant or rotate a flag38
Other platform failures14
Agent failed the readiness check32
Agent harness failed unexpectedly14
Cancelled by an operator25
Malformed results4

Q1: Adaptive defense versus evaluation protocol

We clarify that the agents are not evaluated under two different protocols. Figure 7 shows that challenges an agent can solve alone become harder once it faces a defender. We use solo games only to identify these challenges. Each of GPT-5.6 Sol, GLM-5.3, and Opus 4.8 attacks an undefended service in two single-round tries, to account for round-to-round variation. On 30 challenges solvable by the three agents, the average capture rate falls from 89% in round 1 to 53% in round 2 and 42% in round 3.

For example, in one rceaas game, GPT-5.6 Sol patched a path-traversal flaw within 74 seconds of round 1. Opus 4.8, which had solved this challenge alone, found that the same exploit failed against the patched service but still worked on its own unpatched copy. It concluded in its messages that "team1 patched the traversal".

Q4: Robustness to integrity issues

We re-ranked the entrants after excluding all 42 games in which our automated audit found an accepted shortcut capture or a defender shortcut, such as hiding the flag instead of fixing the flaw, and found little impact. The ranking agrees with the full one at Kendall τ = 0.96, with one swap between two adjacent entrants whose difference is not significant. Because many of these attempts failed, the test, overstates the impact of the flagged shortcuts. Other confirmed failed attempts, such as SSH login to opponent host, are not considered.

Q5: Source code

Yes, we will release our code, including the challenges.

References

Dear Reviewer MntR, we wanted to kindly follow up on our rebuttal. we would greatly appreciate if you could let us know whether our response has addressed your concerns. Here we are always ready to provide any further clarification.

Dear Reviewer MntR,

Thank you again for taking the time to review our paper and for your helpful comments. Since the discussion phase closes soon, we’d really appreciate it if you could look over our rebuttal. If any points still need clarifying or more detail, please let us know. We’re happy to keep the discussion going.

Comments on the paper

1 comment

Summary for the AC

We thank the AC and reviewers. We have responded to all reviewer concerns with analyses and two additional experiments during the rebuttal period.

Strengths recognized by reviewers

We are grateful for the reviewers' recognition and encouraged that they see the value of live adversarial evaluation. R2Kk and FSs6 both highlighted that existing benchmarks test attack or defense against static targets, and that a live contest fills this gap. FSs6 called the work the first of its kind to explore attack versus defense in a live simulation. We thank R2Kk for recognizing the substantial evaluation and the principled adaptive game selection, FSs6 for finding the integrity audit "incredibly valuable" given reward hacking in cyber evals, and MntR for acknowledging the importance of the area.

Shared concerns and our responses

Novelty and scope (MntR W1, R2Kk W3, FSs6 W1)

§2 now credits three prior works on attack and defense evaluation, which use self-play, fixed roles, or an opponent built by the authors. CyberArena instead has 11 frontier agents attack and defend at the same time on 40 challenges, and shows that a live defender makes solvable challenges harder. We use CTFs for their verifiable scoring, and we narrow their realism gap by building 26 challenges from real vulnerabilities, while §6 notes the limit of fixed objectives. On these challenges, our analysis goes beyond description by measuring how often emergent behaviors occur across agents.

What the live contest measures (MntR Q1, R2Kk Q4, FSs6 Q2)

The live contest measures how agents adapt to an opponent. On challenges that agents solve alone, their capture rate drops over three rounds as defenders' patches force them to find a new route. Prior benchmarks such as CyberGym-E2E already evaluate a fixed defender, whereas in our arena the opponent drove the second patch in 60 of 108 cases where a defender patched in at least two rounds. Taking the service offline is a valid mitigation, but it blocks every attack without measuring patching, so defenders must keep the service functional.

Time budget (R2Kk W1, FSs6 Q1)

Our evaluation does not prioritize speed. Each agent has 20 minutes per round to triage, exploit, and patch, since 96% of first captures occur by minute 16. Testing Opus 4.8 against itself, we found that reasoning effort changes speed only slightly and leaves game outcomes evenly matched.

Reviewer-specific concerns and our responses

MntR: integrity of the ranking (W2, W3, Q2–Q5)

We analyzed three sources of bias raised by MntR and found no evidence of impact on the ranking. When we split the open-source challenges by exposure, none of the three models with a clear knowledge cutoff changes significantly in score, and the two rankings agree in line with random splits (τ = 0.71 vs. 0.67). Failed runs are excluded by mechanical checks that never read the score or the winner, and replayed in the same configuration. Removing all 42 games flagged by our integrity audit leaves the ranking nearly unchanged (τ = 0.96). Code will be released on acceptance.

R2Kk: presentation and motivation (W1, W2, Q1–Q3, Q5)

We rewrote the introduction and §3–4 to clarify our motivation. The ranking model predicts all pairings from a subset of games, avoiding a projected cost of about $32k. It matches Bradley–Terry in predicting held-out game outcomes and in the final ranking, while scoring attack, defense, and availability separately for interpretability. EIG selection saves games by choosing the most informative ones. In simulation, it matches the accuracy of uniform sampling with about 30% fewer games, and on a similar number of real games, its ranking agrees better with public security benchmarks than a random selection does. Building challenges by hand is labor-intensive, so the curation pipeline automates it while humans audit the output.

Changes to the paper (highlighted in blue)

  • Abstract and Introduction: revised motivation and contributions (R2Kk).
  • §2: credit to prior live-opponent studies (MntR).
  • §3: motivation for the ranking model and game selection (R2Kk).
  • §4: motivation for the curation pipeline (R2Kk).
  • §6 Limitations: predefined attack objectives (FSs6).
  • Appendix: integrity catalog counts over all 386 games (MntR), and only the 40 evaluated challenges listed (R2Kk).
  • Reproducibility statement: release of the arena code, challenge images, and evaluation scripts (MntR).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.