acceptodds
Under review as a conference paper at ICLR 2027

Trident: How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)

Abstract

Autonomous cyber defense systems based on Deep Reinforcement Learning (DRL) have attracted significant research attention, yet remain evaluated almost exclusively against static, heuristic red agents, leaving their robustness against adaptive threats critically understudied. Meanwhile, recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have improved LLM reasoning, but their integration into cybersecurity remains elusive due to the absence of suitable benchmark environments and interaction datasets. To bridge this gap, we introduce Trident, an agentic LLM red teaming framework comprising three components: a dynamic benchmark with isolated sandbox servers spanning CybORG CAGE 4 and CyberWheel, a dataset comprising over 13,000 high-fidelity red-blue interaction trajectories for RLVR, and a “Code-as-Policy” RLVR agentic architecture (Trident Agentic). The latter reformulates red agent training as a contextual bandit via a tripartite Log Summarizer–Planner–Coder design, where a trainable Planner generates complete attack strategies from compressed execution logs, which a frozen Coder translates into executable Python policies deployed against live DRL defenders. Empirical evaluations reveal consistent brittleness in existing defenses: a single trainable 7B planner degrades blue-agent performance by an average of 628% across four state-of-the-art CAGE 4 defenders and eliminates over 99% of the defender’s episodic reward in CyberWheel, while autonomously discovering novel behaviors such as decoy avoidance and adaptive state prioritization that static heuristics entirely fail to uncover.

Primary area:
reinforcement learning
Keywords:
RLVR, Cybersecurity, Agentic LLM, Red Teaming

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MWReOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Trident is a RLVR-based approach to train automated red agents in ACD games. The approach uses a trainable planning agent, which outputs natural language descriptions of strategies, which are then translated into code by a frozen coding agent. The output is then used to generate traces in the simulators, which provides rewards to update the planner’s parameters. This method successfully degrades all blue agents it was trained against by wide margins, more successfully than the other works it was compared to, and contains thorough ablation studies in the supplemental material.

Critical strengths
  • Research on automated attacks in simulated cyber environments is sparse. The findings of this work could in turn be used to train even better blue defenders, and enable further adaptive defense agents
  • The experiments included a PPO-based agent to serve as a baseline to distinguish what makes the LLM-based approach work. I was surprised how poorly the DRL red agents did, even compared to weaker LLM-based attacks.
  • In addition to the experiments, the authors released many traces of their games which appear diverse Having the only trainable parameter be the planner, and executing the model it generates is highly efficient, and makes LLM-based attackers more feasible than prior works which directly use LLM actions as policies.
  • The paper is well-written and easy to follow, and contains many useful experiments beyond the main contribution
Critical weaknesses
  • Table 1 should be organized to emphasize better which benchmarks are static datasets, evaluation tasks for LLMs, and multi-step environments. As it is now, the second row in particular feels very mixed
  • The attack is trained on each defense separately. It would be interesting to see if it learns a general policy if defenders were selected randomly and to see how this generalized policy affects each of the blue agents individually
  • As noted in the limitations section, this approach requires expert-crafted prompts to describe the actions and goals for each environment.
  • Minor: King et al. is cited as a preprint, but it has since been published (DSN 2026)
Questions for authors
  • Is there potential for data leakage about the network configuration? The baseline LLMs used were published in 2024, as was CAGE-4 and CyberWheel. Does this affect the POMDP assumption if it’s possible information about these environments was in the LLMs’ original training data?
  • How generalizable are the trained attackers? E.g. can the attacker trained to exploit King et al. also degrade Singh et al. and Lyu et al.?
Reviewer_M4hLOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Trident to generate attack policies against fixed reinforcement-learning cyber defenders. A trained Planner proposes strategy changes, and a frozen Coder implements them. The Planner learns through GRPO using simulator feedback. Experiments cover CAGE4 and CyberWheel.

Critical strengths

The framework provides a useful way to test defenders beyond fixed rule-based attacks.

The experiments compare the trained Planner with its untrained version in two simulation environments.

Critical weaknesses

The methodological contribution appears limited. The system applies GRPO to a Planner within an existing attack framework. In CAGE4, it modifies three methods while retaining the underlying execution logic. Comparisons with simpler search methods in the same strategy space would help establish the value of this design.

The Planner baselines are limited. The LLM comparisons use Qwen2.5 and GPT-4o, too outdated. The authors also acknowledge that the prompts were developed around the 7B model. These results do not establish an advantage over well-tuned Planner.

Transfer across defenders is unclear. A separate attack Planner is trained for each fixed defender. The results therefore show target-specific optimization, rather than a general attacker that works against unfamiliar defenders.

Questions for authors

How does Trident compare with a stronger Planner given comparable prompt-tuning effort?

Does GRPO outperform simpler search over the same strategy space under a similar execution budget?

Can a Planner trained against one defender attack another without further training? Or do some updated models already can do such things well even without training?

Reviewer_jmwjOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to address a critical evaluation blind spot in autonomous cyber defense: RL-trained defensive agents are trained and evaluated against static, rule-based heuristic attackers, such as finite state machines or simple BFS scripts, leving them vulnerable to adaptive attackers. This paper introduces Trident, an infrastructure/dataset suite and an agentic training architecture. They formulate the problem as a contextual bandit problem via a code-as-policies transformation and isolate the RL signal from the syntax noise by splitting the agent into 3 modules: log summarizer (frozen), trainable planner, and red policy coder (frozen), and optimize the trainable module via GRPO and a dense reward. They collected ~14k trajectories from 17 strategies to serve as initial contexts. The paper found that an adaptive attacker can easily exploit the defenders.

Critical strengths
  1. The paper exposes a critical issue that most of the evaluation attackers are static and heuristic-based, and thus defenders can be easily exploited. (I believe the main blocker to not having an adaptive attacker evaluation is due to limited compute resources, though adaptive attackers have been discussed intensively in the security/safety domain)
  2. The paper proposes a practical formulation of the attacker agent, thus making it easier for the attacker to find exploitation.
  3. Valuable open-source infrastructure and dataset
  4. Insightful alignment analysis
Critical weaknesses
  1. Overclaiming "autonomous discovery" despite explicit prompt engineering (Section 4.2 vs. Appendix G.3.1 L1986-2000 show the LLM is following a hand-crafted prompt instruction rather than discovering a novel tactic from scratch, limiting the evidence for whether the method is scalable without human-written instruction)
  2. Overstating "code-as-policies" in CAGE 4 (Appendix G.2.3 L1785-1824, the planner is operating on finite state machines, which raises the question of whether it is a real code-as-policy)
  3. Unverified role of the context C in the CB formulation. The paper never tests whether the trained planner actually conditions on C at test time, or if it simply collapses to outputting the same static Python strategy regardless of input log. Also, I am curious whether the test set is in-distribution with respect to contexts that have been used for training/validation.
Questions for authors
  1. Can you provide an ablation study running GRPO in cyberwheel without RULE_DECEPTION in the system prompt?
  2. Since the planner's output is a structured numerical JSON schema that parameterizes an existing finite state machine, why is a 7B LLM coder required instead of a deterministic 5-line Python template that directly injects the JSON dictionary for CAGE 4?
  3. Why does adding CoT hurt the 7B model so drastically? What are the exact pass/execution failure rates (pre-filtering) for GPT-4o, Zero-Shot, CoT, and Trident Agentic across both benchmarks?
  4. How many distinct strategies c does the trained planner actually generate?
  5. (comment) Some math might need to be corrected in Eqs. (1),(3),(5)
  6. I expect an experiment on generalization to differen/larger topologies without retraininng

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript Uploaded and Appreciation for the Review Process

Dear Area Chairs and Reviewers,

We have uploaded the revised manuscript incorporating our responses to the reviewers' comments.

All revisions are highlighted in blue for ease of review.

We sincerely appreciate your time and consideration throughout the review process.

(minor) The definition of πc\pi_c is added to improve the clarity of our manuscript (L157). (Updated on Nov 16)

Added the Random Search baseline over the CAGE 4 strategy space (Table 2, Section 4.1–4.2, and Appendix E.2, Eqs. 7–8) as requested by Reviewer M4hL. (Update on Nov 17)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.