acceptodds
Withdrawn from ICLR 2027

Cyberworld: World Models for Sample-Efficient Autonomous Cyber Defense

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deep reinforcement learning has become a prominent approach to autonomous cyber defense. Existing methods are predominantly model-free and consequently require extensive environment interaction. World models provide an alternative by learning predictive dynamics and optimizing policies through imagined trajectories, yielding substantial gains in sample efficiency in robotics and embodied control. Extending this paradigm to cybersecurity raises a fundamental question: what should constitute the "world" in a cyber world model? We introduce CyberWorld, a Dreamer-style world modeling framework that learns latent cyber dynamics from vector, graph, textual, and multimodal representations of the defended network. Across all four scoreable CyberWheel attack strategies, the graph-based CyberWorld variant exceeds a strategy-agnostic control after 3.6k–15.8k environment steps, compared with millions of steps required by model-free PPO. Across representation choices, graph structure provides greater robustness under topology-dependent attacks, while simpler representations remain competitive in overall performance. Among successful runs, the number of episodes required to reach the control remains approximately constant as network size increases from 15 to 100 hosts. These results establish learned cyber dynamics as a sample-efficient and scalable basis for autonomous defense, and identify world representation as a central design axis for robustness and scalability.

Primary area:
reinforcement learning
Keywords:
World Models, Cybersecurity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KuCjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents CyberWorld, a DreamerV3-based world model for autonomous cyber defense in the CyberWheel simulator. The defender's view is compiled into an intermediate representation with five binary features per host and a static host adjacency, and this record is rendered into five observation arms (vector, graph via GAT, vector plus graph, text encoded by a frozen Qwen2.5-3B, and a vector plus text arm with a frame reconstruction head). Two cyber-specific changes to the DreamerV3 losses are introduced, a change-weighted state reconstruction loss and a reward loss that balances rare reward-bearing steps against the rest. The policy is trained entirely in imagination.

The evaluation covers four scoreable scripted attackers, networks of 15 to 100 hosts, and three seeds per cell. The primary metric is the number of environment steps until the trailing-10 episode return first exceeds a fixed deploy_then_stop control policy. The authors report crossings after 3.6k to 15.8k steps for the graph arm against 2.3M to 3.1M for CyberWheel's PPO, comparable crossing speed across representations on three attackers, an advantage for the graph arm on the topology-following BFSExfiltration attacker, and a roughly constant number of episodes to reach the control as the network grows, among the runs that succeed.

Critical strengths
  • The paper presents a controlled representation study. All arms share one architecture and the same underlying information, and the paper includes a representation-matched GNN-PPO baseline plus smoothing and retention analyses.

  • The results show better retention than GNN-PPO. With the same encoder and budget, GNN-PPO collapses to a late median return of −3,700 on the ServerDowntime attackers with 66 to 237 decoy-limit penalties per 100 episodes, while the world-model arms incur far fewer (Table 3). This is the strongest evidence that learning the dynamics helps.

Critical weaknesses
  • The headline claim does not isolate the contributions from the world model. GNN-PPO which uses the same PPO algorithm run inside the CyberWorld loop reaches the control in 4.0k to 25.9k steps and in 92% of runs, matching the world-model arms (Table 2, Figure 8). The orders-of-magnitude gap to PPO therefore reflects the default PPO configuration, yet the abstract and contributions present it as the main result.

  • There is little evidence for stable and scalable defense. After crossing, the vector and graph arms beat the simple control in only 21% to 51% of episodes. The retention is measured on training rollouts for CyberWorld and on checkpoint evaluations for PPO, and key cells rest on three seeds or on one crossing seed out of three. The adversary-shift motivation is never tested, and the Theorem 1 constant (about 1.1e9 here) makes the bound vacuous.

Questions for authors
  1. What differs between CyberWheel's PPO and GNN-PPO, and does a PPO tuned for sample efficiency close the gap?
  2. Can you report greedy-policy returns on held-out episodes for all methods, together with the control return per attacker and network size?
  3. How much do the change-weighted and balanced reward losses contribute? An ablation would answer this.
  4. What seeds are the BFSExfiltration at N=15, 50 and 100 run at? Please document this for completion.
Reviewer_BtRFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CyberWorld as a DreamerV3 based world model optimized on cyber defense data from the CyberWheel environment simulator. The method differs from base DreamerV3 by adding a few cyber specific loss terms along with domain specific state encoders. The authors primarily analyze the data efficiency of training models with this approach, noting that their method often converges faster than model-free approaches.

Critical strengths
  • The paper proposes the first world model for cyber environments to the best of my knowledge.

  • The cross-modal fusion is a nice idea for this domain.

  • The new loss terms are well motivated for the domain.

  • Overall results showing faster convergence is convincing.

Critical weaknesses

Method and Motivation

Limited Novelty The method is essentially just an application of DreamerV3 to a new environment. While it is useful to try and see how world models work in other domains, this does greatly limit the novelty of the paper.

Misaligned Motivation and Method The method also doesn't really solve most of the issues raised in the intro and abstract. Throughout the paper the authors note that it is important to decouple red agent behavior from environment mechanics, but then the method they implement doesn't provide the decoupling explained in Equation 1, instead the world model Predicts the combined Tc\mathcal{T}_c. This seems strange and counter to the paper's motivation, the world model necessarily mixes red agent actions with environment mechanics. The authors have some analysis in the appendix showing that there are some bounds on the mechanism prediction error, but the point is that their method cannot adapt to new adversaries as designed as there is no way for the trained world model to model different adversaries during training or deployment with planning based approaches. The method is necessarily tied to the red agent policies it is trained with.

It would make a lot more sense if the world model accepted both red and blue agent actions, being able to model both action spaces. This would follow directly from the paper's motivation, potentially solve the proposed problem, and likely require a novel solution. My critique here moves more into "future work" territory, but I just want to be clear that the method does not seem to solve the very problems the paper proposes.

Similarly, the authors imply in Figure 1 that their method is better at adapting to new attack patterns, however the final deployed system is still just a policy, the world model is not used at deployment time. Therefore there is no way for the blue agent to utilize the world model to better adapt to new attack patterns. It feels like the paper's motivation is generally in the right place, but the actual method doesn't solve any of the problems excluding the empirical evidence showing faster policy convergence.

Soundness in Metrics

The main evaluation metric used throughout the paper is something akin to convergence speed, specifically how many time steps it takes for the different methods to match the performance of a given "control" baseline. The problem is that it's unclear if this metric is measured properly. Specifically, looking at Table 1 it appears that GNN-PPO converges the quickest, achieving a score near the control in <15k steps across all setups for N<100, however in Figure 3 it appears that GNN-PPO has a significantly lower score than all the other methods by quite a margin. It seems that the primary score in Table 1 was measured as the "first crossing", which appears to be the first time the policy crosses the control threshold on average over the last 50 episodes. The problem with this approach is that it mixes both policy learning variance and environment stochasticity into a single metric. I think it would make a lot more sense to instead measure the performance of each frozen policy after a certain number of learning steps (e.g. eval across 50-100 seeds every 1k env steps). This should greatly reduce variance in the plots and results at the expense of some compute. I know the authors try to justify their approach in the appendix, but I don't find it convincing.

Results

The main results in Table 1 seem to indicate that GNNs are the most important component here, as PPO + GNN performs similarly to the world model based approach, excluding the findings from Table 3 as discussed above. If the authors can clarify the variance in their metrics it may help to distinguish their approach from the GNN-PPO baseline.

Also, as discussed earlier, a primary point of motivation in the paper is that it is important to adapt to new red agent policies (Figure 1), however the authors never evaluate against out of distribution red agents. This would do something in helping support the claim that the model learns "general mechanics" and doesn't just memorize specific red agent policies.

Questions for authors
  • How do your learned policies perform against out-of-distribution attackers?

  • What other improvements does this approach provide over model-free methods aside from improved sample efficiency?

Reviewer_ZBGXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CyberWorld, a Dreamer-style world model for autonomous cyber defense. It learns latent network dynamics and trains defense policies through imagined trajectories, while comparing vector, graph, text, and multimodal representations. Experiments in CyberWheel evaluate sample efficiency, representation choice, and scalability across multiple attacker strategies and network sizes.

Critical strengths

1.The paper addresses the sample-efficiency problem in autonomous cyber defense and introduces Dreamer-style world modeling into this setting. The overall CyberWorld architecture is described in Section 3 and Figure 2, where latent cyber dynamics are learned and defensive policies are optimized through imagined trajectories. Section 4.2 (RQ1), particularly Figure 3 and Table 1, further evaluates interaction efficiency against model-free PPO and the representation-matched GNN-PPO baseline. This direction has clear practical relevance given the high cost of extensive environment interaction in cyber defense. 2.The paper provides a relatively systematic study of cyber-world representations. Section 3.2 defines the vector, graph, text, and multimodal observation representations, while Section 4.1 specifies five representation arms under a shared training configuration. Section 4.2 (RQ2) and Figures 4–5 analyze their behavior across different attacker strategies, and RQ3, Figure 6, Figure 7, and Table 1 further examine vector and graph representations as the network scales from 15 to 100 hosts. These experiments provide useful empirical insight into representation choices, particularly the role of explicit graph structure under topology-dependent attacks.

Critical weaknesses

1.The evidence for some of the headline experimental claims is not fully controlled. First, the sample-efficiency comparison does not use a fully consistent evaluation protocol: in Section 4.1, PPO’s first crossing is computed from 30-episode checkpoint evaluations, whereas CyberWorld uses online training returns. Meanwhile, the more closely matched GNN-PPO baseline operates at a similar interaction scale to several CyberWorld variants in terms of first crossing. Second, the representation comparison in Section 3.2 changes both representation and information content, since the graph arm additionally receives explicit network topology (G=(V,E)), making it difficult for RQ2 to separate the effect of graph representation from the benefit of additional topology information. Finally, the paper introduces two cyber-specific losses in Eq. (9) and Eq. (10) without corresponding ablations, so their individual contributions to the final performance remain unclear. 2.The paper also has noticeable readability issues in presentation and notation. In Figure 1, some text and icons are crowded or overlapping; Figure 2 is overly dense; and Figures 6 and 7 contain overlaps among numerical annotations, legends, annotations, and curves. The main text and appendix also contain repeated presentations of crossing behavior, learning curves, and scaling results. In addition, the notation is not fully consistent: for example, (s_t) denotes the true POMDP environment state in Section 3.1 but is reused for the RSSM learned latent state in Section 3.2 and Figure 2, while (N) is used for both the number of IR nodes and the number of physical hosts. These issues increase the cognitive burden on the reader.

Questions for authors

1.Regarding sample efficiency, could the authors clarify why PPO uses 30-episode checkpoint evaluations while CyberWorld uses online training returns to define the first crossing? Would the main sample-efficiency conclusion still hold under a fully consistent evaluation protocol? 2.In the representation comparison, the graph arm explicitly receives network topology while the vector arm does not. How do the authors disentangle the effect of the graph/GNN representation itself from the benefit of the additional topology information? 3.Eq. (9) and Eq. (10) are presented as cyber-specific modifications. Do the authors have ablation results showing the individual contributions of these two losses to sample efficiency and post-crossing stability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.