acceptodds
Under review as a conference paper at ICLR 2027

Learning Cyber Defense with Limited Interaction: An Empirical Study of Reflective Prompt Optimization

Abstract

Cyber defense resembles a game of Whack-A-Mole: restoring a compromised host removes attacker access, but incurs an immediate cost and may provide only temporary protection. Successful remediation can be verified, but whether it was worth the cost depends on subsequent events. We investigate whether reflective prompt optimization can learn effective defense policies with fewer environment interactions than proximal policy optimization (PPO) and the critic-free group relative policy optimization (GRPO). We also identify classes of rewards for which it works well and where it falls short. Using CybORG CAGE-2, a simulated network-defense environment, we modify the original objective and design when-to-whack scenarios emphasizing verified remediation, sustained defense, and a hybrid that strikes a balance between the two. We also design which-to-whack scenarios, in which the defender chooses which host to restore under distinct or uniform damage costs for pairs of compromised hosts. Our evaluation compares GEPA, which optimizes a pretrained language-model agent's instructions using return-only or enriched trajectory feedback, with PPO and GRPO tuned per scenario. On the when-to-whack scenarios, after about one hundred episodes of interaction, every GEPA run exceeds the mean return of PPO and GRPO. Moreover, PPO needs 10–50 more interactions to catch up while GRPO matches GEPA after 200K steps in mixed and sustained object, but fall short after five million steps in verified remediation and original settings. On the which-to-whack scenarios, both PPO and GRPO outperform GEPA. Overall, we find that GEPA is best where reward follows a locally verifiable event or restoration is cheap, where it closes 85–98% of the gap between an untrained PPO and PPO trained for five million steps. We also find that enriched feedback can improve GEPA's performance under verified remediation and the original objective, while return-only feedback suffices under the mixed and sustained objectives. Separate matched-trajectory experiments show that changing feedback while holding the instruction and experience fixed changes the corrective strategies proposed during reflective prompt optimization.

Primary area:
datasets and benchmarks
Keywords:
autonomous cyber defense, prompt optimization, LLM agents, sample efficiency, reward design

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Lw3aOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the effectiveness of reflective policy optimization for learning automated cyber defense strategies in the CAGE-2 environment. In particular, it studies the policies with respect to the “whack a mole” problem created in the CAGE game where the red agent compromises several hosts, and it is unclear to the agent which host(s) should be restored (either due to partial knowledge, or because the priority ordering is unclear). To test this, the authors selected GEPA, a recent reflective prompt optimization method, and used it on DeepSeek-v3 to improve its strategy for CAGE-2 across the different reward configurations and objectives. The results were that GEPA could consistently reach the same scores that the RL-based models achieved by episode 500k in far fewer steps. They also show that when models are trained for only 10k steps, GEPA outperforms them in all of the “when to whack” tests (though none of the “which mole” tests).

Critical strengths
  • The experimental protocol is very well documented in the appendices, and appears reproducible.
  • Automated defense is a promising area to apply prompt optimization. The proposed experiments were well thought out and show that this approach has promise for certain aspects of the problem
  • The enhanced feedback experiments were especially interesting, and I think deserve more attention in the paper. Highlight these results more
Critical weaknesses
  • Some portions of the experimental setup were unclear. E.g. “standing cost weight”, and “privileged greedy” were not explained well; “Meander” is presented without explanation in 4.1. Readers unaware of the CybORG environment would be unlikely to infer the explanations from later descriptions.
  • The justification for making hosts pairwise-compromised (sec 3.3) is unclear. Is this meant to just make the game more difficult? More justification for this design choice is needed.
  • It would be informative to see how the agents compare to a heuristic (also zero-shot) model like the React-Restore agent bundled in with the CAGE-2 code. This would be a quick test to run in the new environments and would help contextualize the results in Table 2
  • The conclusion, and really the thesis of the work feels underwhelming. “Reflection works well when a few trajectories support useful instruction changes” reads like reflection works well only for simple environments. It seems like in simple environments with easy to interpret trajectories, we should be using lighter-weight models, rather than billion-parameter LLMs. An argument could be made that the LLMs can write the strategies in plain English, or even generate heuristic agents to execute them on their behalf. Just leaving the conclusion as prompt optimization only works for easy environments feels weak to me unless this were a purely negative results paper.
  • A main concern of mine is that 10k episodes with PPO vs 10k episodes with chains of LLM reasoning are not identical. PPO models are generally very small and the training bottleneck (especially in CybORG) is the speed of the environment rather than the models. With local LLMs, especially if the full 37B DeepSeek model was used, and instructions being given to Claude via API, it’s possible that model processing time could be a bigger concern. It may be that wall clock time is a more fair way to compare these methods than episodes sampled. The conclusion even mentions this, “fewer environment interactions need not imply lower runtime or financial cost”, but there is no further discussion of this very important point.
  • The repeated assertion that “cyber defense is like whack-a-mole” is a bit of an overstatement. CybORG models it that way, but real APTs endeavor to move completely without detection. The threat model of CybORG is not very realistic, which doesn’t detract from this work, but it is worth toning down the claims throughout the paper to the contrary.
  • I don’t know that applying an existing model to a new dataset/environment is a large enough contribution to warrant publication at this venue

Nitpicks:

  • A few sources in Section 2 are cited as their arXiv preprints, but have since been published in conferences (Hammar et al., and King et al.).
Questions for authors
  • RL based approaches have the potential to get very high scores if they train for long periods of time. Do prompt optimization methods share this quality, or is the score they get at 10k episodes close to the ceiling? The tradeoff between early convergence vs larger potential scores with more training is worth discussing
  • How does the baseline “React-Restore” heuristic agent score in the modified environments?
  • What is the runtime comparison to train a PPO/GRPO model for 1 episode vs 1 round of GEPA optimization?
  • What were the un-normalized scores for PPO in the default CAGE-2 environment? Do they match what’s reported on the leaderboard?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Lw3a

Thank you for the detailed review and the concrete requests.

Q1: Setup and pairwise objectives.

Thank you. Indeed, the previous submission relied too heavily on the appendix for key definitions. Section 3 now introduces them before they are used (changes in blue):

  • Environment and attackers. The new opening of Section 3.1 (page 4, lines 174–182) describes the network, defender actions and observations, and stock reward. It also introduces both scripted attackers: B-line follows the shortest path to the operational server. Meander, used in all main comparisons, escalates privileges on every host of a subnet before moving on.

  • Standing cost weight: a per-step charge for every host on which the attacker holds root (lines 178–180). “Rewards and payback” (page 4, lines 191–199, Eq. 1) then defines the weight w as a multiplier on that charge, and states that together with the Restore charge it sets how long a compromise must persist before restoring pays for itself (Section 3.2).

  • Privileged greedy: In the which-to-whack scenarios, scores are reported on a scale set by two reference policies. Zero is the uniform policy, which restores an arbitrary flagged host. One is privileged greedy, which chooses the Restore that is best one step ahead, using the hidden pair-cost table and the true compromised set. It marks the top of the scale but is not a ceiling and not a learner: GRPO scores 1.12 under uniform pair costs. The submission first used the term in the caption of Figure 2 and in Section 4.1. Section 3.3 now defines both references where the scenarios are introduced (page 5, lines 246–249), and Appendix D.5 shows the details.

Q2 Why pairwise costs?

These costs are not meant to make the game harder. They isolate the second decision faced by the defender, namely which host to restore. In the when-to-whack objectives we study the recovery for damage context, whether that damage comes from the host itself or from its joint compromise with other hosts, so the damage a Restore removes is fixed for each host. Pair costs make it depend on which other hosts are compromised. Every host has the same total pair weight, so no host is the right target on its own. Distinct pair costs test target selection with unequal, hidden weights. Uniform pair costs remove that variation and keep only the need to avoid ineffective Restores. Section 3.3 now presents this as a controlled test of target selection. The weights are a design parameter; we do not present them as calibrated operational losses.

AuthorsOpenReview ↗

Response to Reviewer Lw3a

Q3: Heuristic comparison.

We ran the three defenders bundled with CybORG and report them in a new Appendix C (pages 24–25). A new sentence directly above Table 2 points to it (Section 4.2, page 8, lines 382–383). Table 12 gives the three defenders on the stock objective for both attackers at 30, 50, and 100 steps. Table 13 gives them under the four when-to-whack objectives, on the same held-out episodes as the learners in Table 2. Mean returns from Table 13 are below, with the zero-shot prompt from Table 6; the appendix reports variation and the full protocol.

ObjectiveNo-opReact-RemoveReact-RestoreZero-shot promptGEPA 10KPPO 10KGRPO 10K
Verified Remediation0.00.0−92.6−261.136.57.9−0.8
Original CAGE-2−977.3−677.5−226.3−203.8−32.0−335.6−319.1
Sustained Defense−988.4−705.1−386.2−417.7−230.2−391.0−354.5
Mixed−494.2−352.5−239.4−328.7−143.5−215.6−183.5

GEPA at 10K exceeds all three heuristics under all four objectives. The zero-shot prompt does not: it is below React-Restore under Verified Remediation and Mixed and level with it under the other two, so the margin over the heuristic is acquired during optimization. Neither PPO nor GRPO exceeds all three heuristics under every objective at 10K: React-Restore is above both under Original and level with PPO under Sustained Defense, and under Verified Remediation the two defenders that never restore are level with GRPO. This is the small-budget regime the paper studies; PPO at five million steps is far above every heuristic.

We present these as fixed reference policies. They are not heuristics tuned to each objective. In the two which-to-whack scenarios the defender already restores one flagged host per step, so React-Restore’s rule reduces to restoring an arbitrary flagged host. That is the count policy (Appendix D.2 and D.5), the zero of the which-to-whack scores in Table 2.

Q4: Is the takeaway for prompt optimization more than “works in easy environments”?

Our contribution is to study the interplay between prompt optimization and actions like Restore, which is a direct but costly way to remove cyber risk. We asess its value through two attributes: its payback horizon, how long avoided damage must accrue to repay the net Restore cost, and its damage context, whether that damage comes from the host itself or from its joint compromise with other hosts. Each attribute defines one recovery problem, when to Restore and which host to Restore, and six objectives instantiate them on one simulator (Table 1). We measure which learner is ahead on each problem at 10K interactions:

  • The damage context decides which learner is ahead, at every payback horizon (Table 2). GEPA scores 0.58–0.98 against 0.19–0.52 for PPO and GRPO with damage per host, and 0.40 and 0.10 against 0.75–1.12 with damage on pairs.
  • Both kinds of learner cut wasted Restores. Only GEPA also learns to target them or act early, where the payback horizon is short (Figure 3; Table 2, last column).
  • Feedback that sets the damage beside the Restore’s cost raises the reflector’s corrective proposals from 14% to 72% (Figure 4).

This analysis is not a split by difficulty: GEPA trails on the smallest problems in the paper, ten-step episodes on eight hosts (Section 3.3). Section 5 now states the conclusion in these terms.

Reviewer_tJPkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper evaluates and compares three algorithms (PPO, GRPO, and GEPA) in the CAGE cyber network defense environment. The results find that GEPA usually learns faster than PPO and GRPO when the environment is fully observable, but performs worse under partial observability.

Critical strengths
  • The which-to-whack vs when-to-wack comparison is interesting, it is useful to have these two problems unambiguously evaluated.
Critical weaknesses

Highly Limited Contributions

The paper essentially just evaluates three existing algorithms in an existing RL environment. While it is nice to have results like this, I don't think this kind of study is sufficient for a major AI conference submission. It would have been more meaningful to study and improve upon weaknesses with GEPA, perhaps showing that the weaknesses also show up in non-cyber environments as well, but as it stands this paper is just comparing 3 algorithms.

Comparing GEPA, using an LLM, to PPO and GRPO, using small neural networks, is also highly questionable. The paper doesn't seem to question if the comparison is fair.

Writing

The paper's writing is unclear and not up to the standard of ICLR. The central "whack-a-mole" framing is unsupported, lacking any relevant citations, and is taken as a given. The wasted space on page 5 and 9 also indicates a lack of care.

Questions for authors
  • Do you anticipate these relationships (differences between PPO, GRPO, and GEPA) to persist across other non-cyber domains? What about other cyber tasks?
1 comment
AuthorsOpenReview ↗

Response to Reviewer tJPk

Q: Do you anticipate these relationships (differences between PPO, GRPO, and GEPA) to persist across other non-cyber domains? What about other cyber tasks?

  • Other cyber tasks. To provide additional diversity to the experimental setup, we rebuilt the four objectives on a second simulator Yawning Titan, with a different network model, action set and attacker (new Appendix H). GEPA is ahead of PPO at 10K interactions under all four scenarios (Table 28). PPO catches up after about 500K–1.5M interactions under three and not within 5M under the fourth.

  • Non-cyber domains. Our paper is scoped to cyber defense, a growing topic in machine learning research and a concern for society as a whole.

Reviewer_yt29OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The main idea of the paper is to empirically study when reflective prompt optimization such as GEPA learns cyber-defense policies with fewer environment interactions than reinforcement learning such as PPO and GRPO.

The author extend the CAGE-2 simulation to include six simulated defense settings that help them cover restoration timing and target selection. These allow them to analyze performance differences, and feedback-design experiments.

They show that at 10,000 interactions, GEPA outperforms both RL baselines in the four timing settings, while both RL methods outperform GEPA in the two targeting settings.

Overall, they find that task-dependent such as GEPA is more sample efficient within their simulation settings. There is no real-world defense evaluation.

Critical strengths

The experiements are carefully designed and analyzed.

They identify both advantages and failure cases. For example, at the same 10K interaction budget, GEPA leads in four restoration-timing settings, while PPO and GRPO lead in two targeting settings.

The feedback study goes beyond aggregate performance with matched experiments, separate optimization runs, etc.

Critical weaknesses

Limited realism and external validity. The study is purely simulation based. The attackers and defenders behaviors are far from realistic. They use a 13-host simulator with scripted attackers, predefined action effects, synthetic monitoring signals, and manually assigned costs. There is no validation on real telemetry or an operational network.

It could be ok if there is major innovation in terms of methodology. There is very limited conceptual novelty. The paper applies the existing GEPA optimizer unchanged to the established CAGE-2 environment. Its contribution is primarily an empirical comparison, modified reward settings, and feedback-design experiments, rather than a new learning algorithm.

Questions for authors
  • What evidence supports the relevance of the chosen reward costs, monitoring assumptions, and restoration dynamics to operational cyber defense?

  • Which conclusions remain justified without validation on real telemetry or an emulated network?

  • Beyond applying the existing GEPA optimizer to CAGE-2, what is the principal new, generalizable insight?

1 comment
AuthorsOpenReview ↗

Response to Reviewer yt29

Q1. What supports the chosen costs and dynamics?

We adopt the dynamics of CAGE-2, a widely used single-agent environment, and (newly added) Yawning Titan, an established graph-based environment.

The costs are designed to reflect two dimensions of the Restore action: its payback horizon and the damage context in which it is taken. The base values and host tiers come directly from CAGE-2. We introduce five additional scenarios that vary the Restore charge, the bonus, and how damage is charged (Table 1). A Restore incurs an upfront charge and breaks even when the damage it prevents, together with any bonus, offsets that charge, as formalized by the payback calculation in Section 3.2 (Eq. 2).

Q2. Which conclusions remain justified without validation on real telemetry or an emulated network?

We expect the relative sample efficiency conclusion to carry over to an emulated network: under the per-host damage objective, GEPA outperformed PPO and GRPO at limited interaction budgets in the two simulators evaluated. Because interactions are more expensive in an emulated range, we expect GEPA’s advantage under constrained interaction budgets to persist.

However, we distinguish this expectation from the transferability of trained policies. We cannot claim that a trained PPO policy would transfer directly to an emulated network: PPO had to be retrained from scratch on the second simulator. Finally, our experiments do not establish effectiveness on real telemetry, which may differ substantially from the simulated environments in the execution of actions.

Q3. Beyond applying GEPA to CAGE-2, what is the principal new, generalizable insight?

The insight of this work is that a defender’s objective in autonomous cyber defense has two dimensions that can improve the defense reward, and which one is better depends on what the objective is. Two dimensions are as follows: the payback horizon is how many steps of avoided damage cover the cost of a Restore. The damage context is whether the damage a Restore removes is the host’s own (per-host) or shared with other compromised hosts (joint). We further design six scenarios on two simulators to evaluate this. The takeaway of these case studies is that RL and prompt optimization agents perform differently in two dimensions and the result is consistent across two simulators. Any defense task with a remediation has these two properties can be generalized.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Paper Contributions

The Restore action is a direct but costly way to remove cyber risk. We study its value through two dimensions: 1) payback horizon, how long avoided damage must accrue to repay the net Restore cost. 2) damage context, whether that damage comes from the host itself or from its joint compromise with other hosts. Each attribute defines one recovery problem:

  • When to Restore (“when to whack”): Is intervention worth its cost in the time remaining? We consider four objectives use host-level damage and vary the effective payback horizon.

  • Which host to Restore (“which to whack”): When several hosts are compromised, which intervention removes the most damage? We consider objectives based on pairwise damage, so the answer depends on the other compromised hosts and the weights of their connections.

We compare GEPA, PPO, and GRPO under all six objectives at 10,000 interactions. In CAGE-2, GEPA leads under all four timing objectives, while PPO and GRPO lead under both target-selection objectives. Hence, the contribution is this controlled account of how the structure of a Restore decision changes what learners acquire from a limited interaction budget.

Additional Environment

To provide additional diversity to our evaluation, we also developed scenarios for a second simulator Yawning Titan. These scenarios use a different network model, action set and attacker (Appendix H), and so provide additional evidence in support of our main findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.