- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the effectiveness of reflective policy optimization for learning automated cyber defense strategies in the CAGE-2 environment. In particular, it studies the policies with respect to the “whack a mole” problem created in the CAGE game where the red agent compromises several hosts, and it is unclear to the agent which host(s) should be restored (either due to partial knowledge, or because the priority ordering is unclear). To test this, the authors selected GEPA, a recent reflective prompt optimization method, and used it on DeepSeek-v3 to improve its strategy for CAGE-2 across the different reward configurations and objectives. The results were that GEPA could consistently reach the same scores that the RL-based models achieved by episode 500k in far fewer steps. They also show that when models are trained for only 10k steps, GEPA outperforms them in all of the “when to whack” tests (though none of the “which mole” tests).
Critical strengths
- The experimental protocol is very well documented in the appendices, and appears reproducible.
- Automated defense is a promising area to apply prompt optimization. The proposed experiments were well thought out and show that this approach has promise for certain aspects of the problem
- The enhanced feedback experiments were especially interesting, and I think deserve more attention in the paper. Highlight these results more
Critical weaknesses
- Some portions of the experimental setup were unclear. E.g. “standing cost weight”, and “privileged greedy” were not explained well; “Meander” is presented without explanation in 4.1. Readers unaware of the CybORG environment would be unlikely to infer the explanations from later descriptions.
- The justification for making hosts pairwise-compromised (sec 3.3) is unclear. Is this meant to just make the game more difficult? More justification for this design choice is needed.
- It would be informative to see how the agents compare to a heuristic (also zero-shot) model like the React-Restore agent bundled in with the CAGE-2 code. This would be a quick test to run in the new environments and would help contextualize the results in Table 2
- The conclusion, and really the thesis of the work feels underwhelming. “Reflection works well when a few trajectories support useful instruction changes” reads like reflection works well only for simple environments. It seems like in simple environments with easy to interpret trajectories, we should be using lighter-weight models, rather than billion-parameter LLMs. An argument could be made that the LLMs can write the strategies in plain English, or even generate heuristic agents to execute them on their behalf. Just leaving the conclusion as prompt optimization only works for easy environments feels weak to me unless this were a purely negative results paper.
- A main concern of mine is that 10k episodes with PPO vs 10k episodes with chains of LLM reasoning are not identical. PPO models are generally very small and the training bottleneck (especially in CybORG) is the speed of the environment rather than the models. With local LLMs, especially if the full 37B DeepSeek model was used, and instructions being given to Claude via API, it’s possible that model processing time could be a bigger concern. It may be that wall clock time is a more fair way to compare these methods than episodes sampled. The conclusion even mentions this, “fewer environment interactions need not imply lower runtime or financial cost”, but there is no further discussion of this very important point.
- The repeated assertion that “cyber defense is like whack-a-mole” is a bit of an overstatement. CybORG models it that way, but real APTs endeavor to move completely without detection. The threat model of CybORG is not very realistic, which doesn’t detract from this work, but it is worth toning down the claims throughout the paper to the contrary.
- I don’t know that applying an existing model to a new dataset/environment is a large enough contribution to warrant publication at this venue
Nitpicks:
- A few sources in Section 2 are cited as their arXiv preprints, but have since been published in conferences (Hammar et al., and King et al.).
Questions for authors
- RL based approaches have the potential to get very high scores if they train for long periods of time. Do prompt optimization methods share this quality, or is the score they get at 10k episodes close to the ceiling? The tradeoff between early convergence vs larger potential scores with more training is worth discussing
- How does the baseline “React-Restore” heuristic agent score in the modified environments?
- What is the runtime comparison to train a PPO/GRPO model for 1 episode vs 1 round of GEPA optimization?
- What were the un-normalized scores for PPO in the default CAGE-2 environment? Do they match what’s reported on the leaderboard?
2 comments
Response to Reviewer Lw3a
Q3: Heuristic comparison.
We ran the three defenders bundled with CybORG and report them in a new Appendix C (pages 24–25). A new sentence directly above Table 2 points to it (Section 4.2, page 8, lines 382–383). Table 12 gives the three defenders on the stock objective for both attackers at 30, 50, and 100 steps. Table 13 gives them under the four when-to-whack objectives, on the same held-out episodes as the learners in Table 2. Mean returns from Table 13 are below, with the zero-shot prompt from Table 6; the appendix reports variation and the full protocol.
| Objective | No-op | React-Remove | React-Restore | Zero-shot prompt | GEPA 10K | PPO 10K | GRPO 10K |
|---|---|---|---|---|---|---|---|
| Verified Remediation | 0.0 | 0.0 | −92.6 | −261.1 | 36.5 | 7.9 | −0.8 |
| Original CAGE-2 | −977.3 | −677.5 | −226.3 | −203.8 | −32.0 | −335.6 | −319.1 |
| Sustained Defense | −988.4 | −705.1 | −386.2 | −417.7 | −230.2 | −391.0 | −354.5 |
| Mixed | −494.2 | −352.5 | −239.4 | −328.7 | −143.5 | −215.6 | −183.5 |
GEPA at 10K exceeds all three heuristics under all four objectives. The zero-shot prompt does not: it is below React-Restore under Verified Remediation and Mixed and level with it under the other two, so the margin over the heuristic is acquired during optimization. Neither PPO nor GRPO exceeds all three heuristics under every objective at 10K: React-Restore is above both under Original and level with PPO under Sustained Defense, and under Verified Remediation the two defenders that never restore are level with GRPO. This is the small-budget regime the paper studies; PPO at five million steps is far above every heuristic.
We present these as fixed reference policies. They are not heuristics tuned to each objective. In the two which-to-whack scenarios the defender already restores one flagged host per step, so React-Restore’s rule reduces to restoring an arbitrary flagged host. That is the count policy (Appendix D.2 and D.5), the zero of the which-to-whack scores in Table 2.
Q4: Is the takeaway for prompt optimization more than “works in easy environments”?
Our contribution is to study the interplay between prompt optimization and actions like Restore, which is a direct but costly way to remove cyber risk. We asess its value through two attributes: its payback horizon, how long avoided damage must accrue to repay the net Restore cost, and its damage context, whether that damage comes from the host itself or from its joint compromise with other hosts. Each attribute defines one recovery problem, when to Restore and which host to Restore, and six objectives instantiate them on one simulator (Table 1). We measure which learner is ahead on each problem at 10K interactions:
- The damage context decides which learner is ahead, at every payback horizon (Table 2). GEPA scores 0.58–0.98 against 0.19–0.52 for PPO and GRPO with damage per host, and 0.40 and 0.10 against 0.75–1.12 with damage on pairs.
- Both kinds of learner cut wasted Restores. Only GEPA also learns to target them or act early, where the payback horizon is short (Figure 3; Table 2, last column).
- Feedback that sets the damage beside the Restore’s cost raises the reflector’s corrective proposals from 14% to 72% (Figure 4).
This analysis is not a split by difficulty: GEPA trails on the smallest problems in the paper, ten-step episodes on eight hosts (Section 3.3). Section 5 now states the conclusion in these terms.
Response to Reviewer Lw3a
Thank you for the detailed review and the concrete requests.
Q1: Setup and pairwise objectives.
Thank you. Indeed, the previous submission relied too heavily on the appendix for key definitions. Section 3 now introduces them before they are used (changes in blue):
Environment and attackers. The new opening of Section 3.1 (page 4, lines 174–182) describes the network, defender actions and observations, and stock reward. It also introduces both scripted attackers: B-line follows the shortest path to the operational server. Meander, used in all main comparisons, escalates privileges on every host of a subnet before moving on.
Standing cost weight: a per-step charge for every host on which the attacker holds root (lines 178–180). “Rewards and payback” (page 4, lines 191–199, Eq. 1) then defines the weight w as a multiplier on that charge, and states that together with the Restore charge it sets how long a compromise must persist before restoring pays for itself (Section 3.2).
Privileged greedy: In the which-to-whack scenarios, scores are reported on a scale set by two reference policies. Zero is the uniform policy, which restores an arbitrary flagged host. One is privileged greedy, which chooses the Restore that is best one step ahead, using the hidden pair-cost table and the true compromised set. It marks the top of the scale but is not a ceiling and not a learner: GRPO scores 1.12 under uniform pair costs. The submission first used the term in the caption of Figure 2 and in Section 4.1. Section 3.3 now defines both references where the scenarios are introduced (page 5, lines 246–249), and Appendix D.5 shows the details.
Q2 Why pairwise costs?
These costs are not meant to make the game harder. They isolate the second decision faced by the defender, namely which host to restore. In the when-to-whack objectives we study the recovery for damage context, whether that damage comes from the host itself or from its joint compromise with other hosts, so the damage a Restore removes is fixed for each host. Pair costs make it depend on which other hosts are compromised. Every host has the same total pair weight, so no host is the right target on its own. Distinct pair costs test target selection with unequal, hidden weights. Uniform pair costs remove that variation and keep only the need to avoid ineffective Restores. Section 3.3 now presents this as a controlled test of target selection. The weights are a design parameter; we do not present them as calibrated operational losses.