acceptodds
Under review as a conference paper at ICLR 2027

Adaptable and Timely Incident Response for Industrial Enterprise Networks Using LLM-based Predictive Reinforcement Learning

Abstract

Adaptable and timely incident response is critical for industrial enterprise networks to recover from diverse cyberattacks. However, existing works have two limitations: 1) performance degradation when facing diverse attacks or deployed in different network environments; and 2) insufficient timeliness and effectiveness without jointly considering recovery delays and operational failures. Therefore, we develop an incident response method for industrial enterprise networks using large language model (LLM)-based predictive reinforcement learning, consisting of three parts: prior candidate-plan generation using LLMs, lookahead prediction using world-model rollouts, and posterior decision-making using proximal policy optimization (PPO). We propose selecting a plan by jointly considering its LLM-derived prior preference and the return and uncertainty from world-model rollouts. This approach captures regularities among prior preference, return, and uncertainty across incident scenarios, improving generalization to diverse attacks and network environments. Furthermore, we propose a reward function that penalizes recovery delays and operational failures to guide plan-selection policy learning, improving response timeliness and effectiveness. Evaluations show that our method outperforms selected SOTAs under diverse attacks and across different network environments, and our reward function achieves higher recovery precision, shorter recovery delay, and fewer operational failures than the reward functions used in selected SOTAs.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Incident Response Planning, Model-based Reinforcement Learning, LLM Priors, Industrial Enterprise Networks

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_1SAXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an automated incident-response framework for industrial enterprise networks. An LLM generates candidate multi-step response plans and assigns prior preferences. An ensemble of probabilistic dynamics models rolls out these plans to estimate their returns and predictive uncertainty. A PPO-trained policy selects among the plans using incident information and these estimates, executes the first action, and replans at the next step. The second claimed contribution is a reward function jointly penalizing recovery delay and operational failures.

Critical strengths
  1. The optimization objective addresses an important operational trade-off.

  2. The reported experiments provide preliminary evidence of practical benefit.

Critical weaknesses
  1. It is unclear how much the proposed decision mechanism adds. LLM-based plan generation, dynamics prediction, and PPO are all existing methods. The main question is whether learning to combine LLM preferences, predicted returns, and uncertainty leads to better decisions. The ablations mostly remove whole modules. They do not compare PPO with simple rules, such as choosing the highest predicted return or subtracting an uncertainty penalty, using the same plans and predictions. This makes it hard to tell whether the learned selector is needed and how much uncertainty helps.

  2. The experiments do not fully support the generalization claims, and some results need clarification. Good performance across four network zones shows that the method works in those settings. However, the paper does not clearly explain which attacks and network configurations were unseen during training, or whether the models were updated during testing. The full method’s reward is also −426.97 in Tables 2 and 5, but −446.97 in Tables 4 and 6. The standard deviations for operational-failure penalties differ as well. These unexplained differences make the results harder to trust.

Questions for authors
  1. With the same candidate plans and world model, how much better is PPO than simply choosing the plan with the highest predicted return, or subtracting an uncertainty penalty from that return? What happens if uncertainty is removed from PPO’s inputs?
  2. How accurate is the world model? Do plans with higher predicted returns actually perform better? Does disagreement between the models reflect larger prediction errors?
  3. How long does each decision take, including LLM generation and model rollouts? Is the method still faster overall when this time is included?
1 comment

We sincerely thank the reviewer for your constructive feedback. We have addressed all comments, and the revised portions in the revision are highlighted in RED.

Q.1 PPO Decision-Making Effectiveness. We have compared two simple plan-choice strategies: 1) Max-Return, which chooses the plan with the highest predicted return, and 2) Return-U, which subtracts an uncertainty penalty from the return. Our PPO outperforms both strategies. Max-Return and Return-U allocate 0.11% and 0.09% of actions to remove and restore, respectively, leading to conservative responses that fail to eliminate malicious activities adequately. PPO allocates 13.55% and 14.58% of actions to remove and restore, respectively, improving malicious activity elimination and rewards. Details are in Section 4.2 and Appendix A of the revision.

MethodCC4 Official RewardOperational Failure PenaltyRecovery PrecisionRecovery Time
PPO−349.79 ± 68.9079.94 ± 53.1089.75 ± 3.0854.27 ± 3.57
Max-Return−6908.08 ± 1339.865791.46 ± 1279.1266.67 ± 43.8999.49 ± 0.95
Return-U−6977.58 ± 1276.165833.29 ± 1197.1263.25 ± 39.6099.21 ± 1.02

We have added an ablation without uncertainty in Section 4.5 of the revision. As shown in the table below, removing uncertainty from PPO's inputs degrades its performance.

MethodCC4 Official RewardOperational Failure PenaltyRecovery PrecisionRecovery Time
Full method−395.64 ± 93.7691.43 ± 70.8478.95 ± 4.8381.12 ± 11.00
Without uncertainty−403.13 ± 142.56111.13 ± 115.8977.44 ± 4.6384.85 ± 13.15

Q.2 World Model Effectiveness. To evaluate the world model's accuracy, we have added an ablation with and without predicted returns in Section 4.5 of the revision. The Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) between predicted returns and Monte Carlo returns are 5.252 and 6.727 with predicted returns, versus 8.247 and 10.685 without them, indicating the world model's return-prediction accuracy.

In Section 4.2 of the revision, we evaluated the proportion of cases where the plan with the highest predicted return was also the actual optimal plan, achieving a top-1 hit rate of 94.59%. This indicates the world model's effectiveness in guiding plan selection.

We have evaluated whether model disagreement reflects prediction errors in Section 4.2 of the revision. For each candidate plan, the average Spearman's ρ between model disagreement and prediction errors across predicted states is 0.552. This indicates the world model's uncertainty calibration capability.

Q.3 Overall Time Efficiency. The average LLM generation and model rollout times are 3.925s and 2.540ms, respectively. Furthermore, the overall latencies of our method and UAMCTS are 3.931s and 5.071s, respectively. Our method responds to incidents faster than selected existing LLM-based MBRL. Details are in Section 4.4 of the revision.

Q.4 Generalization Evaluation. We have added held-out generalization experiments in Section 4.2 of the revision. We train our method and three baselines (PriorRL, WM-RL, and UAMCTS) in CC4 using the attack strategy FiniteStateRedAgent, topology T0 (office network router—public access zone router—admin network router), and 3—6 hosts per subnet. Models remain fixed during testing. We evaluate three held-out settings: 1) an unseen attack strategy (DiscoveryFSRed), 2) an unseen modified topology T1 (office network router—admin network router—public access zone router), and 3) an unseen per-subnet host count (7—10). Our method achieves the highest reward across all three held-out settings, indicating its better generalization to unseen attack strategies, topologies, and host counts.

SettingOur MethodPriorRLWM-RLUAMCTS
Unseen attack strategy−198.50±33.23−263.04±49.02−237.96±47.62−255.33±63.22
Unseen topology−252.50±14.85−274.54±31.96−327.08±83.48−311.33±33.29
Unseen subnet host counts−281.00±16.97−322.33±36.53−371.83±131.86−471.67±226.33

Q.5 Inconsistent Result Clarification. For CC4 Reward, Tables 2 and 5 incorrectly report −426.97, whereas Tables 4 and 6 report −446.97. The correct result is −446.97, as confirmed by the original experimental records and the result files in our GitHub repository. For the standard deviation of Operational Failure Penalty, Table 2 reports 80.73, Table 4 and 6 report 30.73, and Table 5 reports 40.73. The correct value is 80.73, as confirmed by the original experimental records and the result files in our GitHub repository. We corrected all four tables in the revision.

Note that result variations across experiments for the same method stem from stochastic PPO decisions. Despite identical seeds and initial conditions, different PPO trajectories can vary host states, triggered events, and action durations.

Reviewer_FyuPOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an adaptable and timely incident response method using LLM-based predictive reinforcement learning (RL).

Critical strengths

The paper addresses a very important problem and proposes a nice solution.

Critical weaknesses

I just do not understand the novelty of this work. The method consists of three parts:

  1. prior plan generation. The large language model (LLM) generates a set of candidate multi-step plans and assigns a prior preference to each plan;

  2. lookahead prediction. The world model rolls out each plan to predict its return and quantify the predictive uncertainty; and

  3. posterior decision-making. Based on incident information and each plan’s prior preference, return, and uncertainty, proximal policy optimization (PPO) optimizes a plan-selection policy to minimize recovery delays and operational failures.

This approach seems very close to the MMST paradigm used in adaptive control (at a very high level), and close to many other paradigms used across engineering.

Questions for authors

What is the novelty of this work precisely. It seems very close to many similar ideas used in other areas. Is the novelty the use of the LLM and world models?

1 comment

We sincerely thank the reviewer for the careful review and constructive comments. We apologize for not articulating the novelty of this work. The novelty is not the use of the LLM and world models, but a PPO-based decision-making mechanism that jointly leverages three complementary observations: 1) the LLM’s prior preferences provide additional decision basis when world-model predictions are unreliable in unseen environments; 2) the world model’s predicted returns indicate plan effectiveness under different environments; and 3) the world model’s predictive uncertainties indicate prediction errors in unseen environments. PPO learns to leverage these observations jointly to make decisions, improving generalization when facing varying attacks or deployment in different network environments.

Our method differs from the MMST paradigm and existing MBRLs in the following ways:

  1. The MMST paradigm. It uses multiple candidate models to predict future states of the same controlled system, and selects the model with the lowest prediction error for control. However, a low prediction error indicates that a model fits recent system responses well. This alone does not guarantee that the model will achieve a high return under a new attack strategy or network configuration. In contrast, our method jointly considers the LLM’s prior preferences, the world model’s predicted returns, and predictive uncertainty to select candidate plans through PPO, improving generalization to varying attack strategies and network environments.

  2. MBRLs with training-time planning. They use a model to simulate trajectories and learn a policy from them, but performance degrades under training-test distribution shifts. In contrast, our method uses decision-time planning.

  3. MBRLs with return-maximizing decision-time planning. They use a model (such as a world model and an LLM) to roll out candidate plans and execute the action with the highest return estimated from the rollouts, but suffer from inaccurate predictions in changing environments. In contrast, our method incorporates predictive uncertainty into decision-making, mitigating performance degradation caused by inaccurate return predictions.

  4. MBRLs using uncertainty penalties. They subtract an uncertainty penalty from the highest predicted return, leading to conservative actions at the expense of performance. In contrast, our method feeds the LLM-derived prior preference, along with the return and uncertainty from world-model rollouts, into PPO as observations, allowing PPO to make posterior decisions. Our method selects high-return plans despite high model uncertainty when beneficial, improving decision-making under uncertainty.

Another novelty of this work is a reward function that penalizes both recovery delays and operational failures, enabling timely and effective recovery from attacks while improving operational availability. Our reward function differs from existing ones in two ways: 1) some existing reward functions penalize compromised components, operational failures, or both. This may delay effective response, increasing the risk of attack propagation and subsequent availability loss; and 2) other reward functions penalize recovery delays, which may encourage overly aggressive recovery actions and compromise operational availability.

We have improved the descriptions of our novelty and distinctions from existing work in the Introduction and Related Work sections. The revised portions in the revision are highlighted in RED. We greatly appreciate your insights and the opportunity to improve our work.

Comments on the paper

1 comment

We sincerely thank the reviewer for your constructive feedback. We have addressed all comments and the revised portions in the revision are highlighted in RED.

Q.1 Held-Out Generalization Evaluation. In the original submission, the test attack strategies and network configurations were seen during training. We have added held-out generalization experiments with matched baselines in Section 4.2 of the revision. We train our method and three baselines (PriorRL, WM-RL, and UAMCTS) in CC4 using the attack strategy FiniteStateRedAgent, topology T0 (office network router—public access zone router—admin network router), and 3—6 hosts per subnet. We test three held-out settings: 1) an unseen attack strategy (DiscoveryFSRed), 2) an unseen modified topology T1 (office network router—admin network router—public access zone router), and 3) an unseen per-subnet host count (7—10). Our method achieves the highest CC4 Official Reward across all three held-out settings, demonstrating its better generalization to unseen attack strategies, topologies, and host counts.

SettingOur MethodPriorRLWM-RLUAMCTS
Unseen attack strategy−198.50±33.23−263.04±49.02−237.96±47.62−255.33±63.22
Unseen topology−252.50±14.85−274.54±31.96−327.08±83.48−311.33±33.29
Unseen subnet host counts−281.00±16.97−322.33±36.53−371.83±131.86−471.67±226.33

Q.2 Equation and Result Clarifications. We apologize for the errors in the manuscript and have made corrections:

  1. Indexing in Eq. (4). The index should start from 0 (instead of 1) to include the immediate reward. We have corrected this in Eq. (4).

  2. Optimization direction in Eq. (9). The objective should be to maximize the reward. We have corrected this in Eq. (9).

  3. Inconsistent results across Tables 2, 4, 5, and 6 in the original submission. For CC4 Reward, Tables 2 and 5 incorrectly report −426.97, whereas Tables 4 and 6 report −446.97. The correct result is −446.97, as confirmed by the original experimental records and the result files in our GitHub repository. For the standard deviation of Operational Failure Penalty, Table 2 reports 80.73, Tables 4 and 6 report 30.73, and Table 5 reports 40.73. The correct value is 80.73, as confirmed by the original experimental records and the result files in our GitHub repository. We corrected all four tables in the revision.

Q.3 Predictive Uncertainty Contribution. We have added an ablation without uncertainty in Section 4.5 of the revision and a simple fixed-weight plan-selection baseline in Section 4.2 of the revision. The latter selects candidate plans based on a weighted sum of prior preference, predicted return, and predictive uncertainty, with weights of 1, 0.5, and 1, respectively. These results show the uncertainty's independent contribution to performance.

MethodCC4 Official RewardOperational Failure PenaltyRecovery PrecisionRecovery Time
Full method−395.64 ± 93.7691.43 ± 70.8478.95 ± 4.8381.12 ± 11.00
Without uncertainty−403.13 ± 142.56111.13 ± 115.8977.44 ± 4.6384.85 ± 13.15
Fixed-weight plan selection−452.00±130.33180.67±139.0032.97±2.0666.41±25.89

Q.4 LLM-based Replanning Latency and Cost. We have evaluated the inference latency and token cost of LLM-based replanning in Section 4.4 of the revision. LLM-based replanning has an average inference latency of 3.925s, with 720 input tokens and 299 output tokens. Our method and UAMCTS have overall latencies of 3.931s and 5.071s, respectively. Our method responds to incidents faster than the selected existing LLM-based MBRL.

Q.5 Statistical Significance Analysis. We have conducted statistical significance tests to compare our method with the baselines in terms of effectiveness (Table 2 in the revision) and generalization (Table in Q.1). Our method significantly outperforms most baselines in both. While its effectiveness advantage over PriorRL is not significant, its generalization advantage is. We report details in Section 4.2 of the revision.

  1. Statistical Significance Tests for Effectiveness | Metric | TERLA | PriorRL | CARL | DCA | UAMCTS | RSMBRL | |---|---:|---:|---:|---:|---:|---:| | CC4 Official Reward | 4.09×10^-27 | 0.2118 | 1.06×10^-13 | 1.23×10^-12 | 5.19×10^-15 | 1.36×10^-24 |

  2. Statistical Significance Tests for Generalization | Setting | WM-RL | FixedFusion | PriorRL | UAMCTS | |---|---:|---:|---:|---:| | Unseen attack strategy | 0.000485 | 0.002605 | 1.397×10^-6 | 0.0002055 | | Unseen topology | 0.0002202 | 2.173×10^-5 | 0.002589 | 1.078×10^-8 | | Unseen subnet host counts | 0.002614 | 0.0001311 | 1.224×10^-5 | 0.0004101 |

Note that result variations across experiments for the same method stem from stochastic PPO decisions. Despite identical seeds and initial conditions, different PPO trajectories can vary host states, triggered events, and action durations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.