- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an automated incident-response framework for industrial enterprise networks. An LLM generates candidate multi-step response plans and assigns prior preferences. An ensemble of probabilistic dynamics models rolls out these plans to estimate their returns and predictive uncertainty. A PPO-trained policy selects among the plans using incident information and these estimates, executes the first action, and replans at the next step. The second claimed contribution is a reward function jointly penalizing recovery delay and operational failures.
Critical strengths
-
The optimization objective addresses an important operational trade-off.
-
The reported experiments provide preliminary evidence of practical benefit.
Critical weaknesses
-
It is unclear how much the proposed decision mechanism adds. LLM-based plan generation, dynamics prediction, and PPO are all existing methods. The main question is whether learning to combine LLM preferences, predicted returns, and uncertainty leads to better decisions. The ablations mostly remove whole modules. They do not compare PPO with simple rules, such as choosing the highest predicted return or subtracting an uncertainty penalty, using the same plans and predictions. This makes it hard to tell whether the learned selector is needed and how much uncertainty helps.
-
The experiments do not fully support the generalization claims, and some results need clarification. Good performance across four network zones shows that the method works in those settings. However, the paper does not clearly explain which attacks and network configurations were unseen during training, or whether the models were updated during testing. The full method’s reward is also −426.97 in Tables 2 and 5, but −446.97 in Tables 4 and 6. The standard deviations for operational-failure penalties differ as well. These unexplained differences make the results harder to trust.
Questions for authors
- With the same candidate plans and world model, how much better is PPO than simply choosing the plan with the highest predicted return, or subtracting an uncertainty penalty from that return? What happens if uncertainty is removed from PPO’s inputs?
- How accurate is the world model? Do plans with higher predicted returns actually perform better? Does disagreement between the models reflect larger prediction errors?
- How long does each decision take, including LLM generation and model rollouts? Is the method still faster overall when this time is included?
We sincerely thank the reviewer for your constructive feedback. We have addressed all comments, and the revised portions in the revision are highlighted in RED.
Q.1 PPO Decision-Making Effectiveness. We have compared two simple plan-choice strategies: 1) Max-Return, which chooses the plan with the highest predicted return, and 2) Return-U, which subtracts an uncertainty penalty from the return. Our PPO outperforms both strategies. Max-Return and Return-U allocate 0.11% and 0.09% of actions to remove and restore, respectively, leading to conservative responses that fail to eliminate malicious activities adequately. PPO allocates 13.55% and 14.58% of actions to remove and restore, respectively, improving malicious activity elimination and rewards. Details are in Section 4.2 and Appendix A of the revision.
We have added an ablation without uncertainty in Section 4.5 of the revision. As shown in the table below, removing uncertainty from PPO's inputs degrades its performance.
Q.2 World Model Effectiveness. To evaluate the world model's accuracy, we have added an ablation with and without predicted returns in Section 4.5 of the revision. The Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) between predicted returns and Monte Carlo returns are 5.252 and 6.727 with predicted returns, versus 8.247 and 10.685 without them, indicating the world model's return-prediction accuracy.
In Section 4.2 of the revision, we evaluated the proportion of cases where the plan with the highest predicted return was also the actual optimal plan, achieving a top-1 hit rate of 94.59%. This indicates the world model's effectiveness in guiding plan selection.
We have evaluated whether model disagreement reflects prediction errors in Section 4.2 of the revision. For each candidate plan, the average Spearman's ρ between model disagreement and prediction errors across predicted states is 0.552. This indicates the world model's uncertainty calibration capability.
Q.3 Overall Time Efficiency. The average LLM generation and model rollout times are 3.925s and 2.540ms, respectively. Furthermore, the overall latencies of our method and UAMCTS are 3.931s and 5.071s, respectively. Our method responds to incidents faster than selected existing LLM-based MBRL. Details are in Section 4.4 of the revision.
Q.4 Generalization Evaluation. We have added held-out generalization experiments in Section 4.2 of the revision. We train our method and three baselines (PriorRL, WM-RL, and UAMCTS) in CC4 using the attack strategy FiniteStateRedAgent, topology T0 (office network router—public access zone router—admin network router), and 3—6 hosts per subnet. Models remain fixed during testing. We evaluate three held-out settings: 1) an unseen attack strategy (DiscoveryFSRed), 2) an unseen modified topology T1 (office network router—admin network router—public access zone router), and 3) an unseen per-subnet host count (7—10). Our method achieves the highest reward across all three held-out settings, indicating its better generalization to unseen attack strategies, topologies, and host counts.
Q.5 Inconsistent Result Clarification. For CC4 Reward, Tables 2 and 5 incorrectly report −426.97, whereas Tables 4 and 6 report −446.97. The correct result is −446.97, as confirmed by the original experimental records and the result files in our GitHub repository. For the standard deviation of Operational Failure Penalty, Table 2 reports 80.73, Table 4 and 6 report 30.73, and Table 5 reports 40.73. The correct value is 80.73, as confirmed by the original experimental records and the result files in our GitHub repository. We corrected all four tables in the revision.
Note that result variations across experiments for the same method stem from stochastic PPO decisions. Despite identical seeds and initial conditions, different PPO trajectories can vary host states, triggered events, and action durations.