- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents a new framework for finding high-impact contingency scenarios of a power grid using a genetic algorithm that is coupled with a surrogate model that approximates a physics based solver of the grid. The surrogate scores each candidate attack so the search avoids calling the solver every time. The attack space covers both line cuts and bounded load changes under a shared budget. The paper demonstrates advantages in two settings. Against static solvers the method identifies more contingency violations than the comparison methods. Against the dynamic L2RPN agents it replaces the challenge’s own opponent and drops defender survival well below what that opponent achieves, including against the 2023 challenge winner.
Critical strengths
This paper tackles a challenging problem in power grid operations of finding contingencies that threaten the stability of the grid. This matters because prior opponents in the L2RPN setup do not actively search for a defender’s weak points, so the agents were only tested against weak adversaries, and this work shows that gap clearly. The search design is reasonable for the combinatorial problem. The genetic algorithm handles a mixed discrete–continuous action space, line cuts and load changes together under one budget, and using a learned model as the fitness lets the search avoid a solver call per candidate. They also test in two settings and report an exhaustive upper bound so the gap to the ceiling is visible.
Critical weaknesses
• The RQ1 success metric is too weak. The paper measures success as the attacked state has more violations than the clean state, however there are instances in the experiments where one bus is barely over its limit but still counts the same as a collapse. This metric does not quantify severity of the contingency as a result, does not predict real impact of these outages. • The post-contingency state in RQ1 is set up in an unrealistic way. All non-slack generators are frozen at the surrogate’s predicted dispatch and the whole imbalance is put on the slack bus. Real post-contingency flow spreads the pickup across units through droop and participation factors. • RQ1 uses one set of limits with no emergency band. Voltage and thermal violations are checked against a single static bound. In operations there are separate normal and emergency limits, and overload is time dependent. A voltage excursion that is fine under emergency criteria gets counted here as a successful attack.
• Once concern is the accuracy of the surrogate itself. The search is guided entirely by the surrogate’s predicted violations, so if the surrogate is wrong, the attacks it selects are chosen on a flawed signal. This is reported in the manuscript as generator RPA sits around 0.74 and the authors admit it cannot respect tight reactive limits. In addition, the fitness has no thermal term at all even though line overloads are the real failure mode in RQ2. The accuracy also drops on the larger grid. This raises a deeper question about the evaluation. If the surrogate carries error, how can the success of PEGAS be judged against it, when the same model both selects the attack and underlies how impact is assessed? The paper never bounds the surrogate’s error or shows that it is small enough for the reported success rates to be trusted.
• There is no comparison to sensitivity based contingency screening (LODF/PTDF ranking), which is what operators actually use to find worst case outages without a solve per candidate. Since the results are driven by bridge cuts, a one shot LODF ranking might find the same lines much faster than a 500 generation GA. • Due to the way success is defined, the severity of the contingencies that were missed are never considered. The 6.8% in RQ1 and the 5 to 6% in RQ2 that survive are not analyzed. There is nothing on whether the missed cases are the critical ones, or whether the failed searches run longer because of the early stopping. • The improvement in runtime is only shown at 118 buses. The 6.34 s per attack is small, but it holds partly because the load genes (63 s on their own) are barely used at k=1. The 2000 bus case has no search timing and no dynamic run, so the “near real time, grid pace” claim is not backed at a realistic grid size.
Questions for authors
• In D.6, 613 of 727 collapses end on a line 26 disconnection, and line 26 is a bridge in the grid while RQ2 turns off the connectivity check. This suggests most of the result comes down to finding the one line whose removal islands the grid. Why is the full surrogate-plus-genetic-algorithm method needed to find this, when a simple graph check for bridges or cut vertices would identify line 26 directly at a fraction of the cost? Can you show how PEGAS performs against such a baseline, and how much of the reported success remains once the bridge cuts are set aside?
• Can you quantify the surrogate’s error on the quantity that actually drives the attack, i.e. how often its predicted violations match the power-flow ground truth? Given that generator RPA is ~0.74 and there is no thermal term, how do you know the search is being guided toward truly damaging actions rather than surrogate artifacts? • Since the surrogate both selects the attack and (in RQ1) underlies how the state is evaluated, how do you rule out that part of the measured success is the surrogate’s own error rather than real grid impact?
• Your success criterion only checks whether the number of violations increased with no indicator of severity. How many of your reported RQ1 successes correspond to large violation and can you report overload magnitude, voltage deviation size, or load shed?
7 comments
Realism of our study
R3W2: The post-contingency state in RQ1 is set up in an unrealistic way. Droop and participation factors are not considered.
RQ1 does not consider any corrective measure to the grid, or grids with "defense" mechanisms. We are following a static setting of "single slack", contrary to the "distributed slack" setting that allows participation factors.
Droop and participation factors are dynamic mechanisms that require considering primary frequency responses, which is out of scope for RQ1.
It is worth noting that the Grid2Op environment of the challenge in RQ2 solves a steady-state AC power flow at each 5-minute step. There is no frequency state and no governor dynamics, so droop is not modeled. RQ1 uses the same model and the same hypotheses as the challenge.
We will clarify the threat model of RQ1 and clearly scope our study to the "static single slack" model and explain the motivation of our setting (to match the target environment of RQ2).
R3W3: In operations there are separate normal and emergency limits, and overload is time dependent. A voltage excursion that is fine under emergency criteria gets counted here as a successful attack.
In RQ1, we are using only one threshold following well established power grid standards and referentials (see R3W1). Considering two thresholds is only relevant in multi-step time-dependent settings (as you state in the weakness), which is the setting of RQ2. RQ1 only considers independent single step snapshots.
In RQ2, the RL environment uses a multi-step simulator that considers this time dependency. The grid collapses if a line exceeds emergency limits immediately, or normal limits (limits used in RQ1) for multiple consecutive steps (stochastically).
Therefore, our single-threshold limit is perfectly suitable for the setting of RQ1. We will further explain this design choice in the threat model.
Evaluation & baselines
R3W4: One concern is the accuracy of the surrogate itself. If the surrogate carries error, how can the success of PEGAS be judged against it, when the same model both selects the attack and underlies how impact is assessed?
We would like to clarify that the success rate is never evaluated using the surrogate. An independent solver scores the success metric. In RQ1, we use a power flow solver implemented within PowerModels library, and in RQ2, the Grid2Op environment has its own solver to check the survival or collapse of the grid.
R3W5: There is no comparison to sensitivity based contingency screening (LODF/PTDF ranking)
Thank you for suggesting this non-AI baseline. Indeed, we only compared ourselves in RQ1 to Machine Learning based attacks. We have implemented this baseline for 118-bus and 2000-bus grids and our results show that within our threat model (same knowledge as PEGAS), this baseline yields 0% success rate both in the N scenario (in distribution) and the N-1 scenario (out of distribution):
| Grid | Method | Success N (%) | Success N-1 (%) |
|---|---|---|---|
| case 118 | PEGAS | 93.2 ± 3.0 | 86.3 ± 5.4 |
| case 118 | LODF | 0.0 ± 0.0 | 0.0 ± 0.0 |
| case 2000 | PEGAS | 71.6 ± 5.9 | 66.8 ± 6.1 |
| case 2000 | LODF | 0.0 ± 0.0 | 0.0 ± 0.0 |
We will include these new baselines in the appendix.
Severity and impact of the outages
We thank the reviewer for highlighting the importance of the problem and the significance of the gap we are addressing. We address below the weaknesses and questions you raised (grouped when overlapping).
R3W1/Q3: The success rate metric in RQ1 does not quantify severity of the contingency as a result, does not predict real impact of these outages.
Thank you for raising this discussion. Indeed, we are not distinguising between contingencies by severity in RQ1. We are using the common threshold in power systems literature: 1.05 PU.
The threshold we consider to rate the attack as successful is used by multiple official power grid referentials and standards for Transmission Grids Operations as the limit for safe operations: ENTSO-E parameters document[1], ERCOT voltage limits maintenance[2], SIMONA model documentation[3].
While we agree that lines can survive a few steps over 1.05 without grid collapse, the phenomenon is stochastic, and we are strictly following the safety threshold in RQ1. In RQ2, with the agentic simulator, the success criteria is the collapse of the grid.