- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper made one dataset for seven power system task on the same grid, ACOPF, SCUC, SE and N-k contingency on PGLib transmission grids, then DSSE and distribution OPF on feeders and one reliability dataset. Its stored in one graph format so reader can load any task like an intermediate representation. For ACOPF they scaled the entire load range so as to cover more operating points, in comparison to OPFData dataset. They had tested two public foundation models on it and also train standard GNN and the main result they said is that these models mostly lose to simple baselines.
Critical strengths
The idea of making one big dataset consisting of both transmission and distribution doing multiple tasks is actually something that does not exist and is needed so liked the idea. The ACOPF data is more varied covering more range of operating points in than OPFData and table 3 and figure 4 shows that properly , also probelm in the DOPF labels is also hoenslty stated.
Critical weaknesses
I find benchmarks of the paper are not properly designed.
-- The GNN models seem to be not trained enough. As per the authors’ own specification they used a tiny 3 layer model with 64 numbers per node and trained it for only 25 minutes(pg 27). Also, they have not mentioned what their cost-gap means, that is, how much more expensive the model’s dispatch is than the true optimum, whether it is a fraction, a percent or something else. Whatever it is, a value of 38.72 (table5) means the dispatch is far more expensive than optimal, so the model is clearly not trained enough. Also there are a lot of values that are exactly 1.00(table6) or −1.000(table 5), so is the metric capped at 1? If so, the real errors could be even bigger and we cannot see them. The authors show no training curves and try no bigger models, yet on p.7 they say this is a capacity floor and not under-training.
-- In Fig 2 the foundation models are compared with a KNN that was built using training samples from the same grid, while the foundation models have never seen that grid. table 8 does the fair comparison, a KNN without that grid, and there the foundation model wins on every grid by a big margin (5 to 96 times, page 8), so the claim in th abstract is not supported by their own Table 8 . Also, the two biggest grids where the foundation models looked worst in Figure 2 are missing from the fair table.
-- State estimation problem contains noisy, incomplete measurements and must estimate the true voltages. But here the model is given exact P and Q at every bus with no noise or missing measurements, how is it a complete SE dataset without all variations in noise, gaussian non-gaussian, bad data injection etc?. With information given in paper, voltages follow directly from solving the power flow equations, so it is a power flow task, not estimation. The proper baseline for SE is weighted least squares (WLS), the standard estimator every utility uses, which they mention but never run.
-- ACOPF is compared to DC-OPF (a real solver). SE, DSSE and DOPF are compared to “predict the training average” (not a method, just a constant). Beating a constant proves little. For ACOPF, the average already explains 95% of the output, so the constant predictor and a simple linear (ridge) regression should be shown; the GNN probably loses to both. ACOPF is also only scored on cost, a model that under-generates looks btter but is physically wrong, so they need a violation score too. The only place a GNN beats the baseline is SCUC SMALL (+0.10), but the spread there is ±0.82, much bigger than the gain itself, so this gain is doesn't appear to be reliable.
-- The authors do not show that their more varied data actually trains a better model (train on theirs, test on OPFData). They list four generalisation splits every submission must report, but report only but report only one of them(topology, scale, regime shift, N-k perturbation), only topology transfer is reported (Table 7).
Questions for authors
-- Why pick a 3 layer GNN which can only pass information 3 buses away, so on a 13,659 bus grid most of the grid is invisible to each node. See CANOS made on OPFdata and how does your model size compare against that?
-- Where are contingency and reliability task results located? Also “N-k” means k element out of the system, but authors have only removed one element, so how fair is to say and label it as N-k?
-- In Table 5, why are the baseline and KNN scores exactly the same in every tier for DOPF, DSSE and SE, and why do the GNN scores hardly change? It is mentioed that each tier uses different grids (Table 9), eg. SE has 1 grid in NANO and 14 in SMALL, and that each tier is tested on its own grids. So the training and test data are different in each tier, and the scores should be different too. Please explain.
-- ρ measures how much generator outputs vary between samples. But the denominator compares all generators to one global average, so a 1000 MW generator and a 10 MW generator look “very different” just because of size, not because they vary. Normalizing each generator by its capacity (pg/pmax) is the standard way actually. Also, authors say their data is as good as or better than PGLearn, but in Table 3 PGLearn is higher on case118 on all three diversity numbers. Please explain.
2 comments
Reply to Authors Comments
I thank authors for the response. However, I do not see any new results or updates in the paper. In view of this, I cannot assess if revisions promised will be helpful or not. Therefore, I maintain my earlier assessment.
Thank you for taking the time to read the paper. After reviewing your comments and those of the other reviewers, we found an evaluation bug in the production of Table 5 that was caused by incorrect stopping criteria in our tests truncating the selection of evaluation data. This is an issue in the evaluation harness that's included in the code supplement, and it will be corrected. The dataset itself is not impacted, so we will include a corrected table in revision.
With regards to the model training, models were trained per the protocol described on page 7, which includes early-stopping criteria that may have impacted model performance. In revision, we will re-run the reference comparisons with an expanded training budget to verify whether this is the case and correct the corresponding table if so. As for the model size choice, this was arbitrary and could also have impacted the comparison, we can expand the reference model comparison to include existing public training recipes in order to address this.
For the notes on comparison metrics, beating a constant is primarily relevant in that many of the reference models fail to beat a constant predictor, but you're right that a better reference comparison should use comparable choices of baselines across tasks. As you suggest, we will also add violation metrics beyond the cost metrics so that issues related to the feasibility of model output are also tracked.
On the note regarding the foundation model negative results, you're right that the models are compared to kNN on training samples, but 10/50 grids are part of the training set for LUMINA and results on both in-distribution and out-of-distribution grids are given in Table 8. The primary setting in which FMs win outright is on transfer tasks to unseen topologies, within a fixed topology a lookup still generally wins. We will relax the language of the claims accordingly.
For your comment on normalization issues in Table 3, the denominator pooling is primarily an issue when comparing the same metrics across cases, within an individual case the generator fleet is the same so the comparison is not entirely unfair, although we will adopt your suggestion on using a more standard normalization.
As far as whether the data trains a better model, that's a fair question, but training a multi-task model without having a public reference multi-task model for comparison covering the full breadth of the tasks would be out of scope for a dataset paper. The primary contribution of this work is the data corpus, but we do agree that the reference results need to be held to a higher standard, and we will address each of the issues you raise in revision.