acceptodds
Under review as a conference paper at ICLR 2027

A Unified Multi-task Benchmark for Data-driven Models of Power Systems

Abstract

Can a learned representation of a structured physical system transfer across tasks, topologies, scales, and interventions? Despite growing interest in foundation models for scientific and engineering systems, existing power-grid datasets largely isolate individual tasks and operating regimes, making such questions difficult to study. We introduce a unified multi-task benchmark for power systems whose tasks are genuinely different learning problems: constrained regression (AC and distribution optimal power flow), mixed-integer structured prediction (security-constrained unit commitment), rare-event screening ( contingency analysis), inverse problems under partial observability (transmission and distribution state estimation), and Monte-Carlo risk estimation (reliability assessment), spanning transmission and distribution networks ranging from 3 to 13659 buses. Alongside the data, we specify benchmark results on cross-task and topology transfer. Surprisingly, stock GNNs, cross-task pretraining, and publicly released foundation models frequently fail to outperform simple classical or nearest-neighbor baselines, revealing substantial headroom for models that truly generalize across structured physical systems. We release the dataset, generalization splits, and evaluation harness as an open benchmark for graph learning, scientific foundation models, and structured prediction.

Primary area:
datasets and benchmarks
Keywords:
grid foundation models, power system dataset, synthetic data generation, AC optimal power flow, security constrained unit commitment, machine learning, benchmark dataset.

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_LSZoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper made one dataset for seven power system task on the same grid, ACOPF, SCUC, SE and N-k contingency on PGLib transmission grids, then DSSE and distribution OPF on feeders and one reliability dataset. Its stored in one graph format so reader can load any task like an intermediate representation. For ACOPF they scaled the entire load range so as to cover more operating points, in comparison to OPFData dataset. They had tested two public foundation models on it and also train standard GNN and the main result they said is that these models mostly lose to simple baselines.

Critical strengths

The idea of making one big dataset consisting of both transmission and distribution doing multiple tasks is actually something that does not exist and is needed so liked the idea. The ACOPF data is more varied covering more range of operating points in than OPFData and table 3 and figure 4 shows that properly , also probelm in the DOPF labels is also hoenslty stated.

Critical weaknesses

I find benchmarks of the paper are not properly designed.

-- The GNN models seem to be not trained enough. As per the authors’ own specification they used a tiny 3 layer model with 64 numbers per node and trained it for only 25 minutes(pg 27). Also, they have not mentioned what their cost-gap means, that is, how much more expensive the model’s dispatch is than the true optimum, whether it is a fraction, a percent or something else. Whatever it is, a value of 38.72 (table5) means the dispatch is far more expensive than optimal, so the model is clearly not trained enough. Also there are a lot of values that are exactly 1.00(table6) or −1.000(table 5), so is the metric capped at 1? If so, the real errors could be even bigger and we cannot see them. The authors show no training curves and try no bigger models, yet on p.7 they say this is a capacity floor and not under-training.

-- In Fig 2 the foundation models are compared with a KNN that was built using training samples from the same grid, while the foundation models have never seen that grid. table 8 does the fair comparison, a KNN without that grid, and there the foundation model wins on every grid by a big margin (5 to 96 times, page 8), so the claim in th abstract is not supported by their own Table 8 . Also, the two biggest grids where the foundation models looked worst in Figure 2 are missing from the fair table.

-- State estimation problem contains noisy, incomplete measurements and must estimate the true voltages. But here the model is given exact P and Q at every bus with no noise or missing measurements, how is it a complete SE dataset without all variations in noise, gaussian non-gaussian, bad data injection etc?. With information given in paper, voltages follow directly from solving the power flow equations, so it is a power flow task, not estimation. The proper baseline for SE is weighted least squares (WLS), the standard estimator every utility uses, which they mention but never run.

-- ACOPF is compared to DC-OPF (a real solver). SE, DSSE and DOPF are compared to “predict the training average” (not a method, just a constant). Beating a constant proves little. For ACOPF, the average already explains 95% of the output, so the constant predictor and a simple linear (ridge) regression should be shown; the GNN probably loses to both. ACOPF is also only scored on cost, a model that under-generates looks btter but is physically wrong, so they need a violation score too. The only place a GNN beats the baseline is SCUC SMALL (+0.10), but the spread there is ±0.82, much bigger than the gain itself, so this gain is doesn't appear to be reliable.

-- The authors do not show that their more varied data actually trains a better model (train on theirs, test on OPFData). They list four generalisation splits every submission must report, but report only but report only one of them(topology, scale, regime shift, N-k perturbation), only topology transfer is reported (Table 7).

Questions for authors

-- Why pick a 3 layer GNN which can only pass information 3 buses away, so on a 13,659 bus grid most of the grid is invisible to each node. See CANOS made on OPFdata and how does your model size compare against that?

-- Where are contingency and reliability task results located? Also “N-k” means k element out of the system, but authors have only removed one element, so how fair is to say and label it as N-k?

-- In Table 5, why are the baseline and KNN scores exactly the same in every tier for DOPF, DSSE and SE, and why do the GNN scores hardly change? It is mentioed that each tier uses different grids (Table 9), eg. SE has 1 grid in NANO and 14 in SMALL, and that each tier is tested on its own grids. So the training and test data are different in each tier, and the scores should be different too. Please explain.

-- ρ measures how much generator outputs vary between samples. But the denominator compares all generators to one global average, so a 1000 MW generator and a 10 MW generator look “very different” just because of size, not because they vary. Normalizing each generator by its capacity (pg/pmax) is the standard way actually. Also, authors say their data is as good as or better than PGLearn, but in Table 3 PGLearn is higher on case118 on all three diversity numbers. Please explain.

2 comments

Thank you for taking the time to read the paper. After reviewing your comments and those of the other reviewers, we found an evaluation bug in the production of Table 5 that was caused by incorrect stopping criteria in our tests truncating the selection of evaluation data. This is an issue in the evaluation harness that's included in the code supplement, and it will be corrected. The dataset itself is not impacted, so we will include a corrected table in revision.

With regards to the model training, models were trained per the protocol described on page 7, which includes early-stopping criteria that may have impacted model performance. In revision, we will re-run the reference comparisons with an expanded training budget to verify whether this is the case and correct the corresponding table if so. As for the model size choice, this was arbitrary and could also have impacted the comparison, we can expand the reference model comparison to include existing public training recipes in order to address this.

For the notes on comparison metrics, beating a constant is primarily relevant in that many of the reference models fail to beat a constant predictor, but you're right that a better reference comparison should use comparable choices of baselines across tasks. As you suggest, we will also add violation metrics beyond the cost metrics so that issues related to the feasibility of model output are also tracked.

On the note regarding the foundation model negative results, you're right that the models are compared to kNN on training samples, but 10/50 grids are part of the training set for LUMINA and results on both in-distribution and out-of-distribution grids are given in Table 8. The primary setting in which FMs win outright is on transfer tasks to unseen topologies, within a fixed topology a lookup still generally wins. We will relax the language of the claims accordingly.

For your comment on normalization issues in Table 3, the denominator pooling is primarily an issue when comparing the same metrics across cases, within an individual case the generator fleet is the same so the comparison is not entirely unfair, although we will adopt your suggestion on using a more standard normalization.

As far as whether the data trains a better model, that's a fair question, but training a multi-task model without having a public reference multi-task model for comparison covering the full breadth of the tasks would be out of scope for a dataset paper. The primary contribution of this work is the data corpus, but we do agree that the reference results need to be held to a higher standard, and we will address each of the issues you raise in revision.

Reviewer_LSZoOpenReview ↗

Reply to Authors Comments

I thank authors for the response. However, I do not see any new results or updates in the paper. In view of this, I cannot assess if revisions promised will be helpful or not. Therefore, I maintain my earlier assessment.

Reviewer_bBvBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark that covers multiple tasks for power grid modeling across the transmission and distribution domains. For this, it introduces a unified schema with a deterministic projection to typed heterogeneous graphs, a synthetic corpus comprising operating-envelope coverage, standardized splits/tiered tracks, and an evaluation harness with task-appropriate baselines and normalized skill scores. Experimental results show that kNN baselines outperform GNNs and even foundation models under different regimes, highlighting the importance of models that generalize across tasks, scales, and topologies.

Critical strengths
  • A broad task coverage, including shared transmission networks, distribution feeders, and a reliability corpus, unlike other benchmarks that focus on a specific power grid task.
  • The proposed strategy includes multiple baselines per task for power grids, including classical methods and topology-aware/agnostic models.
  • An experimental analysis of foundational models such as LUMINA and GridSFM on the proposed benchmark, revealing some difficulties these models have in handling the tasks correctly.
Critical weaknesses
  • The analysis only considered frozen models, which raises the question of performance drop due to mismatches between the pretraining distribution and model limitations.
  • Lack of an ablation study on label budgets, adapters, and multi-task training strategies to further understand the cross-transfer task results.
  • The proposed benchmark does not integrate other well-known benchmarks for specialized power grid tasks.
Questions for authors
  • How do the results change under the option of a stronger ML-based SCUC?
  • What would the results be under standardized fine-tuning to assess the adaptation of foundational models under the broader evaluation regime?
2 comments

We thank the reviewer for their comments.

As for why the paper only considered frozen models, the tested foundation models are released as general topology independent foundation models, which are claimed to work without a need for retraining or finetuning. Several topologies we test on are also included in one of the model's training datasets and results on four of those in-distribution topologies are included in Table 8.

With regards to ablations on label budget, the transfer runs already cover different label budgets and frozen vs fine-tuned encoders. Fine-tuning did not systematically beat a frozen encoder. A full study of multi-task training strategies is beyond the scope of a dataset paper, but we can provide additional detail on the label budgets used for fine-tuning in revision.

On your question regarding integration with other datasets, the set of chosen topologies overlaps with PGLib/OPFData, PGLearn, and SMART-DS, so results can be compared directly to other existing benchmarks and Table 3 does compare against OPFData and PGLearn on the same cases. If there are specific benchmarks you have in mind, we are happy to include comparisons against these in revision.

To address your question about a stronger model for ML-based SCUC, we know of no publicly released pretrained SCUC model at this time that we could compare against, but would be happy to include such a comparison in revision if you have one in mind.

​​I appreciate the authors' answers. Nevertheless, they don’t address my concerns, and I've seen no updates on the work. Therefore, I maintain my score.

Reviewer_Gc7ZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces a unified multi-task benchmark and evaluation framework for power grid representation learning. It aims to address a gap in existing datasets lacking multi-task, shared-representation corpora over identical network topologies. This work provides a unified schema which casts diverse power system problems into a heterogeneous graph representation.

Critical strengths

This work considers seven different tasks on the 30 power grids. It can enable researchers in the field to test if an AI model can learn one task and use that knowledge for another tasks.

Critical weaknesses

A group of graph learning and foundation models are tested in this work but it turns out they often fail to outperform simple classical or kNN baselines. Would any of the reported SOTA in the literature perform reasonably well which could help further improve the credit of the work?

It seems that it's not assessed about the realism of the synthetic data compared to real-world utility cases. For the DOPF task, 70-100% of the correct answers (labels) violate the physical voltage limits [0.95, 1.05], which is a bit worrying.

Questions for authors

In Table 5, the simple kNN lookup beats the classical baselines on several tasks. Is there any reason or explanation for the results?

The 70-100% of the AC-validated states violate the [0.95, 1.05] voltage band, which are fed into AI models, likely leading to infeasible states?

Since the corpus is synthetic, is there any understanding of possible performance drop if a model trained on this dataset and then deployed on real-world utility data (even a small-scale validation)?

For the cross-topology transfer split, the node counts and graph structures change completely between train and test grids. How does the unified heterogeneous graph schema handle the alignment of node features and the change in input dimensions?

2 comments

Thank you for your time and for your consideration.

To address your question on whether a SOTA model would do better, we evaluated the only two foundation models with public weights that were available at the time of writing. CANOS and most other published ACOPF models do not release weights, although additional models have been released since the submission deadline and we can include these in a revision. Table 8 shows the foundation models do generalize across topologies better than a lookup, but within distribution a dense lookup is very hard to beat. Per the responses to other reviewers, we have also corrected several issues in our reference model training and will revise the reference table to reflect that.

As for the realism of synthetic data, we agree this is a significant limitation and state as much in the motivation for the generation of this corpus. Public utility operating data at this resolution and breadth of tasks is not currently publicly available for comparison.

In our comparisons, kNN performs favorably to classical baselines for two reasons, firstly as you address in your comments some of the selected baselines are weak comparisons, secondly the sampling strategy we use densely samples the operating envelope, so a nearest neighbor usually has neighbors very close to the query point.

On the question of feasibility, the data stores solver convergence and voltage violations separately, so users can use convergence as a proxy for filtering feasible samples or otherwise use those flags as supervisory input to estimate feasibilty. We acknowledge that solver convergence or non-convergence is not proof of AC-feasibility on its own.

With regards to the gap between synthetic and real data, we cannot quantify this without public utility datasets of much greater breadth than currently exist, a gap that partly motivated the generation of this dataset.

Regarding feature alignment, each node and edge type have a fixed feature schema, so features are aligned across nodes and edges instead of across graphs. Graphs are represented as sets of variable size, and message-passing models are designed to handle different node counts natively.

Thank the author(s) for the response. The graph-schema explanation is helpful, however, the major concerns remain regarding the evaluation and results.

If zero-shot foundation-model evaluation is restricted, but it does not prevent training other published SOTA architectures. Also, on the data realism, even some partial validation would be useful, for example statistical comparison, if utility operating data are restricted. The explanation on kNN is useful but more evolutions are needed to provide more robust findings, such as on out-of-distribution and cross-topology splits.

Reviewer_BeMkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a corpus of evaluation of machine learning models for power system tasks. The benchmark includes 5 tasks (AC and distribution optimal power flow (ACOPF, DOPF), security-constrained unit commitment (SCUC), N-k contingency screening, and transmission and distribution state estimation (SE, DSSE), Monte-Carlo reliability assessment (RA)) over four case-size tiers scale (nano, small, medium, full) ranging from 3 to 13659 buses. In addition to the actual solved dataset, the paper presents some empirical results on simple non optimized baselines, and on 2 power Grid foundation models.

Critical strengths
  • There are not many unified benchmarks of multiple tasks for power grids systems, but all "benchmarks" the paper references that are peer-reviewed are not just dataset with brute forced baselines, they all brought insights for the community (PowerGraph, PF∆); the ones with just solved grids were not accepted in peer reviewed venues (OPFDataset), and for a reason.

  • Extensiveness of the grid sizes, and diversity of tasks and distributions.

Critical weaknesses

1- AI generated paper. The paper clearly uses wording and expression from AI agents that are not normal for a human (use of "contracts") like "Four case-size tiers scale compute at a fixed contract". The whole writing, protocol and study does not require any human inputs, and just brute force solving a lot of cases and training existing architectures.

2- Most of the results are negative results, and they do not reflect findings, but rather may reflect training or evaluation failure. For instance Table 5 gives best-of-bank cost gaps of 0.91 (NANO), 38.7 (SMALL), 0.63 (MEDIUM) and 123.1 (FULL), against DC-OPF baselines of 0.21 to 1.19. Table 7 reports 715.5 for case8387. Table 6 reports exactly 1.00 for random init and for several pretrained encoders, which looks like a clipped ceiling. A cost gap of 3,872% or 71,550% is not a model that “trails” a baseline. It is a model that has not learned.

Another case is with models that are worse than, or indistinguishable from, the mean predictor. On DSSE the best GNN has per-unit RMSE 0.053 against 0.039 for the mean predictor, and this is identical across all four tiers.

3- Misleading or incomplete claims: No results for N-k contingency or RA. Table 4 defines metrics and classical baselines for both, but Table 5 contains neither. Two of the seven advertised tasks therefore have no reported baseline performance in the paper. The N-k labels are still “being regenerated (v2)” (Table 9 footnote), and a “label-fidelity audit” is mentioned (D.3) but not reported. Not all of the other tasks are evaluated across tiers. SCUC has no NANO or FULL results, and SE has no FULL results. The FULL tier is said to need “leadership scale computing”, yet no such run is reported beyond ACOPF.

The paper was submitted while the AI agent was still running, and it is clearly stated.

Questions for authors

None. This is an AI slop paper.

2 comments

Thank you for taking your time to review this.

With regards to your comments on AI use, there's no question of it, in fact we disclose in the appendix explicitly that AI was used for drafting and copyediting. However, the corpus, sampling strategy, problem formulations, solver harness, and data schemas are all human designed. We refer you to the ICLR AI usage policy, which permits this.

On your point regarding negative results, yes, you are correct. These issues were due to an evaluation harness bug that erroneously truncated evaluation data and an early stopping condition that triggered too early in reference model training. These are issues with the code supplement and reference models, but do not themselves impact the generated dataset. We thank you for helping us identify these problems, and we will correct the included code supplement and revise the corresponding reference tables.

We agree that in general a dataset paper should bring insight beyond just solved instances, and to the extent that we do it is only that existing public ACOPF datasets have comparatively narrow ranges of operating conditions or active constraint sets, which is a concern for model training intended to generalize to unseen operating conditions but is not a novel observation. One consideration is that the general unavailability of existing multi-task datasets like this is a direct impediment to the kind of broader analysis we would all like to see in this space.

Reviewer_BeMkOpenReview ↗

Thaank you for your answer, but I am not discussingh papers where the AI content was not checked and the paper should be desk rejected

  1. ICLR AI usage policy permits the usage of AI for support, not for fully automated papers and rebuttals
  2. You did not address any of my actionnable and precise concerns. You agree with the fact the experiments are incomplete and no insights.

I maintain therefore my score

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.