acceptodds
Under review as a conference paper at ICLR 2027

When the Counterfactual Is Adjudicated: A Grid-Curtailment Benchmark Separating Structure from One-Sided Estimation

Abstract

Methods that recover an outcome suppressed by an intervening policy are often evaluated against a simulator written by the evaluator, and their ranking changes with the simulator. On one power grid, the suppressed outcome is adjudicated instead. Whenever the Brazilian system operator restricts a wind or solar plant, it publishes a half-hourly estimate of what the plant would have generated. The estimate is computed from wind-speed or irradiance measurements at the plant, under the procedure used to settle compensation payments. It is not a second meter reading, but the process that produces it predates this work and uses those measurements, which the benchmark withholds from every method. This paper builds a benchmark from the published record. The benchmark covers 18 wind clusters over 40,896 hours and 22 solar clusters over 18,984 hours, and it enforces five invariants as build-time assertions. Its evaluation protocol declares what each method may read, splits the record by date, and scores recovery on the hours the operator restricts separately from false corrections on the hours it does not. At a configuration fixed before any test score is seen, a structured latent decomposition models the censoring in its likelihood and floors its predictions at realised output. On restricted hours it beats a least-squares ratio estimator on solar but loses on wind. A ratio estimator that instead fits the 0.75 quantile of realised output, with calendar inputs and no latent structure, beats the decomposition on censored hours on all 15 seeds of both technologies (on seeds 1 to 5, 0.1237 against 0.1327 on solar and 0.1388 against 0.1713 on wind). On solar it also cuts the error on unrestricted hours from 0.2483 to 0.0198. With or without calendar terms in its inputs, a least-squares fit loses to the decomposition on every solar seed, and a 0.75-quantile fit with the same inputs beats it on at least 14 of 15. For those inputs, the change of loss alone reverses the ordering. The quantile grid is fixed in advance and reported in full, and this arm is its best cell on solar. A validation block inside the training window picks cells that still beat the decomposition on 14 of 15 solar seeds and all 15 wind seeds. Over all clusters, the solar advantage of the decomposition over least squares therefore does not show a benefit from its latent structure. On the solar clusters live throughout the record, the decomposition gains more and keeps a lead of about 0.02 over the quantile fit on restricted hours, so one-sided estimation reproduces only part of that gain. Scoring the unrestricted hours separately is what exposes the false corrections of the decomposition.

Primary area:
datasets and benchmarks
Keywords:
benchmark, counterfactual evaluation, censored data, potential outcomes, quantile regression, latent variable models, distribution shift, evaluation protocol, renewable energy curtailment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_656AOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses counterfactual values in renewable generation data. These are values that the TSO estimates from weather or similar inputs but that are not actually measured, because the real values are curtailed. The authors study how to benchmark models despite these values, distinguishing between censored and uncensored data. To this end, they compare several models that estimate the available generation yavaily^{\mathrm{avail}} from the observed actual generation yacty^{\mathrm{act}}. The main part of the paper is the evaluation across the two data regimes, censored and uncensored of different models.

Critical strengths
  1. The topic is highly relevant for transmission system operators and difficult to tackle.
Critical weaknesses
  1. Overall: there are no structuring sentences that guide the reader through the paper. The paper is hard, if not impossible, to read. Here are a few examples: The abstract is too long and not understandable. It gives neither a clear introduction to the data and the method nor to the results and the contribution. The introduction needs major revision in terms of fluency and clarity. Figure 1 is referenced in Section 5. No clear contributions can be inferred from this introduction. Point out what your major claims are and what your methodological contribution is. The contributions need to be clearer. Less is more in this case: one detailed and sharp sentence, not a lurid AI headline. Related Work: there is no clear narrative that can be followed, and no clear derivation of your novelty claim. At this point I resigned trying to understand the paper in further detail.

  2. The methodology is spread everywhere. The method is described partly in the appendix. The reader has to search the appendix and the main text for each candidate to find out what it means, which makes the paper very hard to follow. A complete, detailed description of every method, fitting strategy, floor strategy and input strategy should be in the main part of the paper.

I recommend defining every compared model in the main text, ideally in one dedicated subsection that Table 3 refers to directly.

Based on my reading, this subsection should cover the following (this is my understanding of your methodology, which could also be wrong):

Models:

  • Structured decomposition: I suggest calling it the structured decomposition estimator, so that it is clear that it is an estimator. Please define it formally. Please state which inputs go in and which outputs come out, with their dimensions.
  • Ratio estimator: I could only infer how it is computed from Figure 1 and the surrounding text. Please define it formally. Please state which inputs go in and which outputs come out, with their dimensions.

Three fitting configurations:

  • quantile,
  • least squares,
  • fitting this model directly on yavaily^{\mathrm{avail}}.

Please describe each of them explicitly and state how they differ.

Inputs:

  • weather (which variables, e.g., wind speed or solar irradiance, and where do they come from?),

  • grid state (please describe it in detail),

  • ycurty^{\mathrm{curt}},

  • yavaily^{\mathrm{avail}}.

    I would like to know the exact size of each of them. What is the size of the input to the model?

It is possible, that this paper and its topic is interesting for energy community. However, it suffers methodological a lot in clearance of description and in narrative, therefore the merit here is a clear reject, as these major problems avoid sufficent evaluation of technical and methodological content.

Questions for authors
  1. Is there a second model besides the GRU system, I understand the ratio method more or less as a benchline?
  2. Related Work: Forecasting/nowcasting? Benchmarking with counterfactuals? Evaluating counterfactuals? Replacing censored data and inferring them?
  3. Related Work , Appendix A: Minor remark: Is the claim search an LLM prompt? This is not the same as a real search of the literature of a scientific community (Appendix A, which also contains typos and arXiv artefacts).
  4. Methodology, Inputs: Which weather variables do you use (e.g., wind speed or solar irradiance), and where do they come from?
  5. Methodology, Inputs: What is the size of the input to the model?
  6. Data split: When does the data end, and how many days does the evaluation span? Is a whole year covered, so that the yearly seasonality is captured?
Reviewer_Fpu6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a benchmark for reconstructing renewable power generation suppressed by grid curtailment, using the Brazilian system operator's estimates of uncurtailed power generation as evaluation targets. The paper also provides an evaluation protocol that scores the prediction of uncurtailed generation against the published value and a test that separates the latent structure from one-sided estimation. A main inference from the test is that the upper-quantile regression reproduces, or sometimes exceeds, gains over the latent structure model. There are also settings where the structured model retains an advantage over the upper-quantile regression.

Critical strengths
  1. Adjudicated counterfactuals from operator estimates provide a useful external operational target against which modeling assumptions can be tested.

  2. The paper is very transparent that their benchmark, although rooted in real-world measurements, is also an estimate and therefore is not the ground truth. The authors provide a substantial amount of sensitivity analysis with additional seeds, regularization sweeps, target perturbations, operator revisions, etc.

Critical weaknesses
  1. I believe this work is very limited in scope and impact for the broader causal learning community. I could not see how this benchmark would be beneficial for future research in causal inference or counterfactual prediction. This work and its conclusions could fit better in a domain-specific venue.

  2. The readability of the paper is very poor. I am not very familiar with the domain of power grids or curtailment. However, heavy use of AI in writing has made the paper very verbose, less rigorous and hard to understand. Terms like "MAE", which I assume is Mean Absolute Error", "deployed arm" etc. are used without explaining (or before explaining) it properly. I think the paper requires substantial rewriting before it can be accepted.

  3. The manuscript in its current form does not rigorously specify the methodology sufficiently. A figure of the architecture is provided, but details on the exact likelihood, training objective, etc. are absent.

Questions for authors

Please address the weaknesses

Reviewer_urByOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmarking for reconstructing the renewable generation suppressed by grid curtailment. The benchmark covers multiple wind an solar clusters and uses chronological evaluation separations. Based on the empirical comparison the paper argues that improvements over least squares should not be attributed to latent structures.

Critical strengths
  • The paper is generally understandable and provides a clear and consequential benchmark.
  • The paper points out a weakness in attributing improvements to structure. It provides evidence in Table 3 and Table 4 that the least squares is insufficient to benifit from latent decomposition.
Critical weaknesses
  • The main assumption of the paper is that the operator's reference generation is a sufficiently faithful target to evaluate the recovery of generating absent curtailment, but the procedure does not verify the calibration. The difficulty is identified, where the productivity curve is fitted on unrestricted hours and applied to restricted hours. The calculation also changes during the wind test.
  • The paper states, in section 6, that the structured model uses three constraints intended to make curtailment and availability separately identifiable, however, it does not provide the necessary conditions where these restrictions recover the intended physical factors. It seems a regularizer can select a decomposition that its factors correspond to physical availability and curtailment. This could also be a direct evidence that the input restriction is not a clean separation of information.
Questions for authors
  • What independent evidence bounds the error of the operator’s reference generation specifically during restricted hours?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.