- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This is a controlled study on UNISOLAR: 42 inverters on 5 campuses, 1-hour-ahead forecasts with 19 quantiles and one GRU. It compares physics supplied as inputs (clear-sky GHI, zenith and air mass) with physics added as loss penalties. The design includes a parameter-matched control, a LightGBM replication, a campus-clustered bootstrap and leave-one-campus-out folds. Physics inputs help (-2.21% CRPS), and the penalties do not. Transfer varies by campus, and single-fold degradation numbers are mostly a capacity artefact.
Critical strengths
This is the kind of evaluation paper the area needs. The statistics are unusually careful and honest. The paper treats the campus as the unit of inference, and it reports the seed sensitivity, the statistics it discarded, and the fact that LightGBM beats the neural model. It also clearly shows pseudo-replication and the capacity confound in cross-site evaluation. The design is well controlled.
Critical weaknesses
My main issue is with the regularisation arm, because the central claim depends on it. The thresholds were set so that the training targets violate them in only 0.0000% and 0.0244% of rows, and validation pinball loss selects λ = 0.1. In the supplement's Table3_night_violations.csv, the regularised model has about the same share of windows with a negative quantile as the controls (47.5% vs 48.4% and 48.1%). So the regulariser barely binds. The night comparison (L252-255) is not like-for-like either. The night ceiling is 5% of each inverter's 99th-percentile output, while the metric counts medians above 0.1 kWh/15 min. So for larger inverters (campuses 4 and 5, at about 17.8 and 6.1), a night median between these two levels is counted but not penalised. Only λ = 0.1 is reported on test, and the paper does not test stronger λ, hard output constraints or a clear-sky-index target. As it stands, the results show that this loose penalty set is ineffective, not that physics as regularisation is.
The scope is also narrow for the title: one dataset, one climate, one lead time and one small backbone. Campus 1 holds 66% of the test rows, and with five clusters the bootstrap p = 0.015 is not reliable. The paper itself notes that a campus-level test cannot go below 0.0625 (see also Cameron, Gelbach & Miller, 2008). Physics inputs are already standard in PV forecasting, so the positive result is expected. The mechanism and transfer claims are the novel part, and they remain unresolved.
This is a careful paper, and I would like to see it improved. For now, the central claim is not tested well enough, which is the main reason for my score.
Questions for authors
The questions that would most change my view: What were the validation results of the λ search, and what are the test results with λ = 1 and λ = 10, or with a hard constraint? How often is each penalty active, and how does a clear-sky-index arm compare? Does the headline hold with a wild cluster bootstrap or campus-level randomisation? Strong answers here would move me towards acceptance.
Also: does the control receive the campus identity or coordinates? If not, part of the gain may be location information. Does the ranking hold at other lead times?