- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper applies split conformal prediction to deterministic neural weather forecasting models like Stormer and ClimaX. The authors define a spatially aware nonconformity score based on the 90% percentile of the absolute error field. This score targets a field level coverage event to guarantee that at least 90% of grid cells fall within the prediction interval. The work also explores Mondrian conformal prediction by stratifying the globe into latitude bands to account for regional error differences. The authors evaluate these methods on ERA5 reanalysis data. They highlight the difference between mean grid coverage and field event rates. Furthermore, they conduct chronological stress tests to assess performance under temporal distribution shifts.
Critical strengths
-
The paper articulates the difference between average spatial coverage and field level event coverage. This is a subtle but important distinction when evaluating spatial forecasts.
-
The chronological stress test honestly exposes the limitations of the exchangeability assumption in standard conformal prediction when applied to real world time dependent weather data.
-
The ablation studies on label delay sensitivity provide a realistic look at operational constraints.
Critical weaknesses
-
Lack of Novelty: Applying standard split conformal prediction using a spatial quantile as the nonconformity score is a trivial methodological extension.
-
Assumption Failure: The core theorem in Proposition 1 requires exchangeability. Weather data inherently violates this requirement. The chronological tests presented by the authors show the global conformal prediction field event rate drops to 71.4 percent compared to the 90 percent target. This renders the theoretical guarantee practically void.
-
Metric Relevance: The field level event metric can mask severe regional failures. The paper admits that under the Mondrian approach the worst band coverage drops to 48.2 percent. This means the uncertainty quantification is highly unreliable in specific regions.
-
Missing Baselines: The paper fails to compare against established probabilistic ensemble methods like GenCast or ECMWF ENS. This omission makes it impossible to assess if this post hoc calibration offers any competitive advantage over current standard practices.
-
Poor Performance on Extremes: The method systematically fails to provide adequate coverage during extreme weather events. The coverage drops to 66.7 percent for 99th percentile anomalies. This is arguably the most important use case for uncertainty quantification.
Questions for authors
-
Weather data inherently violates the exchangeability assumption as demonstrated by the chronological stress test dropping to 71.4 percent coverage. Under this condition what is the practical value of Proposition 1? Why not use adaptive conformal inference methods explicitly designed for time series?
-
The worst band score event rate is only 48.2 percent under the Mondrian approach. How can an operational meteorologist trust this calibration method for regional warnings?
-
There is no comparison against probabilistic neural weather models like GenCast or FuXi ENS. Comparing the quality of the uncertainty quantification is necessary to benchmark your approach against the state of the art even if these models use different backbones. Why was this comparison omitted?
-
How do you justify the inclusion of the EVT fine tuning negative results? In what way does this failed experiment support the main narrative of conformal prediction?
3 comments
I have carefully considered the authors' rebuttal and the other reviewers' comments. Although the authors have resolved several of my initial concerns, the fundamental issue of limited novelty persists. Utilizing a spatial quantile as the nonconformity score in standard split conformal prediction represents only a marginal methodological increment. Consequently, I intend to keep my original rating.
Thank you for reconsidering the added evidence and for clarifying the remaining concern. The contribution beyond the standard split-conformal theorem is the specification and audit of operational coverage targets for weather fields. A complete forecast field is the sampling unit, spatial dependence is retained, and field, regional, chronological, and label-availability questions are evaluated within one framework.
The added experiments show why this matters. Separate six-band calibration attains 97.02% global field-event coverage, but only 16.96% simultaneous six-band coverage; calibrating the maximum regional score raises simultaneous coverage to 77.68%. Under a 90-day label delay, ACI raises field-event coverage from 85.42% to 89.29% with finite intervals and a 0.377 K width increase. We will foreground these target-specific and deployment-relevant findings in the revision.
Thank you for the questions. The added experiments directly address regional coverage and adaptive calibration: joint regional calibration substantially improves simultaneous coverage, and ACI improves delayed-feedback coverage with finite intervals at gamma=0.005.
Regional and extreme-event results
We evaluated the same 336 Stormer test forecasts on six latitude bands. Alongside separate band calibration, we calibrated the maximum of the six band scores to target their simultaneous event. Both rules use the same 240 calibration forecasts.
Each band event requires coverage of at least 90% of its unweighted cells.
Joint calibration raises the worst-band rate from 48.21% to 82.44% and simultaneous coverage from 16.96% to 77.68%. The latter has a 30-day-block 95% interval of [70.54%, 84.04%], compared with [11.14%, 23.12%] for separate calibration. The joint rule uses one common width: it is substantially wider and still falls below 90% simultaneous coverage on this chronological split.
For cells exceeding the same calibration-defined 99th-percentile absolute-anomaly threshold:
These results show how changing the calibration target improves regional and extreme-cell coverage, together with its width cost. They do not turn the original 97.02% global field-event rate into a guarantee for regional warnings.
Adaptive calibration under temporal shift
Rolling calibration and adaptive conformal inference (ACI) use identical forecasts and the latest 60 eligible labels. Feedback arrives only at verification time plus the assumed delay.
All intervals in these rows are finite. At a 90-day delay, ACI gains 3.87 percentage points for an additional 0.377 K of width. We also tested gamma=0.01 and 0.02. Larger step sizes can raise coverage further, but introduce unbounded intervals; gamma=0.01 reaches 90.77% with 23.51% unbounded dates at the 90-day delay.
Proposition 1 specifies the event and conditions under which it is guaranteed. It does not certify chronological coverage. The adaptive comparison measures what can be recovered under temporal shift and delayed labels. Release delays remain retrospective assumptions applied to final archived targets.
Ensembles and EVT
The application is interval construction for an existing deterministic forecast without retraining or ensemble generation. We have not performed matched GenCast, FuXi ENS, or ECMWF ENS comparisons, so no advantage over those systems is claimed.
We will remove the EVT fine-tuning experiment from the main narrative. The revised empirical conclusion will focus on the measured gains from regional joint calibration and adaptive feedback, with interval width reported alongside coverage.