- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper evaluates 12 AI and physics-based weather forecasting models against ECMWF IFS using ten months of European weather-station observations. Through climatological quantile-based evaluation and selected extreme-event case studies, it finds that both AI and physical models underestimate high observations, while relative forecasting skill varies across models and weather variables rather than exhibiting a uniform AI-specific deficit.
Critical strengths
-
Broad empirical coverage. The evaluation includes diverse AI and NWP systems across four weather variables, 13 European countries, and multiple forecast horizons.
-
Observation-based verification. Using independent station observations instead of relying solely on reanalysis provides practical value for assessing operational forecast performance.
-
Systematic analysis. The paper distinguishes conditional bias from relative forecasting accuracy and includes lead-time, regional, and uncertainty analyses.
Critical weaknesses
W1. Limited extreme-event coverage and definition. The evaluation focuses on four surface variables and primarily defines extremes through climatological quantiles (e.g., P95), which do not necessarily correspond to real hazardous or record-breaking events. Even the event-based analysis aggregates station observations within broad event windows, without adequately considering diverse, spatially aligned real-world extreme events as studied in HR-Extreme (ICLR 2025) and UniExtreme (KDD 2026).
W2. Limited spatial evaluation. Although the underlying forecasting models operate on spatiotemporal weather fields, the evaluation is reduced to station-level pointwise errors. It does not assess the spatial extent, localization, or evolution of extreme events, limiting its ability to characterize spatiotemporal forecasting performance.
W3. Limited novelty over existing studies. Prior studies, including Do data-driven models beat numerical models in forecasting weather extremes? and Physics-based models outperform AI weather forecasts of record-breaking extremes, have already investigated AI–NWP differences under extreme conditions, while WeatherReal established station-based verification. The paper does not sufficiently distinguish its contributions from these works or establish substantially new insights beyond broader model coverage.
W4. Incomplete extreme-event evaluation. Evaluating MAE conditioned on observed extremes overlooks false alarms and does not fully characterize event detection performance. Although CSI is reported for heavy precipitation, comparable detection metrics (e.g., POD, FAR, and precision–recall) are missing for extreme wind and temperature.
W5. Insufficient evaluation of probabilistic models. Probabilistic and generative ensembles are evaluated only through their ensemble means, leaving their ability to represent extreme-event uncertainty and tail distributions unexamined. Metrics such as CRPS, Brier Score, reliability, exceedance probabilities, and ensemble spread are needed to substantiate the motivation for including these model classes.
W6. Confounded model comparisons. The evaluated systems differ substantially in spatial resolution, training objectives, regional specialization, initialization, and training data. Consequently, the superior performance of certain regional AI models cannot be clearly attributed to AI-based forecasting or generative modeling rather than these confounding factors.
Questions for authors
-
Were all forecasts generated prospectively or as retrospective hindcasts, and what were the model versions and training cutoffs relative to the evaluation period?
-
How sensitive are the main conclusions and model rankings to the rolling bias correction? Could results without this correction be reported?
11 comments
W2. Spatial verification
We agree that station-level errors alone do not characterize the full spatial extent or evolution of an extreme event. Our evaluation uses station observations (Section 2.2) to provide an independent assessment of forecast accuracy, and we will clarify this scope. Could the reviewer specify which additional spatial diagnostic they consider most relevant? This would help us address the concern without conflating local accuracy with spatial-event verification.
W5. Full-ensemble probabilistic verification
We thank the reviewer for this suggestion. We have added an entire appendix section dedicated to probabilistic verification (Appendix A.7), covering CRPS, tail-weighted CRPS, Brier scores, reliability and ensemble spread. It evaluates all available members of EPT-2 HRRR, ECMWF AIFS ENS and EPT-2e, representing diffusion, CRPS-trained and perturbation-based ensembles, alongside the ECMWF Integrated Forecasting System ensemble (ECMWF ENS). We use matched wind/temperature cases at 6–48 h with the same additive correction; compatible hourly solar and precipitation accumulations are unavailable across all four systems.
All three AI ensembles have better temperature CRPS point estimates than ECMWF ENS. For wind, ECMWF ENS outperforms AIFS ENS and EPT-2e and is comparable to EPT-2 HRRR (Figure S16; Table S28). ECMWF ENS also has lower wind tail-weighted CRPS and Brier scores than all three AI ensembles, while temperature tail-score gains vary across models (Figure S17). None is consistently calibrated: all four overstate their highest event probabilities and have spread/RMSE below one (Figures S16 and S19). Thus the full-ensemble analysis also supports model- and variable-dependent performance, with substantial uncertainty-representation errors in both AI and NWP.
We thank the reviewer for the careful reading and constructive comments. We address the individual points below.