acceptodds
Under review as a conference paper at ICLR 2027

Do AI Weather Models Miss Extremes? Evidence from Ten Months of Verification Against European Weather Stations

Abstract

AI weather models are often reported to underpredict extremes, but most evidence comes from deterministic regression systems verified against reanalysis. We compare twelve physical and AI forecast systems with ECMWF IFS using European station observations over ten months. The evaluation covers 10 m wind, 2 m temperature, solar radiation, and precipitation, with regimes defined from a fixed ERA5 1991–2020 climatology and models compared on matched samples. We find no uniform AI-specific loss of relative skill in the tails. Tail performance varies substantially among both learned and numerical systems: several AI systems remain more accurate than IFS in extreme regimes, while others deteriorate markedly. This conclusion persists across variables, although rankings are sensitive to forecast range, regime definition, and mean-bias correction. All systems also show a similar observation-conditioned error structure, overpredicting low observations and underpredicting high observations. Extreme-regime performance is therefore specific to the forecast system and evaluation setting rather than a uniform weakness of AI weather forecasting.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI weather forecasting, extreme weather, station-based verification, generative weather models, forecast evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WnonOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates 12 AI and physics-based weather forecasting models against ECMWF IFS using ten months of European weather-station observations. Through climatological quantile-based evaluation and selected extreme-event case studies, it finds that both AI and physical models underestimate high observations, while relative forecasting skill varies across models and weather variables rather than exhibiting a uniform AI-specific deficit.

Critical strengths
  1. Broad empirical coverage. The evaluation includes diverse AI and NWP systems across four weather variables, 13 European countries, and multiple forecast horizons.

  2. Observation-based verification. Using independent station observations instead of relying solely on reanalysis provides practical value for assessing operational forecast performance.

  3. Systematic analysis. The paper distinguishes conditional bias from relative forecasting accuracy and includes lead-time, regional, and uncertainty analyses.

Critical weaknesses

W1. Limited extreme-event coverage and definition. The evaluation focuses on four surface variables and primarily defines extremes through climatological quantiles (e.g., P95), which do not necessarily correspond to real hazardous or record-breaking events. Even the event-based analysis aggregates station observations within broad event windows, without adequately considering diverse, spatially aligned real-world extreme events as studied in HR-Extreme (ICLR 2025) and UniExtreme (KDD 2026).

W2. Limited spatial evaluation. Although the underlying forecasting models operate on spatiotemporal weather fields, the evaluation is reduced to station-level pointwise errors. It does not assess the spatial extent, localization, or evolution of extreme events, limiting its ability to characterize spatiotemporal forecasting performance.

W3. Limited novelty over existing studies. Prior studies, including Do data-driven models beat numerical models in forecasting weather extremes? and Physics-based models outperform AI weather forecasts of record-breaking extremes, have already investigated AI–NWP differences under extreme conditions, while WeatherReal established station-based verification. The paper does not sufficiently distinguish its contributions from these works or establish substantially new insights beyond broader model coverage.

W4. Incomplete extreme-event evaluation. Evaluating MAE conditioned on observed extremes overlooks false alarms and does not fully characterize event detection performance. Although CSI is reported for heavy precipitation, comparable detection metrics (e.g., POD, FAR, and precision–recall) are missing for extreme wind and temperature.

W5. Insufficient evaluation of probabilistic models. Probabilistic and generative ensembles are evaluated only through their ensemble means, leaving their ability to represent extreme-event uncertainty and tail distributions unexamined. Metrics such as CRPS, Brier Score, reliability, exceedance probabilities, and ensemble spread are needed to substantiate the motivation for including these model classes.

W6. Confounded model comparisons. The evaluated systems differ substantially in spatial resolution, training objectives, regional specialization, initialization, and training data. Consequently, the superior performance of certain regional AI models cannot be clearly attributed to AI-based forecasting or generative modeling rather than these confounding factors.

Questions for authors
  1. Were all forecasts generated prospectively or as retrospective hindcasts, and what were the model versions and training cutoffs relative to the evaluation period?

  2. How sensitive are the main conclusions and model rankings to the rolling bias correction? Could results without this correction be reported?

11 comments

We thank the reviewer for the careful reading and constructive comments. We address the individual points below.

AuthorsOpenReview ↗

W2. Spatial verification

We agree that station-level errors alone do not characterize the full spatial extent or evolution of an extreme event. Our evaluation uses station observations (Section 2.2) to provide an independent assessment of forecast accuracy, and we will clarify this scope. Could the reviewer specify which additional spatial diagnostic they consider most relevant? This would help us address the concern without conflating local accuracy with spatial-event verification.

AuthorsOpenReview ↗

W5. Full-ensemble probabilistic verification

We thank the reviewer for this suggestion. We have added an entire appendix section dedicated to probabilistic verification (Appendix A.7), covering CRPS, tail-weighted CRPS, Brier scores, reliability and ensemble spread. It evaluates all available members of EPT-2 HRRR, ECMWF AIFS ENS and EPT-2e, representing diffusion, CRPS-trained and perturbation-based ensembles, alongside the ECMWF Integrated Forecasting System ensemble (ECMWF ENS). We use matched wind/temperature cases at 6–48 h with the same additive correction; compatible hourly solar and precipitation accumulations are unavailable across all four systems.

All three AI ensembles have better temperature CRPS point estimates than ECMWF ENS. For wind, ECMWF ENS outperforms AIFS ENS and EPT-2e and is comparable to EPT-2 HRRR (Figure S16; Table S28). ECMWF ENS also has lower wind tail-weighted CRPS and Brier scores than all three AI ensembles, while temperature tail-score gains vary across models (Figure S17). None is consistently calibrated: all four overstate their highest event probabilities and have spread/RMSE below one (Figures S16 and S19). Thus the full-ensemble analysis also supports model- and variable-dependent performance, with substantial uncertainty-representation errors in both AI and NWP.

Reviewer_Y2bJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares very short term forecast accuracy of AI weather models with IFS during extreme weather. The models are both global and regional and both deterministic and stochastic. The forecast periods are limited to 48hrs. Evaluations are based on ten months of European weather station observations. Extremes are identified using ERA5 percentiles and a catalogue of major heat events and windstorms. Some AI models outperform IFS during extreme conditions, some don't.

Critical strengths

The main strength is the broad comparison of AI and traditional weather models, including global, regional, deterministic, and stochastic models. The paper compares a broad range of AI and traditional weather models using independent station observations instead of reanalysis. It considers several weather variables and extreme events, reports uncertainty estimates, and acknowledges important differences between models.

Critical weaknesses

I think the framing of the paper is overly generous to weather emulators given the actual test results presented. I also have concerns about the statistical methodology.

The event based analysis averages errors over all stations in affected countries and could be concealing large errors where conditions are most severe.

Events covering more stations or longer periods account for more observations and therefore have more weight in evaluations.

The climatological tests define extremes using the 95th percentile of ERA5. This is not particularly extreme, and ERA5 may smooth out variability seen at individual stations. More generally, the tests provide little evidence about generalization to unprecedented extremes. The authors do not examine whether forecast errors increase when conditions fall outside the training data.

They consider 48 hr forecasts, and count any reading over the defined threshold as an extreme, regardless of how long ago the weather phenomenon formed. So not only are the events highly correlated, but many of the forecast initial conditions may have already been extreme. Their tests do very little to asses how well the models forecast formation of the phenomenon. The tests also do not seem to include July and August.

The event catalogue seems to only include a handful of individual events. Despite the jackknife how confident we can be in the conclusions.

More generally, the paper focuses on whether AI models outperform IFS during extreme weather, rather than whether they predict extremes accurately. Some AI models also have higher resolution than IFS, making it difficult to separate the benefits of resolution from those of the learned dynamics.

Questions for authors

Can you distinguish forecasts initialized before an extreme event forms from those initialized after formation? How much of the reported skill reflects the propagation of existing events?

For models that allow it, can you extend the evaluation to longer forecast horizons (e.g., 7 days)?

Can you examine errors in extreme intensity and extend the event-detection analysis beyond precipitation, rather than focusing primarily on skill relative to IFS?

For ensemble models, have you examined whether predicted probabilities of extreme events are well calibrated?

3 comments
AuthorsOpenReview ↗

Follow-up on our rebuttal

Thank you again for your time and consideration. As the discussion period is approaching its end, we would appreciate your feedback on our responses and the additional analyses, marked in blue in the revised manuscript.

Could you please let us know whether these address your concerns, and whether any points remain unresolved? We would be happy to provide further clarification before the discussion closes.

Thank you for the changes made to address my questions/concerns. I have raised my score.

Thank you for reading our responses and reconsidering your assessment. Your questions helped us improve the paper a lot.

Reviewer_SVpvOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Rather than proposing any algorithms, this work did some empirical study to investigate why AI models failed in predicting extreme conditions. Three folders of contributions have been made:

  1. A broad, station-based data analysis with diverse AI and NWP models: while most existing studies evaluating extremes tested deterministic models against gridded reanalysis (such as ERA5), this work conducts a 10-month evaluation using ground-truth European weather stations, rain gauges (over 4,000 gauges), and solar radiation networks across 13 countries. In addition, it also evaluates twelve forecast models across different paradigms: physical NWP, deterministic regression model, probabilistic/global ensembles, generative regional AI models.

  2. A deep-dive on loss functions for predictive modeling and multi-regime climatological evaluation protocol: This work is aware of the limitation of "conditional mean" as prediction target studied by the previous work, especially the over-smoothing properties/effects of conditional mean, and it extends its scope just beyond the traditional MSE loss or simple variants. It incorporates models with continuous ranked probability score, diffusion modeling, and NWP into the study. To evaluate the performance, in addition to the common prediction errors, it constructs an evaluation protocol using a 30-year local ERA5 climatology (1991–2020) to define stationary, localized quantile regimes (multiple percentile levels) for 10 m wind, 2 m temperature, hourly solar radiation, and hourly precipitation, combined with a causal rolling bias-correction framework.

  3. A reanalysis that disentangs conditional errors from relative tail accuracy: though I think this part is common knowledge in statistics--rooted in the behavior of higher-order moments, skewness, and regression-to-the-mean effects in heavy-tailed weather distributions--it is very good that the authors revisited this principle in the deep learning/AI or so-called large model era. Specifically, much of the recent literature has revisited the conditional underestimation of extremes (a statistical consequence of predicting conditional expectations or optimizing lower-order losses like MSE) with a general failure of AI models. By demonstrating that both data-driven architectures and operational physical NWP baselines exhibit the exact same conditional error signature (overpredicting lower quantiles and underpredicting upper quantiles), the paper decouples absolute attenuation from relative forecast skill.

Critical strengths
  1. This work is a comprehensive and rigorous empirical study: the scale, quality of data, length of evaluation regimes , coverage of models.

  2. This work provides a robust dual-angle evaluation coupling climatological regimes with real-world case studies: subject to the common failure of model prediction, into introduces a comprehensive evaluation protocol that can support the reanalysis.

  3. This work provides a reanalysis that disentangles conditional errors from relative tail accuracy: this work revisited the most critical problem in extreme value prediction, such as weather forecasting. While the authors do not explicitly frame it in formal statistical terminology (such as higher-order moments, skewness, and regression-to-the-mean shrinkage under heavy-tailed distributions), their findings align squarely with established statistical principles.

Critical weaknesses

The lack of higher-order moment analysis is a major limitation, and there are several other notable weaknesses from both statistical and methodological perspectives:

  1. While this work focusing on "extremes", it does not evaluate whether models reproduce the higher-order moments of the true physical process -- such as variance, skewness, kurtosis, power spectra, or spatial/temporal covariance in extreme conditions.
  2. The analysis relies on empirical quantile regimes rather than formal EVT frameworks (e.g., Generalized Extreme Value distributions, Pareto tails, return levels, or shape parameter estimation), leaving the asymptotic tail behavior uncharacterized.
Questions for authors

Please try to address two weakness points above.

5 comments

Thank you for the careful review. We address the two points below.

AuthorsOpenReview ↗

W1. Moments and spatial structure

Following the reviewer’s suggestion, we added moment and Q–Q verification for the bias-corrected Track A wind and temperature point forecasts over the full ten-month period (Appendix A.10). At each initialization and lead, we calculate the forecast and observed mean, SD, variance, skewness and excess kurtosis across the same stations, then average their absolute differences. Figures S26 and S27 report errors at observed-P95 stations and all matched stations, respectively; station observations provide the verification reference and IFS the comparison baseline.

The leading model depends on the variable and statistic. At observed-P95 stations, EPT2-HRRR has the smallest temperature-mean error and Europa the smallest errors in the other four statistics. For wind, ICON leads the mean, IFS the SD and variance, and Europa the skewness. Figure S25 shows that GFS more closely matches marginal upper-wind quantiles, while still underestimating the furthest plotted tail. These results extend the paper’s finding of no uniform AI–NWP ordering to distributional diagnostics.

These additions assess cross-station moments and marginal quantiles, rather than spatial or temporal covariance. Could the reviewer clarify which spectral diagnostic they have in mind? Our verification uses irregularly distributed station observations, which do not directly support conventional spatial spectra. We would appreciate clarification on the quantity and spatial or temporal scales of interest.

AuthorsOpenReview ↗

W2. EVT and asymptotic tails

Following the reviewer’s suggestion, we added a peaks-over-threshold wind analysis at 24 h (Appendix A.10; Figure S28). We fit generalised Pareto distributions to observations and each Track A point forecast above the same pooled observed P95 threshold (8.2 m/s), using matched station–time cases. Forecasts are bias-corrected, with ensemble means used where applicable.

Figure S28 compares fitted exceedance frequencies with empirical observations, with uncertainty estimated by paired seven-day block resampling. All ten displayed point forecasts produce fewer readings above 15 and 20 m/s than observed. GFS is closest to the observed pooled tail, while IFS and the AI systems show larger deficits. This agrees with the Q–Q comparison in Figure S25: upper-wind underrepresentation occurs in both AI and NWP forecasts, with substantial differences between individual systems.

Comments on the paper

9 comments
AuthorsOpenReview ↗

Author Rebuttal by Authors

We thank all three reviewers for their thorough reading of our paper and constructive suggestions. Their comments helped us clarify the contribution and extend the evaluation. We have added experiments and revised the manuscript accordingly; the additional analyses and discussion are highlighted in blue in the revised PDF. We respond to each reviewer’s comments individually below.

AuthorsOpenReview ↗

W1. Framing and statistical interpretation

The paper’s conclusion is that NWP is not uniformly more accurate than AI during the evaluated extremes. Figure 3 shows why underprediction alone does not establish an AI-specific failure: IFS and the AI systems share the pattern of overpredicting low observations and underpredicting high observations.

We address the specific concerns about event weighting and jackknife uncertainty in W3 and W6. If the reviewer has additional concerns about the statistical methodology, we would appreciate clarification on which methods or results they question so that we can respond directly.

AuthorsOpenReview ↗

W2. Severe locations hidden by country averages

We agree that country averages can hide large local errors. Figure S21 therefore shows the distribution of station-level MAEs, with each station contributing one value. At each station, we retain observations above its local P95 within the documented episodes and average the corresponding absolute forecast errors over the 6–48 h leads. Forecasts are bias-corrected, ensemble systems use their means, and all systems are evaluated on the same cases.

The boxes show P25–P75 of station MAEs, with P5–P95 whiskers. They show how forecast accuracy varies across stations without averaging over countries. Errors are still averaged within each station, however; Figure S22 separately examines forecasts at selected observed event peaks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.