- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ExtremePhysBench, a benchmark for evaluating the physical structures of data-driven weather prediction models during extreme events. The benchmark includes 820 events and 6,317 forecast tasks covering heat, cold, extreme precipitation, and tropical cyclones from 2022–2025. It evaluates 25 physical diagnostics organized into physical families, using ERA5 as the reference and a normalized Physics Score for comparison.
The authors evaluate six data-driven weather prediction models against operational GFS. FengWu performs best overall, while Aurora leads for tropical cyclones. The results highlight differences in physical structure representation across models and event categories, providing complementary information to conventional RMSE metrics.
Critical strengths
-
Comprehensive and reproducible benchmark. The benchmark covers multiple extreme weather categories with a reasonably large event catalog. Importantly, the events are selected independently of the evaluated models, with consistent evaluation regions and forecast lead times. This reduces selection bias and provides a useful framework for comparing physical structures across forecasting systems.
-
Process-level evaluation beyond aggregate metrics. I find the physical family decomposition particularly useful. For example, GraphCast performs better in water vapor content, while FengWu leads in moisture transport and convergence. Similar distinctions emerge for tropical cyclone circulation and warm-core structure. These results help identify which physical processes are represented well or poorly, rather than relying solely on aggregate forecast errors (Table 5).
Critical weaknesses
-
Limited justification and sensitivity analysis of the Physics Score. The score relies on prescribed normalization scales and tolerance parameters, followed by an exponential transformation and hierarchical averaging. These choices directly influence the resulting rankings, yet no sensitivity analysis is provided. Moreover, the overall ranking remains strongly correlated with conventional RMSE (Spearman ρ = 0.893), with identical rankings for heat and precipitation. I would like to see whether the conclusions remain robust under alternative normalization and weighting choices. This is particularly important if the Physics Score is intended to serve as a general benchmark metric.
-
Confounded comparison between data-driven models and GFS. All six data-driven models are initialized from ERA5, whereas GFS uses its own operational initial conditions, while ERA5 serves as the verification reference. This creates an inherent advantage for the data-driven models. Indeed, verifying GFS against its own analysis increases its Overall Physics Score from 26.82 to 30.33, with an 8.55-point increase for tropical cyclones (Section 6.3). Although the authors acknowledge this limitation, the main conclusion that all data-driven models outperform GFS in physical structure fidelity is difficult to interpret without a more controlled comparison.
-
Ultimately, this work appears incremental from earlier benchmark, especially ExtremeWeatherBench.
Questions for authors
See above.