acceptodds
Withdrawn from ICLR 2027

ExtremePhysBench: Physical Structure Evaluation of Data-driven Weather Prediction Models under Extremes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Extreme weather events involve organized thermodynamic and dynamical structures, including thermal transport, moisture convergence, and vortex organization. Although data-driven weather prediction (DWP) has become competitive with numerical weather prediction (NWP) in aggregate forecast accuracy, its ability to reproduce these physical structures during extremes remains insufficiently characterized. We introduce ExtremePhysBench, a benchmark for systematically evaluating physical structures in forecasts of extreme weather events. The benchmark provides (i) an event catalog of 820 heat, cold, precipitation, and landfalling tropical cyclone events during 2022–2025, selected independently of the evaluated forecasts and yielding 6,317 forecast tasks; (ii) physical diagnostics comprising 25 universal and event-specific definitions that compare balance, transport, stability, and vortex organization with ERA5; and (iii) Physics Scores and physical families that quantify agreement with ERA5 and organize related diagnostics by physical process, supporting both process-level assessment and aggregate comparisons under a common evaluation protocol. Evaluating six DWP systems and operational GFS as the NWP baseline, we find that all six DWP systems achieve higher Overall Physics Scores than GFS. Physical family scores reveal model strengths and weaknesses in physical structure agreement across extreme weather event categories and processes, providing information complementary to conventional accuracy metrics and aggregate scores. Code is available at: https://anonymous.4open.science/r/ExtremePhysBench-REVIEW/.

Primary area:
datasets and benchmarks
Keywords:
Extreme weather, Data-driven weather prediction, Physical structure evaluation, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1HqqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ExtremePhysBench, a benchmark for evaluating the physical structures of data-driven weather prediction models during extreme events. The benchmark includes 820 events and 6,317 forecast tasks covering heat, cold, extreme precipitation, and tropical cyclones from 2022–2025. It evaluates 25 physical diagnostics organized into physical families, using ERA5 as the reference and a normalized Physics Score for comparison.

The authors evaluate six data-driven weather prediction models against operational GFS. FengWu performs best overall, while Aurora leads for tropical cyclones. The results highlight differences in physical structure representation across models and event categories, providing complementary information to conventional RMSE metrics.

Critical strengths
  • Comprehensive and reproducible benchmark. The benchmark covers multiple extreme weather categories with a reasonably large event catalog. Importantly, the events are selected independently of the evaluated models, with consistent evaluation regions and forecast lead times. This reduces selection bias and provides a useful framework for comparing physical structures across forecasting systems.

  • Process-level evaluation beyond aggregate metrics. I find the physical family decomposition particularly useful. For example, GraphCast performs better in water vapor content, while FengWu leads in moisture transport and convergence. Similar distinctions emerge for tropical cyclone circulation and warm-core structure. These results help identify which physical processes are represented well or poorly, rather than relying solely on aggregate forecast errors (Table 5).

Critical weaknesses
  • Limited justification and sensitivity analysis of the Physics Score. The score relies on prescribed normalization scales and tolerance parameters, followed by an exponential transformation and hierarchical averaging. These choices directly influence the resulting rankings, yet no sensitivity analysis is provided. Moreover, the overall ranking remains strongly correlated with conventional RMSE (Spearman ρ = 0.893), with identical rankings for heat and precipitation. I would like to see whether the conclusions remain robust under alternative normalization and weighting choices. This is particularly important if the Physics Score is intended to serve as a general benchmark metric.

  • Confounded comparison between data-driven models and GFS. All six data-driven models are initialized from ERA5, whereas GFS uses its own operational initial conditions, while ERA5 serves as the verification reference. This creates an inherent advantage for the data-driven models. Indeed, verifying GFS against its own analysis increases its Overall Physics Score from 26.82 to 30.33, with an 8.55-point increase for tropical cyclones (Section 6.3). Although the authors acknowledge this limitation, the main conclusion that all data-driven models outperform GFS in physical structure fidelity is difficult to interpret without a more controlled comparison.

  • Ultimately, this work appears incremental from earlier benchmark, especially ExtremeWeatherBench.

Questions for authors

See above.

Reviewer_z274OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ExtremePhysBench, a benchmark that checks whether data-driven weather prediction (DWP) models keep the physics right during extreme events. It covers 820 events from 2022 to 2025 in four categories: heat, cold, heavy precipitation and tropical cyclones (TCs). For each event, the authors compute physical proxy diagnostics from the forecast and from ERA5, map the RMSE between them to a score from 0 to 100, and average the scores with equal weights into family, category and overall scores. Six DWP models and GFS are ranked, and FengWu comes first.

Critical strengths

S1. The 820 events are picked from external products, and the code is released, so the catalog can be reused by others.

Critical weaknesses

W1 (critical). The score function, the choice of diagnostics and the aggregation are subjective and not justified.

Dividing by the scale S_i = C_i·τ_i removes units and puts the diagnostics on a similar scale, but the paper gives no other reason for these values. The tolerances τ_i are hand-picked and range from 0.001 to 1.0 (Table 13), and the paper itself calls them design choices (Sec. 4.3). The paper also explains what each diagnostic measures, but not why this set of 10 universal and 5 event-specific diagnostics per category is the right one. Averaging by family gives U1 a weight of 1/4, but each of U4 to U8 only 1/20. The paper says the equal weights only reflect reporting priorities, and it does not test other weights, even though the OECD guidance it cites asks for this (App. C.3). These choices can easily flip a ranking. For example, take two models with two diagnostics each, with normalized errors of (0, 2) and (1.1, 1.1). The first model scores 50.92 and the second 29.82. If the scale is doubled, they score 68.39 and 73.90, and the order is reversed. This matters here, because the models ranked 2nd to 4th in Table 4 are within 0.54 points of each other. Also, the ranking is close to the plain RMSE ranking (ρ = 0.893, Sec. 6.3), so it is not clear what the score adds.

W2 (critical). ERA5 is not a reasonable reference for extreme events.

ERA5 itself is not accurate in extreme cases, and extremes are exactly what this benchmark tests. The clearest case is TC intensity, which ERA5 underestimates [1], while several TC diagnostics look at the inner core (e.g., E1 tangential wind within 150 km, E5 warm core). A model with a stronger and more realistic storm would therefore lose points. All six DWP models start from ERA5 initial conditions, but GFS starts from its own analysis (App. E.1). When GFS is checked against its own analysis, its TC score goes up by 8.55 points (Table 17). So the scores partly measure how close a model is to ERA5, not to the real atmosphere. The paper picks precipitation events with IMERG and TCs with IBTrACS, but apart from the GFS test, all scores are computed against ERA5 only, and the errors of ERA5 in extremes are not discussed.

W3. The paper is hard to read.

The core idea is simple: compute the RMSE of proxy variables, turn it into a score, and average. But it is buried under many self-defined terms (e.g., admissible support, terminal coverage), and it took me a lot of effort to work out what is actually computed.

W4. The related work is poorly organized.

The second direction (WeatherQA) has little to do with this paper, while the most relevant work (Hakim & Masanam, 2024; Bonavita, 2024; WeatherBench 2) only shows up in Appendix A.

Reference

[1] Alet, F., Andersson, T.R., Price, I. et al. Operational tropical cyclone forecasting with AI. Nature 657, 680–688 (2026). https://doi.org/10.1038/s41586-026-10953-2

Questions for authors
  1. Score function. How were C_i, τ_i and the exp(−r²) map chosen? How do the rankings change if τ_i is halved or doubled?

  2. Physical diagnostics. Why these 10 universal diagnostics and 5 event-specific diagnostics per category? What errors do they catch that RMSE misses?

  3. Aggregation. Apart from reporting priorities, why equal weights per family and per group? How do the rankings change with other weights?

  4. Justification. Please give a sound, scientific reason for the score function, for using this many universal and event-specific diagnostics, and for the way they are combined. Removing units and scale is one reason for the score function, but it is not enough on its own. At present the paper gives no real evidence that these choices are reasonable, for example a sensitivity analysis, a check against observations, or cases where the score finds physical errors that RMSE misses.

  5. ERA5 as the reference. ERA5 itself is not accurate in extreme cases, so using it as the reference is not reasonable. How large are the errors of ERA5 for these events, for example TC intensity against IBTrACS? How do the scores and rankings change if observations or a higher-resolution analysis are used as the reference?

Reviewer_7ngdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces ExtremePhysBench, a process-oriented benchmark for evaluating forecast physical structures in extreme weather. Using 820 events (heat, cold, precipitation, and landfalling tropical cyclones) from 2022–2025, it establishes 6,317 forecast tasks and 25 diagnostics. Diagnostic errors against ERA5 are mapped to bounded Physics Scores. Evaluating six data-driven systems against operational GFS shows that all six outperform GFS, with FengWu achieving the highest overall score and Aurora leading in tropical cyclones.

Critical strengths
  1. A useful evaluation resource with a carefully specified protocol. The forecast-independent event catalog, common diagnostic operators, terrain and latitude applicability rules, and explicit hierarchical aggregation form a substantial benchmark contribution for machine-learning weather forecasting.

  2. For precipitation, the leading systems differ between water amount and moisture transport/convergence; for TCs, they differ between circulation and warm-core structure. These distinctions make the benchmark potentially useful for diagnosing model weaknesses. The paper also carefully distinguishes agreement with ERA5 from complete physical-budget closure, and the universal-only Overall score from Category scores that include event-specific diagnostics.

Critical weaknesses
  1. It is unclear how excluding TC cases affects the main comparisons. Some TC diagnostics become invalid when the predicted storm position brings part of the evaluation region across the latitude cutoff、. Because comparisons require complete results from all models, these failures can affect which storms remain in the analysis、. The matched analysis includes 55 of the 125 cataloged storms、, but it is unclear how many exclusions arise from these failures and how many storms contribute to each result in Table 4. This matters because adding a model with more invalid diagnostics could change the sample used to compare every model. Please report the sample counts for the main comparisons, distinguish forecast failures from reference-side inapplicability, and check whether the exclusions affect the TC and Overall conclusions.

  2. The main rankings need to be checked under other reasonable scoring choices. The final results depend on the chosen error scales, tolerances, and weights (Eq. 2). These choices are clearly documented, but the paper does not show whether reasonable alternatives would change the main conclusions; Appendix C.3 explicitly notes that no weighting sensitivity experiment was performed. This is particularly relevant because the exponential transformation compresses differences among large errors before averaging. The bootstrap intervals measure uncertainty from sampling events while keeping the scoring rule fixed, so they do not address sensitivity to that rule. A small sensitivity analysis using the stored diagnostic errors would help show which model rankings and physical-family comparisons are stable.

Questions for authors
  1. How many TC events and forecast leads contribute to each result in Table 4? Please distinguish diagnostics that cannot be evaluated in ERA5 from those that become invalid because of a model’s predicted storm position. Do the excluded storms differ from the retained storms in latitude, basin, or intensity? Using the existing outputs, could you repeat the TC and Overall comparisons with U2/U3 removed from the TC evaluation for all models, using the same eligible storms across models? Would the main findings change?

  2. Since the core Physics Score scoring mechanism contains a large number of subjectively set hyperparameters, but lacks the necessary sensitivity and robustness verification, could we utilize the calculated diagnostic errors? Could you please review the main results in Tables 4 and 5 and check if they still hold under several other reasonable tolerance and family weight conditions? For instance, halve or double the tolerance and test different weighting schemes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.