ExtremePhysBench: Physical Structure Evaluation of Data-driven Weather Prediction Models under Extremes

Extreme weather events involve organized thermodynamic and dynamical structures, including thermal transport, moisture convergence, and vortex organization. Although data-driven weather prediction (DWP) has become competitive with numerical weather prediction (NWP) in aggregate forecast accuracy, its ability to reproduce these physical structures during extremes remains insufficiently characterized. We introduce ExtremePhysBench, a benchmark for systematically evaluating physical structures in forecasts of extreme weather events. The benchmark provides (i) an event catalog of 820 heat, cold, precipitation, and landfalling tropical cyclone events during 2022–2025, selected independently of the evaluated forecasts and yielding 6,317 forecast tasks; (ii) physical diagnostics comprising 25 universal and event-specific definitions that compare balance, transport, stability, and vortex organization with ERA5; and (iii) Physics Scores and physical families that quantify agreement with ERA5 and organize related diagnostics by physical process, supporting both process-level assessment and aggregate comparisons under a common evaluation protocol. Evaluating six DWP systems and operational GFS as the NWP baseline, we find that all six DWP systems achieve higher Overall Physics Scores than GFS. Physical family scores reveal model strengths and weaknesses in physical structure agreement across extreme weather event categories and processes, providing information complementary to conventional accuracy metrics and aggregate scores. Code is available at: https://anonymous.4open.science/r/ExtremePhysBench-REVIEW/.