acceptodds
Under review as a conference paper at ICLR 2027

Can AI Forecast Weather Extremes? Evaluating Probabilistic AI Models on Tropical Cyclones

Abstract

Weather prediction is undergoing an important shift, with data-driven AI models outperforming leading numerical weather prediction systems on average at a fraction of the computational cost. Yet whether this success extends to extreme events remains contested. Earlier evaluations have exposed important shortcomings, but have largely focused on deterministic models and metrics, leaving the capabilities of the latest probabilistic models less understood. We conduct an extensive benchmark of tropical cyclone forecasting, comparing GenCast, FourCastNet 3, and FGN with ECMWF’s operational physics-based ensemble, IFS-ENS. We develop a suite of probabilistic metrics and diagnostic analyses to assess cyclone tracking, intensity biases, uncertainty, and the wind–pressure relationship that characterizes storm structure. We find that, despite remaining limitations in intensity and ensemble spread, leading AI models substantially outperform IFS-ENS in cyclone tracking: FGN achieves better probabilistic location scores for approximately 90% of evaluated storms in 2025. Further analyses reveal model-specific failure modes and suggest a lingering influence of pretraining data, pointing to opportunities for targeted improvement. Nonetheless, these results suggest that AI models are capable of capturing physical relationships that remain predictive well beyond typical weather conditions, motivating further investigation into how these capabilities emerge and how to strengthen them.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
machine learning, weather, extremes, tropical cyclones

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1AyiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a benchmark of tropical cyclone forecasting based on several probabilistic DLWF models that are SOTA coming from industries. The evaluation is comprehensive, examining not only cyclone tracking, but also intensity biases, ensemble calibration, spatial agreement, and wind–pressure dependence. and so on. The results show that leading AI models substantially outperform IFS-ENS in cyclone tracking, while limitations in intensity and ensemble spread remain.

Critical strengths
  1. The paper provides a comprehensive evaluation of recent probabilistic DLWF models across multiple evaluation settings and cyclone seasons. These findings may be insightful and helpful to the community for developing next-generation DLWF models for tropical cyclone prediction.

  2. The wind–pressure analysis provides interesting findings that a physically plausible joint structure can emerge without being explicitly added or guided by the training process.

Critical weaknesses
  1. First of all, the presentation of this paper should be substantially improved. This work sounds more like an empirical study, in which several observations, findings, conclusions, and criticisms are explored. However, the content is structured in a tedious way for me. It would be better to concisely present the key take-home messages at the beginning of each section, with reference to more detailed justifications.

  2. Another key concern is that the physical interpretation is limited. Similar pooled wind–pressure dependence does not establish that the models have learned an explicit physical mechanism. The effects of storm-level heterogeneity and aggregation remain insufficiently explored.

  3. Several comments regarding more detailed experimental designs from the perspective of physical plausibility and domain knowledge:

(a) Trajectory NLL only evaluates successfully detected and matched cyclones, retaining 106 of 125 storms in 2021–2023 and 41 of 53 in 2025. The impact of these filtering rules on model rankings remains unclear.

(b) Although the paper focuses on extreme events, rapid intensification and probabilistic forecasts of hazardous wind-speed thresholds are not directly evaluated. Spatial agreement based on percentile thresholds removes systematic intensity biases.

(c) Peak-wind bias is evaluated within regions enclosing observed tracks rather than predicted cyclone centers, so displacement errors may influence the estimated intensity biases.

(d) The main model comparisons lack paired confidence intervals or significance tests. Sensitivity to cyclone detection, matching, and filtering thresholds is not systematically investigated.

(e) The evaluation focuses on known tropical cyclones and excludes periods without observed cyclones. So, the generalizability of this evaluation is limited.

Questions for authors

Can you clarify my concerns about the weakness?

5 comments

Thank you for your review. We restructured the paper the way you suggested, and we ran a new analysis for each of the points you raised:

  • the wind–pressure relationship within storms, storm by storm, and across members (physical interpretation and aggregation);
  • the track scores with different filters, and on all storms, the excluded ones included (a);
  • rapid intensification, for all the models (b);
  • fixed wind thresholds, for the probability of strong winds and for the wind footprints (b);
  • the peak-wind bias at the predicted centre (c);
  • confidence intervals, paired tests, and the sensitivity to the bandwidth, the matching and the tracker (d);
  • a case study of one atypical storm, Cyclone Alfred.

We hope this answers the weaknesses you listed. If one of them is still open for you, tell us which one and we will work on it during the discussion period. And if you feel we have addressed your points, we would be happy if you could reflect it in your rating.

We start with the presentation, which was your main concern, and then go through your other points one by one.

Presentation

We followed your suggestion: each results section now starts with its key message, in italics under the heading, and the details come after. Each message answers one of the research questions that the introduction now lists. These are the messages:

  • Section 3 (2021–2023, models trained on ERA5). GenCast gives more probability to the observed position than IFS-ENS on 68% of storms. This lead is significant and holds across filtering choices, while FourCastNet 3's lead is not significant. The two AI models tie on global scores, but GenCast is 3 to 5% better near cyclones. GenCast underestimates peak winds more than FourCastNet 3, so a better score near the storm does not mean a better intensity.
  • Section 4 (2025, operational models). FGN beats IFS-ENS on tracks on 90% of storms, whatever the filtering choices. The gains of the AI models over IFS-ENS are larger near cyclones than globally, including for strong storms. FGN places the strong winds best, then GenCast, when their magnitude is set aside. At a fixed threshold of 33 m/s, the AI models rarely reach it and fall behind IFS-ENS. Their winds are too weak, more than those of IFS-ENS, especially in strong storms, and their ensembles are underdispersed.
  • Section 5 (wind–pressure relationship). Once the intensity biases are removed, the models link wind to pressure like the references, even FGN, whose loss treats each variable separately. Storm by storm, they still gain less wind per hPa than HRES-fc0, mostly because their distributions are biased. This is a statistical dependence, not evidence of a learned physical mechanism.
  • Section 6 (rapid intensification). It is almost never forecast: an AI member produces fewer than one event over the 53 storms, against 62 observed. After adjusting for the intensity bias, the models catch about one event in five, mostly with false alarms, like IFS-ENS and HRES-fc0.

We also added a case study of one storm, Cyclone Alfred, which has the least typical track of 2025 (Appendix B.11). It shows the main results on a single forecast: the AI models stay closer to the observed track than IFS-ENS for the first ten days, their winds are weaker, and no member of any model reproduces its rapid intensification at the right time.

Following your comment on next-generation models, we also added to the conclusion the directions we think matter most for future work. The tracks are already good, so the effort should go to intensity. The training data is itself too weak in the core of the storm, so this will probably need other data, and not only another loss.

Physical interpretation and aggregation

We used "physical" because the relationship itself has a physical origin. In the core of the storm, the radial pressure gradient is balanced by the centrifugal and Coriolis forces, so a deeper centre pressure is balanced by stronger winds (Chavas et al., 2017, and its empirical form in Knaff and Zehr, 2007). We wanted to test whether the forecasts reproduce it. As you say, and as we noted in the submission (ll. 431–433), this does not imply that the models have learned an explicit physical mechanism. But we see that the wording suggested it, so we removed "physical relationship" from the paper, and we only speak of the empirical relationship between wind and pressure.

The finding itself still holds: FGN is trained with an objective that scores each variable separately, and its forecasts still have a wind–pressure relationship close to the one in the reference.

On aggregation, we looked at the relationship at four levels (rank correlations, 2025):

LevelFGNGenCastIFS-ENSHRES-fc0
All storms pooled, as in the submission−0.83−0.84−0.87−0.88
Within storms (storm means removed)−0.83−0.84−0.84−0.89
Per storm, over time−0.76−0.75−0.71−0.85
Fixed storm and lead time, across members−0.61−0.62−0.71no reference

What we find is that the relationship is still there within storms, so it does not come from pooling the storms together. We modified Section 5 and Appendix B.2 of the paper to better reflect the point you raised.

The result itself is important to the modelling of TC impacts. TC risk assessments rely mostly on large samples of statistically generated synthetic tracks (e.g. STORM; Bloemendaal et al., 2020). These produce tracks and intensities. Wind is derived from pressure through an imposed empirical wind–pressure relationship and parametric profiles (e.g. Holland, 1980). Dependence with other hazards such as rainfall is typically imposed via copulas. NWP ensembles provide consistent multivariate fields, but at a cost that limits ensemble size. AI weather model ensembles provide a simultaneous and coherent modelling of both wind and pressure, at a cost compatible with very large ensembles. This is useful if the dependence structure is realistic, and that is not guaranteed for a model trained with a per-variable objective. Our results show here that the dependence structure is preserved for wind and pressure, although member-level coupling remains weaker than in IFS-ENS (table above), and other impact variables (such as precipitation) remain to be evaluated.

(a) Track filters

We checked this in two ways, and the rankings do not change. First, we varied the minimum number of members (3 to 40) and of lead times (1 to 20), and FGN beats IFS-ENS on 80 to 93% of storms in all 54 settings. Second, we scored all 53 storms of 2025, the excluded ones included, and FGN beats IFS-ENS on 86% of them. In 2021–2023, GenCast beats IFS-ENS in every setting, while FourCastNet 3's lead is small and rarely significant (Appendix B.6).

(b) Rapid intensification, wind thresholds, percentile masks

Rapid intensification. This is an important point, and we now have a section on it (Section 6, Appendix B.7). What we find is that the models rarely reproduce it. This is true of the AI models, but also of IFS-ENS and of HRES-fc0, the analysis the 2025 models are fine-tuned on, so we think it would be unrealistic to expect the AI models to learn it from this data. Once we correct the intensity bias, the AI models find it about as often as IFS-ENS (14% of the observed cases for FGN, 10% for GenCast, 9% for IFS-ENS).

Wind thresholds. We added the probability of exceeding 17, 25 and 33 m/s (Appendix B.9). FGN is the best at 17 m/s, and IFS-ENS at 25 and 33 m/s, because the AI models rarely reach these winds.

Percentile masks. They remove the intensity bias on purpose: we wanted to measure where the strong winds are separately from how strong they are. We added the version with fixed thresholds next to it. FGN's lead holds at 17 m/s, disappears at 25 m/s and is reversed at 33 m/s.

Overall, our results show that AI weather models locate strong winds well, but they underestimate their intensity. Their training targets are themselves too weak in strong storms, which is consistent with this bias and with the rapid-intensification results, although we cannot attribute them to the data alone. For impact applications the intensity of the raw AI ensembles needs calibration before use. Here the preserved wind–pressure relationship discussed above matters. Because the dependence between the two variables is largely kept, a calibration can be applied to both of them consistently.

(c) Observed region or predicted centre

We recomputed the peak-wind bias around each member's own predicted centre, and it is nearly the same: in 2025 the bias of FGN and GenCast is about 0.6 m/s less negative, and at 64 kt and above it does not change. So position errors explain little of the weak winds (Appendix B.6). We kept the observed region in the main text because it does not depend on a tracker, and it allows us to score every member, including those whose tracks were not extracted.

(d) Significance tests and sensitivity

We added confidence intervals over storms and paired tests to the main comparisons (Appendix B.6). The main results are significant: in 2025, FGN beats IFS-ENS on tracks on 90% of storms [81, 98]. Some comparisons are not, for example FGN against GenCast on tracks, and we say it in the text. For the sensitivity, we varied the KDE bandwidth, the matching to IBTrACS and the tracker settings, and FGN beats IFS-ENS on 86 to 93% of storms in every case.

(e) Periods without cyclones

Indeed, we evaluate how well the models forecast cyclones that happened, not whether they produce cyclones that did not happen. We think it is a very interesting question for future work. We chose not to cover it in this paper because it is already quite technical, and we did not want to make it harder to follow. We added it to the conclusion as a limitation.

Reviewer_9aANOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper benchmarks three probabilistic AI weather models against IFS-ENS on tropical cyclones. It evaluates ERA5-trained GenCast and FCN3 on 125 storms from 2021–2023, then operational GenCast and FGN on 53 storms from 2025. The diagnostics cover detected tracks, NLL for observed positions, global and cyclone-region CRPS, peak-wind bias, ensemble calibration, spatial masks of strong winds and low pressures, and the wind–pressure relationship. In 2025, FGN has a lower conditional track NLL than IFS-ENS for about 90% of the 41 storms retained for that metric; the AI models also have better cyclone-region field scores and spatial placement, while retaining weak peak winds and calibration errors.

Critical strengths
  1. A valuable matched ensemble comparison. The study uses common storms, valid times, grid spacing, and similar ensemble sizes within each setting, and evaluates recent AI systems against an operational numerical ensemble. The 2025 paired storm results for FGN versus IFS-ENS are substantial, and the authors appropriately avoid treating the two cohorts as a controlled fine-tuning experiment.
  2. Diagnostics that expose a consequential trade-off. Separating track placement, field CRPS, peak winds, and calibration reveals that better probabilistic or spatial scores need not imply better cyclone intensity. This is more informative for meteorological use than a global average score alone, and the detailed tracking and scoring protocol helps readers assess the comparison.
Critical weaknesses
  1. The verification sample does not consistently isolate active or extreme cyclone conditions. Storms are selected by their lifetime maximum IBTrACS wind of at least 64 kt, while forecasts start near the first reported wind and run for 15 days. Thus the CRPS, peak-wind, and percentile-mask results can include pre-intensification and post-dissipation times. Results should be stratified by observed active-storm status and intensity, with sample counts by lead time. These results show better forecasts around the selected storms overall, but do not establish whether the gains persist when the storms actually reach at least 64 kt.

  2. The wind–pressure result may be driven by differences between storms. Figure 6 mixes data from many storms and forecast times. Strong storms generally have higher winds and lower pressure than weak storms, so the combined data can show a strong relationship even if the model does not capture how wind and pressure change within each storm. The authors should check this relationship storm by storm before drawing that conclusion.

Questions for authors
  1. Performance during strong storms: Do the model rankings for CRPS, peak-wind error, and spatial placement stay the same when you evaluate only times when the observed storm is active? What if you include only times when its IBTrACS wind reaches 64 kt? Please report how many cases remain at each forecast lead time, and make clear that gridded winds and IBTrACS sustained winds have different definitions.

  2. Missed cyclone tracks: FGN beats IFS-ENS on track NLL for most of the 41 storms that qualify for this score. Does it still lead when missed detections and the 12 excluded storms are taken into account? Does the result change if you vary the minimum number of detected members, required forecast times, or KDE bandwidth?

  3. Wind and pressure within each storm: Does the model reproduce the wind–pressure relationship for individual storms, when compared with HRES-fc0 at the same times?

  4. Comparison with earlier studies: Some studies find NWP models perform better than AI models for extreme events or certain cyclone forecasts, while this benchmark finds an AI advantage in tracking and spatial scores. Could you briefly explain which differences in the models, storm samples, reference data, or metrics—especially track versus intensity—are most likely to explain these results?

3 comments

Thank you for your review and for your positive assessment of the paper. Stratifying the results by storm state, as you suggested, showed that the smaller intensity bias of the AI models at long lead times comes from times when the storm has dissipated, and this is now in the paper (Appendix B.6.2). Your question on earlier studies and extreme events also led us to add a case study of a very atypical cyclone of 2025, Cyclone Alfred (Appendix B.11).

Your two main concerns were about soundness, and the results hold for both.

1. Active and extreme conditions. You pointed out that the scores include times before intensification and after dissipation, and asked for results stratified by storm state and intensity. We recomputed every score at active times, and at 34 kt and 64 kt and above. The gains persist, and they are larger at 64 kt: the cyclone-local CRPS gain of FGN over IFS-ENS is 20%, against 12% over all times, and the rankings on spatial placement are unchanged. The peak-wind bias of the AI models remains larger than that of IFS-ENS (at 64 kt and above, −6.4 m/s for FGN against −3.1 m/s for IFS-ENS). See Appendix B.6.2 and Figures 23 to 25.

2. Wind and pressure within storms. You pointed out that pooling the storms could by itself create the relationship of Figure 6. We recomputed it within storms: with the storm means removed, the rank correlation is unchanged (−0.83 for FGN, −0.84 for GenCast). See Appendix B.2.

The details are below, with our answers to your other questions: missed tracks and excluded storms, filters and KDE bandwidth, and the comparison with earlier studies.

If one of your points is still open, tell us which one and we will work on it during the discussion period. Otherwise, we would be grateful if you could update your review in light of these results, in whichever way you see fit.

Performance during strong storms (first weakness, first question)

Results for 2025, against HRES-fc0, over all lead times, when we keep only the times when the observed storm is active, or when its IBTrACS wind reaches 34 kt or 64 kt:

All timesActive34 kt and above64 kt and above
Cyclone-local CRPS, gain over IFS-ENS: FGN12%16%17%20%
Cyclone-local CRPS, gain over IFS-ENS: GenCast7%10%10%12%
Wind footprint overlap (IoU): FGN / GenCast / IFS-ENS0.27 / 0.25 / 0.190.39 / 0.36 / 0.270.40 / 0.37 / 0.270.41 / 0.38 / 0.27
Peak-wind bias (m/s): FGN / GenCast / IFS-ENS−2.3 / −2.1 / 0.0−4.0 / −3.7 / −1.0−4.3 / −4.1 / −1.3−6.4 / −6.2 / −3.1

The rankings are the same in every column, and all the CRPS gains are significant. By lead time, the intensity bias of the AI models no longer gets smaller at long leads once we keep only the active times: in the submission, this came from times when the storm had dissipated (Figure 23).

Sample counts by lead time (number of storms):

Lead time48 h96 h144 h192 h240 h
Active5248342317
34 kt and above4943311913
64 kt and above18221995

On the wind definitions: IBTrACS gives the maximum sustained wind estimated by the agencies, and the gridded wind is the 10 m wind on a 0.25° grid, so the two do not have the same magnitude. We use IBTrACS only to select the times, and all the scores are computed against the gridded reference (Appendix B.6.2). The results per lead time are in Figure 23 for 2025 and Figure 24 for 2021–2023, and the spatial scores in Figure 25.

Missed cyclone tracks

FGN still leads in every case.

  • Missed detections and the excluded storms. On all 53 storms of 2025, the excluded ones included, FGN beats IFS-ENS on 86% of them, against 90% on the retained storms (Appendix B.6.4, Figure 30).
  • Minimum number of detected members and required forecast times. With 3 to 40 members and 1 to 20 forecast times, FGN beats IFS-ENS on 80 to 93% of storms in all 54 settings (Figure 27).
  • KDE bandwidth. With a fixed bandwidth from 28 to 445 km, or with Scott's rule, FGN beats IFS-ENS on 86 to 93% of storms (Figure 29).

In 2021–2023, GenCast beats IFS-ENS in every setting too (Figure 28).

Wind and pressure within each storm (second weakness, third question)

Rank correlations in 2025, with HRES-fc0 at the same times:

LevelFGNGenCastIFS-ENSHRES-fc0
All storms pooled, as in Figure 6−0.83−0.84−0.87−0.88
Within storms (storm means removed)−0.83−0.84−0.84−0.89
Per storm, over time−0.76−0.75−0.71−0.85

The first two rows show that the correlation is the same once the differences between storms are removed. The third row is the storm-by-storm test you asked for: there the AI models are at the level of IFS-ENS (Appendix B.2).

Comparison with earlier studies

The studies that find numerical models better look at intensity, or at record-breaking events, and our results do not contradict them (Section 7). DeMaria et al. (2024), Sahu et al. (2025) and Gomez et al. (2026) find that deterministic AI models underestimate cyclone intensity. These models are trained to minimise the mean error, which averages away the strongest winds. We wanted to see whether this still holds for the newer probabilistic models, which are not trained this way. It does, but less: their peak winds are weaker than those of IFS-ENS, with about half the bias of the deterministic GraphCast on the same storms (−2.3 and −2.1 m/s for an FGN or GenCast member, against −5.6 m/s, over all forecast times; Appendix B.8).

The advantage we find for the AI models is on where the storm is, not on how strong it is: our track score and our spatial scores measure the position of the storm and of its strongest winds, separately from their intensity (Section 4). The metrics differ too. Earlier studies mostly use deterministic metrics: the position error of a single or mean track, and pointwise errors such as the RMSE. These are suited to a single forecast. For an ensemble, a mean track can lie where no member is, and it says nothing of the spread or of the different paths the storm may take. A pointwise error also favours a smooth forecast over a sharp one that is slightly displaced, which is the smoothing that weakens the storms. We therefore score the full distribution of the members (Section 3).

Zhang et al. (2026) and Sun et al. (2025) study events stronger than those in the training data: Zhang et al. look at local record-breaking heat, cold and wind, and Sun et al. remove the strongest cyclones (Category 3 to 5) from the training set. Both find that the AI models underestimate these events. This is a different question from ours. What we can expect a model to capture is a rare combination of things it has seen. This is what the case study of Cyclone Alfred, which we added following your question, looks at (Appendix B.11). Its track is the least typical of 2025, and it ends with the first cyclone to reach this part of the east coast of Australia in about 50 years. The AI models still place it better than IFS-ENS for the first ten days, and a group of GenCast members reach the right part of the coast.

Thank you for having reviewed our paper. In particular your question on stratifying the results by storm state has helped us make the benchmark more comprehensive. Please let us know if the new results in Appendices B.6.2 and B.2 address your two soundness concern? If either remains open, we would be glad to discuss about it in the remaining time.

Reviewer_XgoyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors evaluate tropical cyclone forecasts from three released probabilistic AI weather models (GenCast, FourCastNet 3, and FGN) against ECMWF’s IFS-ENS using historical TCs. They examine TC location, regional atmospheric fields, maximum-wind bias, and pooled wind–pressure dependence. No controlled training-data exclusions are performed, and the evaluated storms are not shown to lie substantially outside the training distribution. Consequently, the study does not establish extrapolation to unseen intensity regimes. Its main finding is improved cyclone-location and regional-field scores and persistent underprediction of maximum winds.

Critical strengths

The empirical comparisons, using multiple diagnostics covering TC location, regional fields, maximum winds, calibration, and the wind–pressure relationship, are a very useful contribution. They provide a systematic evaluation of recent probabilistic models, extending earlier work that largely focused on deterministic models. The paper also clearly illustrates that improved aggregate scores, even when local to cyclone regions do not imply accurate prediction of maximum winds.

Critical weaknesses

The framing in the abstract and introduction is somewhat too generous to the emulators. In fact, they substantially underestimate maximum wind speeds, perhaps the quantity most directly related to the rare extremes motivating the study. The study does not address rapid intensification at all, although this may be one of the most difficult rare TC behaviors to predict. TC location is largely determined by large-scale steering flows, which are likely much better represented in the training data than extreme TC intensities. It is not surprising that they do better on tracks than intensities. It is important not to confuse accurate track prediction with the ability to predict rare events.

The suggestion that the emulators capture physical relationships could also be misleading. The observed association between high winds and low pressure may only reflect statistical correlations between these quantities in the training data. I would hesitate to characterize this as "learning a physical relationship," particularly given the broader and unresolved questions surrounding what AI weather emulators learn about physics. Moreover, the current analysis does not even establish that the models correctly capture wind–pressure dependence conditional on a particular storm and forecast lead time.

Questions for authors

Can you look at rapid insensification?

Can you look at the conditional wind-pressure dependence (thought I would still characterize this as learning a statistical rather than physical relashionship)?

I understand that you want to focus on stochastic emulators. However, the evidence that stochastic emulators outperform deterministic ones in predicting extreme events is still mixed. I would expect both to perform better on TC tracks than on intensity or rapid intensification. This makes a comparison between stochastic and deterministic models potentially more interesting. Have you evaluated deterministic models using diagnostics that apply to both (e.g. maximum-wind bias, spatial agreement, and wind–pressure relationship)? Alternatively, could you leverage comparable evaluations available in the literature? Such a comparison would help clarify what stochastic modeling adds to the prediction of extreme TCs.

6 comments

Thank you for the careful review. We have studied the three important points you raised, and we try to answer your points below. We think the paper's framing is much stronger now, and the benchmark covers intensity more thoroughly.

We added a section on rapid intensification, following your remark that it may be one of the most difficult rare behaviours to predict. You also raised an interesting question on deterministic models, since the evidence that stochastic models do better on extremes is still mixed. We therefore conducted a comparison with the operational GraphCast, on the diagnostics you suggested (maximum-wind bias, spatial agreement, wind–pressure relationship) as well as on the track likelihood. As you expected, rapid intensification is largely missing from all the models. For the deterministic model, we find that GraphCast is on par with IFS-ENS on the track likelihood, but that its winds are too weak, about twice as much as those of the probabilistic models.

We hope this answers your concerns and your questions. If something is still open for you, tell us and we will work on it during the discussion period. And if you feel we have addressed your points, we would be grateful if you could update your review in light of these results, in whichever way you see fit.

Framing

A lot of your concern was about the framing of the paper. We understand that it could be read as too generous to the AI models, which was not our intention. Our goal was an independent, large-scale analysis across all basins, to understand where and why these models fail. As you noted in your review, the probabilistic setting had been little studied: most earlier conclusions on AI models and cyclones come from deterministic models. To fix the framing, we have rewritten the abstract and the introduction.

  • Abstract, last sentence. Before: "AI models are capable of capturing physical relationships that remain predictive well beyond typical weather conditions". Now: "Overall, our findings improve our understanding of where current AI weather models still fail, yielding practical insights into how they can be improved."
  • Introduction. It now states both halves of the result with equal weight: the AI ensembles place cyclones better than IFS-ENS, and their storms are too weak.

Three things the paper now says explicitly:

  1. We make no extrapolation claim. What we claim on extremes is realistic location and joint structure in rare events, not extrapolation to unseen intensities. By "extreme" we meant rare in the data. We computed it: taking a disc of about 650 km (the distance our tracker uses to identify a cyclone) around every IBTrACS position of a tropical storm over 1980–2018, we find that cyclones cover about 0.2% of ERA5 grid points, so they are rare in the training data. As for impacts, where a TC goes matters as much as how strong it is. Damage will be incurred if the TC meets exposed population and assets. A low-intensity storm that makes landfall on a densely populated coast can cause far more damage than a major hurricane that stays offshore. Sandy (2012) was only a Category 1 equivalent, and it was post-tropical at landfall, yet it caused catastrophic surge flooding in the New York area. Tropical Storm Allison (2001) never reached hurricane strength, yet it caused major flooding in Houston. This is why our evaluation separates where strong winds occur from how strong they are. It is also why we consider skill in locating these rare events a central part of forecasting extremes, and not only skill in reaching extreme intensities, even if accurate prediction of intensities is crucial. Cyclone Alfred (2025), now analysed in Appendix B.11, illustrates both points: its track and its landfall near Brisbane have no comparable case in the training period, and it caused major damage after weakening to category 1 before landfall. Because these situations make up a small fraction of the training data, there is no guarantee that a model reproduces the joint structure of the variables there. We test that by focussing on the wind–pressure dependence structure. The models keep a realistic dependence between the two variables in a regime they rarely see.
  1. Tracks and maximum intensity are two different problems. Given how rare cyclones are in the data, models capture the tracks well. FGN has the better track score on 90% of storms [81, 98], and on 80 to 93% across the filter, bandwidth and tracker settings we tested. We also added a case study of Cyclone Alfred (2025), which has the least typical track of the year (Appendix B.11). The AI models place it better than IFS-ENS for the first ten days. Its turn to the west, eleven days ahead, is rare in the record, and most members of the three models miss it. Some members do reach the coast. For GenCast, 8 of 49 reach the right part of it, about three days early. For IFS-ENS, 28 of 48 have reached land by day 11, but several days early and 700 km too far north. Maximum intensity is harder, and we agree with you on this. The strongest winds are in the core of the storm, close to the center, and a 0.25° grid does not resolve it. So the data the models learn from is already too weak. The same goes for rapid intensification, which no model captures, IFS-ENS included (see Q1 below). The paper now makes this distinction clear.
  2. The wind–pressure result is an empirical relationship. We used "physical" because the relationship itself has a physical origin: the radial pressure gradient is balanced by the centrifugal and Coriolis forces, so a deeper central pressure goes with stronger winds (Chavas et al., 2017; empirical form in Knaff and Zehr, 2007) and we wanted to test whether the forecasts reproduce it, specifically FGN. Indeed, FGN is trained with a per-variable proper scoring rule and thus probabilistic dependencies across variables like pressure and wind are never enforced explicitly. We did not mean that the models have learned that mechanism: the submission already said that the result "does not imply that the models have learned an explicit physical mechanism" (ll. 431–433). But we agree that the wording suggested it, so we have removed "physical relationship" from the paper, and only speak of the empirical relationship between wind and pressure.

The new experiments also make the intensity weakness clearer than in the submission. With fixed wind thresholds in place of percentile masks, FGN's advantage on wind footprints holds at 17 m/s, disappears at 25 m/s and reverses at 33 m/s. In the 26 storms that reach 33 m/s in HRES-fc0, no FGN member reaches the storm's peak wind, against 36% of IFS-ENS members. Both are in Appendix B.9.

Q1. Rapid intensification

The second, and very important, point pertains to rapid intensification. We conducted an analysis of all the models, IFS-ENS included, and added it to the paper (Section 6, Appendix B.7).

We use the standard definition from Kaplan and DeMaria (2003): rapid intensification (RI) is a rise of the maximum wind of at least 30 kt in 24 hours. In 2025, IBTrACS has 62 such 24-hour windows, across 28 of our 53 storms. For each model we look at its own peak wind in the same windows, and we count how many windows show a 30 kt rise, and how many of the 62 observed ones it finds. A window counts as found only if the model rises by 30 kt in exactly the same 24-hour window: a model that intensifies 12 hours early or late gets no credit.

What we find is that none of the models reproduces RI when we take their winds as they are. This is true of the AI models, and it is also true of IFS-ENS and of HRES-fc0, the analysis the 2025 models are fine-tuned on. So we think it would be unrealistic to expect the AI models to learn RI from these fields.

Since we already know that all the models are too weak, we also look at this after quantile mapping, as we did for the wind–pressure analysis: each model's winds are mapped onto the IBTrACS wind distribution (fitted on the other storms), and then we apply the same 30 kt definition.

HRES-fc0ERA5IFS-ENSFGNGenCast
Raw winds
Windows with a 30 kt rise, per member (observed: 62)140140.20.4
Share of the 62 observed windows found5%0%5%0.1%0.1%
After quantile mapping
Windows with a 30 kt rise, per member (observed: 62)6041314235
Share of the 62 observed windows found23%13%9%14%10%
Same, allowing up to 24 hours early or late47%31%26%35%32%

After quantile mapping, FGN and GenCast do about as well as IFS-ENS, and slightly better for FGN. All of them, HRES-fc0 included, remain far from the observations.

The case study of Cyclone Alfred (Appendix B.11) shows this on one storm. On 26–27 February, IBTrACS rises by 31 kt in 24 hours, and HRES-fc0 by 28 kt in the same window. No member of the three ensembles reproduces this rise at that time. Eight of the 48 IFS-ENS members do intensify rapidly, but mostly about two days too early, and no FGN or GenCast member does at any time of the forecast.

The fact that HRES-fc0 itself finds only 5% of RI windows with raw winds, and ERA5 none, shows that the main limitation is in the target data the models learn from, more than in the AI models themselves. What we see for IFS-ENS is in accordance with the literature. Torn and DeMaria (2021) find that the RI forecasts of the ECMWF ensemble are not skilful, even after a quantile-based bias correction, and Magnusson et al. (2021) report that the IFS intensifies too slowly. Since RI depends on inner-core processes at the kilometre scale, we do not expect any model trained on 0.25° fields to capture it.

Q2. Conditional wind–pressure dependence

We tested the dependence at the level you describe, and at three others. We also added three analyses to the paper (Appendix B.2): the wind at a given pressure, before and after matching the wind and pressure distributions to HRES-fc0 (Figure 11), the relationship storm by storm (Figure 12), and by storm category and basin (Figure 13).

The table gives the rank correlation between wind and pressure in 2025 at four levels, from all storms pooled down to a single storm and lead time.

LevelFGNGenCastIFS-ENSHRES-fc0
Pooled, as in the submission−0.83−0.84−0.87−0.88
Within storms (storm means removed)−0.83−0.84−0.84−0.89
Per storm, over time−0.76−0.75−0.71−0.85
Fixed storm and lead time, across members−0.61−0.62−0.71no reference
  • Pooling does not create the relationship. It is as strong once the storm means are removed.
  • Storm by storm, it is weaker than in the reference. The AI models are at the level of IFS-ENS, and all three are below HRES-fc0.
  • Across members at a fixed storm and lead time, which is the conditioning you describe, deeper members have stronger winds in at least 97.5% of cases. The relationship is weaker in the AI models than in IFS-ENS, with the caveat that IFS-ENS members spread about twice as much in wind.
  • At a given pressure, the AI winds are too weak in deep storms: 5 to 12 m/s below HRES-fc0 under 970 hPa. After mapping wind and pressure separately onto HRES-fc0's distributions, most of this gap disappears (below 960 hPa, FGN goes from −12.0 to −3.4 m/s). The gap is thus mostly in how strong the winds get, and less in how wind follows pressure.

In short, this is an empirical relationship between wind and pressure, present at every level we can test, and weaker than in the reference. It says nothing about what the models learn about physics. It does matter for impact modelling: most of the error is in the marginal intensity distribution, not in the dependence, so a calibration of wind and pressure removes most of the gap while keeping the joint structure.

Q3. Deterministic against stochastic

We conducted a battery of experiments using the operational GraphCast as an example of a deterministic model, because of its similarity to FGN and GenCast (same group, same resolution, same HRES initial conditions). We studied the tracks, the intensity bias, the spatial agreement and the wind–pressure relationship, on the 53 storms of 2025.

We chose to focus mostly on probabilistic models because, in the context of risk assessment, uncertainty is fundamental, and a single forecast is of limited use for assessing risk.

Tracks. Nevertheless, we can look at GraphCast's performance on cyclone tracks with a best-estimate analysis that uses the same KDE approach as the rest of the paper. GraphCast gives a single position, not 49 members, so its density is one kernel centred on that position. We choose the bandwidth that maximises the aggregate log-likelihood at each lead time, fitted on the other storms. This is the same score as in the paper, on the same storms and lead times for all four models, and it gives GraphCast the best bandwidth it can have.

GraphCastFGNGenCastIFS-ENS
Track NLL (lower is better)12.6611.9412.3812.82

GraphCast’s track density is worse than FGN’s (significantly) and GenCast’s (not significantly), and slightly better than IFS-ENS’s (not significantly).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment

We thank the three reviewers for their time and for their constructive reviews. The review process has been very helpful: it led us to broaden the evaluation, to stress-test the results, and to put them in a broader perspective.

The message of the revised paper is the following. The probabilistic AI weather models behave very similarly to each other on tropical cyclones, and differently from the physics-based ensemble, IFS-ENS: they are better at predicting where a cyclone goes (FGN has the better track score on 90% of the 2025 storms [81, 98]), and they underestimate its intensity. This has implications for model development. Part of the intensity bias comes from the training objective: on the same storms, the probabilistic models have about half the peak-wind bias of the deterministic GraphCast, while the relationship between wind and pressure remains consistent with the data. We also find that the intensity bias could be caused by the training data: the winds of ERA5 and of HRES-fc0 are themselves much weaker than the observed ones, and rapid intensification is largely absent from them, so further progress on intensity will require a more careful fine-tuning on HRES-fc0, which is closer to the observations than ERA5, or other data, rather than a different objective.

The reviewers found the benchmark useful ("a very useful contribution", Xgoy; "a valuable matched ensemble comparison", 9aAN). Their concerns were mostly about the framing of the results, the presentation, and analyses that were missing, and all three rated the soundness concerns as minor or fixable. We have rewritten the abstract and the introduction, and we have added a new section and nine analyses in the appendix.

We also performed a case study in order to go beyond average metrics, on Cyclone Alfred (2025), chosen for its atypical nature. Although cyclones in that region usually tend to move away from the coast, Alfred turned back towards it and made landfall near Brisbane, where no cyclone had done so in all of the models' training data. We have found that AI models still perform better on the tracking than IFS-ENS, and some GenCast members even reach the right part of the coast, although a couple of days early, without capturing any of the rapid intensification.

The additions are marked in blue in the PDF, and the table below lists them.

Raised byAdditionResultWhere
Xgoy, 1AyiRapid intensificationLargely missing from all the models, and also from HRES-fc0 and ERA5Section 6, Appendix B.7
XgoyComparison with a deterministic modelGraphCast is on par with IFS-ENS on tracks, with about twice the peak-wind bias of a probabilistic memberAppendix B.8
9aAN, 1AyiResults when the storm is strongAt 64 kt and above, the gains on the fields are larger (cyclone-local CRPS gain of FGN over IFS-ENS: 20%, against 12% over all times), and the ranking on spatial placement is unchangedAppendix B.6.2
9aAN, 1AyiMissed detections, filters, bandwidth, trackerFGN is ahead on 86% of all 53 storms, and on 80 to 93% across settingsAppendix B.6.4
1AyiFixed wind thresholdsFGN's lead on wind footprints holds at 17 m/s and is reversed at 33 m/sAppendix B.9
All threeWind and pressure within stormsThe rank correlation is unchanged once the differences between storms are removedAppendix B.2
Xgoy, 9aANCase study of a rare storm, Cyclone AlfredThe AI models place it better than IFS-ENS for the first ten daysAppendix B.11
Xgoy, 1AyiFraming and presentationThe abstract and the introduction state both sides of the result; each section starts with its main resultSections 1 to 6

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.