- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark of tropical cyclone forecasting based on several probabilistic DLWF models that are SOTA coming from industries. The evaluation is comprehensive, examining not only cyclone tracking, but also intensity biases, ensemble calibration, spatial agreement, and wind–pressure dependence. and so on. The results show that leading AI models substantially outperform IFS-ENS in cyclone tracking, while limitations in intensity and ensemble spread remain.
Critical strengths
-
The paper provides a comprehensive evaluation of recent probabilistic DLWF models across multiple evaluation settings and cyclone seasons. These findings may be insightful and helpful to the community for developing next-generation DLWF models for tropical cyclone prediction.
-
The wind–pressure analysis provides interesting findings that a physically plausible joint structure can emerge without being explicitly added or guided by the training process.
Critical weaknesses
-
First of all, the presentation of this paper should be substantially improved. This work sounds more like an empirical study, in which several observations, findings, conclusions, and criticisms are explored. However, the content is structured in a tedious way for me. It would be better to concisely present the key take-home messages at the beginning of each section, with reference to more detailed justifications.
-
Another key concern is that the physical interpretation is limited. Similar pooled wind–pressure dependence does not establish that the models have learned an explicit physical mechanism. The effects of storm-level heterogeneity and aggregation remain insufficiently explored.
-
Several comments regarding more detailed experimental designs from the perspective of physical plausibility and domain knowledge:
(a) Trajectory NLL only evaluates successfully detected and matched cyclones, retaining 106 of 125 storms in 2021–2023 and 41 of 53 in 2025. The impact of these filtering rules on model rankings remains unclear.
(b) Although the paper focuses on extreme events, rapid intensification and probabilistic forecasts of hazardous wind-speed thresholds are not directly evaluated. Spatial agreement based on percentile thresholds removes systematic intensity biases.
(c) Peak-wind bias is evaluated within regions enclosing observed tracks rather than predicted cyclone centers, so displacement errors may influence the estimated intensity biases.
(d) The main model comparisons lack paired confidence intervals or significance tests. Sensitivity to cyclone detection, matching, and filtering thresholds is not systematically investigated.
(e) The evaluation focuses on known tropical cyclones and excludes periods without observed cyclones. So, the generalizability of this evaluation is limited.
Questions for authors
Can you clarify my concerns about the weakness?
5 comments
On aggregation, we looked at the relationship at four levels (rank correlations, 2025):
| Level | FGN | GenCast | IFS-ENS | HRES-fc0 |
|---|---|---|---|---|
| All storms pooled, as in the submission | −0.83 | −0.84 | −0.87 | −0.88 |
| Within storms (storm means removed) | −0.83 | −0.84 | −0.84 | −0.89 |
| Per storm, over time | −0.76 | −0.75 | −0.71 | −0.85 |
| Fixed storm and lead time, across members | −0.61 | −0.62 | −0.71 | no reference |
What we find is that the relationship is still there within storms, so it does not come from pooling the storms together. We modified Section 5 and Appendix B.2 of the paper to better reflect the point you raised.
The result itself is important to the modelling of TC impacts. TC risk assessments rely mostly on large samples of statistically generated synthetic tracks (e.g. STORM; Bloemendaal et al., 2020). These produce tracks and intensities. Wind is derived from pressure through an imposed empirical wind–pressure relationship and parametric profiles (e.g. Holland, 1980). Dependence with other hazards such as rainfall is typically imposed via copulas. NWP ensembles provide consistent multivariate fields, but at a cost that limits ensemble size. AI weather model ensembles provide a simultaneous and coherent modelling of both wind and pressure, at a cost compatible with very large ensembles. This is useful if the dependence structure is realistic, and that is not guaranteed for a model trained with a per-variable objective. Our results show here that the dependence structure is preserved for wind and pressure, although member-level coupling remains weaker than in IFS-ENS (table above), and other impact variables (such as precipitation) remain to be evaluated.
(a) Track filters
We checked this in two ways, and the rankings do not change. First, we varied the minimum number of members (3 to 40) and of lead times (1 to 20), and FGN beats IFS-ENS on 80 to 93% of storms in all 54 settings. Second, we scored all 53 storms of 2025, the excluded ones included, and FGN beats IFS-ENS on 86% of them. In 2021–2023, GenCast beats IFS-ENS in every setting, while FourCastNet 3's lead is small and rarely significant (Appendix B.6).
(b) Rapid intensification, wind thresholds, percentile masks
Rapid intensification. This is an important point, and we now have a section on it (Section 6, Appendix B.7). What we find is that the models rarely reproduce it. This is true of the AI models, but also of IFS-ENS and of HRES-fc0, the analysis the 2025 models are fine-tuned on, so we think it would be unrealistic to expect the AI models to learn it from this data. Once we correct the intensity bias, the AI models find it about as often as IFS-ENS (14% of the observed cases for FGN, 10% for GenCast, 9% for IFS-ENS).
Wind thresholds. We added the probability of exceeding 17, 25 and 33 m/s (Appendix B.9). FGN is the best at 17 m/s, and IFS-ENS at 25 and 33 m/s, because the AI models rarely reach these winds.
Percentile masks. They remove the intensity bias on purpose: we wanted to measure where the strong winds are separately from how strong they are. We added the version with fixed thresholds next to it. FGN's lead holds at 17 m/s, disappears at 25 m/s and is reversed at 33 m/s.
Overall, our results show that AI weather models locate strong winds well, but they underestimate their intensity. Their training targets are themselves too weak in strong storms, which is consistent with this bias and with the rapid-intensification results, although we cannot attribute them to the data alone. For impact applications the intensity of the raw AI ensembles needs calibration before use. Here the preserved wind–pressure relationship discussed above matters. Because the dependence between the two variables is largely kept, a calibration can be applied to both of them consistently.
(c) Observed region or predicted centre
We recomputed the peak-wind bias around each member's own predicted centre, and it is nearly the same: in 2025 the bias of FGN and GenCast is about 0.6 m/s less negative, and at 64 kt and above it does not change. So position errors explain little of the weak winds (Appendix B.6). We kept the observed region in the main text because it does not depend on a tracker, and it allows us to score every member, including those whose tracks were not extracted.
(d) Significance tests and sensitivity
We added confidence intervals over storms and paired tests to the main comparisons (Appendix B.6). The main results are significant: in 2025, FGN beats IFS-ENS on tracks on 90% of storms [81, 98]. Some comparisons are not, for example FGN against GenCast on tracks, and we say it in the text. For the sensitivity, we varied the KDE bandwidth, the matching to IBTrACS and the tracker settings, and FGN beats IFS-ENS on 86 to 93% of storms in every case.
(e) Periods without cyclones
Indeed, we evaluate how well the models forecast cyclones that happened, not whether they produce cyclones that did not happen. We think it is a very interesting question for future work. We chose not to cover it in this paper because it is already quite technical, and we did not want to make it harder to follow. We added it to the conclusion as a limitation.
Thank you for your review. We restructured the paper the way you suggested, and we ran a new analysis for each of the points you raised:
We hope this answers the weaknesses you listed. If one of them is still open for you, tell us which one and we will work on it during the discussion period. And if you feel we have addressed your points, we would be happy if you could reflect it in your rating.
We start with the presentation, which was your main concern, and then go through your other points one by one.
Presentation
We followed your suggestion: each results section now starts with its key message, in italics under the heading, and the details come after. Each message answers one of the research questions that the introduction now lists. These are the messages:
We also added a case study of one storm, Cyclone Alfred, which has the least typical track of 2025 (Appendix B.11). It shows the main results on a single forecast: the AI models stay closer to the observed track than IFS-ENS for the first ten days, their winds are weaker, and no member of any model reproduces its rapid intensification at the right time.
Following your comment on next-generation models, we also added to the conclusion the directions we think matter most for future work. The tracks are already good, so the effort should go to intensity. The training data is itself too weak in the core of the storm, so this will probably need other data, and not only another loss.
Physical interpretation and aggregation
We used "physical" because the relationship itself has a physical origin. In the core of the storm, the radial pressure gradient is balanced by the centrifugal and Coriolis forces, so a deeper centre pressure is balanced by stronger winds (Chavas et al., 2017, and its empirical form in Knaff and Zehr, 2007). We wanted to test whether the forecasts reproduce it. As you say, and as we noted in the submission (ll. 431–433), this does not imply that the models have learned an explicit physical mechanism. But we see that the wording suggested it, so we removed "physical relationship" from the paper, and we only speak of the empirical relationship between wind and pressure.
The finding itself still holds: FGN is trained with an objective that scores each variable separately, and its forecasts still have a wind–pressure relationship close to the one in the reference.