- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a way to check whether ML models of physical systems reproduce the system's dynamics, rather than only matching its state. A search over historical data (ERA5) finds conditions that predict how the system evolves next, and these conditions are fixed and used as tests. The authors apply this to a stratospheric circulation index namely the NAM. They identify 672 groups of winter days and compare three pretrained weather models on how well their forecasts match the observed mean and spread within each group. They find that the models rank differently on these tests than on standard forecast error.
I am reviewing this paper from an ML evaluation perspective and am not in a position to judge the atmospheric-physics choices in depth.
Critical strengths
The question is a fairly good one. Low forecast error does not guarantee that a model has learned the right behavior, so evaluations that look beyond average error are valuable. The search procedure is careful: the chance-level comparison reruns the whole search on shuffled data, resampling keeps whole winters together, and the tests are fixed before any model is evaluated.
Critical weaknesses
- The test groups are selected and evaluated on the same data. The groups are chosen because ERA5's following days were unusual on them, and ERA5's outcome on the same days then serves as the reference the models must match. When cases are selected for looking extreme, part of the selected signal is chance, and that part will not reappear in a forecast. Even a perfect forecast model would therefore appear to miss on these groups, so some of the reported model error may come from the selection itself. I would suggest selecting the groups on one set of winters and evaluating the models on the others.
- The main claim is compared against a weak baseline. The abstract states that the differences between models are not reflected by conventional forecast errors. However, the main text compares against error averaged over the whole globe and all variables, which has little to do with the index studied and hardly changes between groups (Table 1). The natural baseline is the forecast error on the index itself, on the same groups of days. This comparison should be the main one in the paper, and the claim should be stated in terms of it.
- The 672 tests are much less independent than the number suggests. The groups all come from the same 772 days and overlap heavily. When near-duplicates are merged into 287 groups (Table 2), FengWu's advantage over GraphCast almost vanishes (0.169 vs. 0.171). Because the paper's main message rests on these group-level differences, they need uncertainty estimates, for example by resampling winters.
- There is no comparison with simpler tests. The central idea is that tests discovered from data are more informative than tests chosen by hand, but the paper does not compare the two. It would be useful to see whether a few standard conditions would rank the models differently.
- In terms of presentation, the mathematical part adds length but isn't used. Section 2 introduces several formal objects, including an "identifiability radius" that never appear again in the method or experiments. The point they make is intuitive. I suggest shortening this to a paragraph and using the space for extra experiments.
Questions for authors
- How do the results change if the groups are selected on some winters and the models evaluated on the remaining ones?
- Compared with forecast error on the index itself, which group-level differences remain, and do they hold up under resampling of winters?
- How do the results compare with a small set of hand-chosen tests?
2 comments
Response to reviewer y2b6 (2/2)
W3 / Q2: Overlapping tests and uncertainty
We thank the reviewer for this comment. We agree that the 672 tests are not independent, and we do not treat them as such. Neighbouring tests, for instance the same precursor averaged over 7 or 10 days, select overlapping but different initializations, and can test different aspects of the same relationship. This dependence is also why we group the tests: each subset of Table 1 collects all tests using the same representation, across averaging windows, levels and latitudes. We use it to ask whether a model responds better to conditions in some representations than in others, for example in zonal means compared with wave-1 EP fluxes. We agree that FengWu and GraphCast cannot be ranked overall, as their nearly coincides with the motif weighting (0.169 against 0.171). Within subsets they do differ. FengWu has the significantly lower NAM error in every subset, whereas in FengWu is significantly lower in eight subsets and GraphCast in three, and these three hold with the motif weighting. Section 6 now reports this.
The main message of the paper is the ranking on these tests, which differs from the ranking by conventional forecast errors (see W2). We agree, however, that Table 1 should report uncertainty. Since each subset collects tests of the same representation, we want the result to be stable across the tests of a subset. Table 1 now marks the lowest error in dark green and the second lowest in light green. The lowest is underlined where it is significantly lower than both other models, and the second lowest where it is significantly lower than the highest, in a paired -test over the tests of the subset (). Pangu-Weather's lowest error is significant for in 15 of the 16 rows and for in 12 out of the 16. We prefer this to resampling winters as the tests are defined by thresholds fitted on the full record, and resampling winters without refitting changes which initializations enter each leaf, so that the resampled tests are no longer the tests we discovered. Refitting the thresholds on part of the winters and evaluating on the others is what we do in W1, where the ranking holds on the held-out winters.
Updates of the manuscript: Table 1 marks the lowest and second-lowest error and underlines them where they are significant in a paired -test over the tests of the subset, and Section 6 reports the subset-level differences between FengWu and GraphCast with the same test.
W4 / Q3: Comparison with hand-chosen tests
We thank the reviewer for this comment. Our aim is not to compete with hand-chosen tests, and an expert can always design a specific test for a specific question. The goal is to automate the construction of such tests, so that this kind of fidelity evaluation becomes available without the knowledge of an atmospheric physicist, and can be applied wherever a catalogue of physical representations exists.
However, this comment made us realize that one of the results we are most satisfied with is not properly presented in the paper, namely how physically meaningful the discovered evaluation tests are. The highlighted maxima of Figure 1 all select initializations that follow a strong wave-1 pulse in the troposphere at 45–82°N, after which the NAM decreases by about one standard deviation. This is the classical route to vortex weakening, through upward-propagating wave activity, with the heat flux over 45–75°N as the classical precursor. The search finds this latitude band, a meaningful wavenumber and sign without being given any of them.
Updates of the manuscript: Section 5 relates the maxima of Figure 1 and two second-level examples to established precursors, and a new Appendix H compares retained relationships with the precursor literature.
W5: Mathematical exposition
We thank the reviewer for this comment and agree that the mathematical part can be tightened. As this weakness is related to the comments of Reviewer 1H2r, we address it together with our response to them, and will revise the notation consistently throughout the manuscript.
Response to reviewer y2b6 (1/2)
We thank the reviewer for a careful and concrete review. The comments have helped us substantially improve the soundness of the claims, and we have added several new results.
W1 and Q1: Selection and evaluation on the same data
We thank the reviewer for this point and agree. Since each threshold is chosen to maximize the FVE on ERA5, the ERA5 leaf statistics are in-sample, and the reference shift is somewhat inflated. As suggested, we evaluated on held-out winters (new Appendix F):
Updates of the manuscript: Section 4.3 states that the ERA5 leaf statistics are in-sample, the Conclusion lists the selection of the precursor variables on all winters as a limitation, and the held-out tests are reported in the new Appendix F.
W2 / Q2: Conventional forecast-error baseline
We thank the reviewer for this comment. We agree that errors averaged over the globe per variable are a simplified view, and have expanded them in the new Appendix G with the T, u and v RMSE at each of the 13 pressure levels and lead days 1–10. They tell the same story as the global errors. Averaged over the 772 initializations, FengWu has the lowest error in 385 of these 390 scores (3 variables, 13 pressure levels and 10 lead days). Comparing the models on each initialization separately, FengWu has the lowest error in 95% of the 772×390 comparisons. By the metrics on which these models are usually compared, FengWu is therefore clearly the preferred model.
Our framework is meant to challenge this way of evaluating models, which says little about process-level fidelity. Physicists spend considerable effort establishing how a quantity such as the NAM should evolve given the recent history of other representations, and pointing out where models fail to reproduce it. Our tests are such conditions, found automatically in the reference, and on them a different model comes out as the best.
We agree that the forecast error on the NAM itself, on the same subsets, belongs in the comparison, and have added it to Table 1 (removing v, which closely follows u). As expected for an error on the same index, Pangu-Weather also has the lowest NAM error in 15 of the 16 rows. The main comparison of the paper remains the one between the conventional forecast errors and the conditional errors, because the conventional errors are the metrics by which these models are ranked in practice. The NAM error does not replace the conditional errors either, as on individual dates it rewards forecasts that are damped toward the average. What a faithful model should get right is how the NAM evolves on average after a precursor and how much it varies around that, which Eμ and Eσ2 test. FengWu, for example, has a lower NAM error than GraphCast in every row of Table 1, yet a higher Eμ in several of them, most clearly for strong negative shifts (NAM error 0.46 against 0.59, Eμ 0.33 against 0.27).
Updates of the manuscript: Table 1 now includes the NAM forecast error and no longer v, the new Appendix G reports the T, u and v RMSE by pressure level and lead time for the 772 initializations.