acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Dynamical Fidelity through Predictive Structure in Physical Representations

Abstract

Machine-learning models for physical systems are currently evaluated primarily through errors between predicted and reference states and, increasingly, through tests of physical consistency. These metrics assess whether predictions are accurate and satisfy selected physical requirements, but provide limited insight into whether learned trajectories reproduce the underlying dynamics. Domain experts examine such relationships through physical representations that expose relevant processes, interactions, and responses, but these analyses are often separated from typical machine-learning evaluation. We introduce a practical framework for evaluating dynamical fidelity through predictive structure in physical representation spaces. Experts define the representations, while reference trajectories determine which relationships are predictive and retained as evaluation tests. We demonstrate the approach in atmospheric forecasting using ERA5 representations of planetary-wave activity and Northern Annular Mode evolution, and evaluate Pangu-Weather, GraphCast, and FengWu. The models exhibit distinct departures from reference predictive structure that are not reflected by conventional forecast errors. The framework thereby turns domain-expert representations into systematic tests of learned physical dynamics without prescribing the relationships in advance.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
dynamical fidelity, model evaluation, physical consistency, process-oriented diagnostics, scientific machine learning, machine learning weather prediction, stratospheric dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y2b6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a way to check whether ML models of physical systems reproduce the system's dynamics, rather than only matching its state. A search over historical data (ERA5) finds conditions that predict how the system evolves next, and these conditions are fixed and used as tests. The authors apply this to a stratospheric circulation index namely the NAM. They identify 672 groups of winter days and compare three pretrained weather models on how well their forecasts match the observed mean and spread within each group. They find that the models rank differently on these tests than on standard forecast error.

I am reviewing this paper from an ML evaluation perspective and am not in a position to judge the atmospheric-physics choices in depth.

Critical strengths

The question is a fairly good one. Low forecast error does not guarantee that a model has learned the right behavior, so evaluations that look beyond average error are valuable. The search procedure is careful: the chance-level comparison reruns the whole search on shuffled data, resampling keeps whole winters together, and the tests are fixed before any model is evaluated.

Critical weaknesses
  • The test groups are selected and evaluated on the same data. The groups are chosen because ERA5's following days were unusual on them, and ERA5's outcome on the same days then serves as the reference the models must match. When cases are selected for looking extreme, part of the selected signal is chance, and that part will not reappear in a forecast. Even a perfect forecast model would therefore appear to miss on these groups, so some of the reported model error may come from the selection itself. I would suggest selecting the groups on one set of winters and evaluating the models on the others.
  • The main claim is compared against a weak baseline. The abstract states that the differences between models are not reflected by conventional forecast errors. However, the main text compares against error averaged over the whole globe and all variables, which has little to do with the index studied and hardly changes between groups (Table 1). The natural baseline is the forecast error on the index itself, on the same groups of days. This comparison should be the main one in the paper, and the claim should be stated in terms of it.
  • The 672 tests are much less independent than the number suggests. The groups all come from the same 772 days and overlap heavily. When near-duplicates are merged into 287 groups (Table 2), FengWu's advantage over GraphCast almost vanishes (0.169 vs. 0.171). Because the paper's main message rests on these group-level differences, they need uncertainty estimates, for example by resampling winters.
  • There is no comparison with simpler tests. The central idea is that tests discovered from data are more informative than tests chosen by hand, but the paper does not compare the two. It would be useful to see whether a few standard conditions would rank the models differently.
  • In terms of presentation, the mathematical part adds length but isn't used. Section 2 introduces several formal objects, including an "identifiability radius" that never appear again in the method or experiments. The point they make is intuitive. I suggest shortening this to a paragraph and using the space for extra experiments.
Questions for authors
  • How do the results change if the groups are selected on some winters and the models evaluated on the remaining ones?
  • Compared with forecast error on the index itself, which group-level differences remain, and do they hold up under resampling of winters?
  • How do the results compare with a small set of hand-chosen tests?
2 comments
AuthorsOpenReview ↗

Response to reviewer y2b6 (1/2)

We thank the reviewer for a careful and concrete review. The comments have helped us substantially improve the soundness of the claims, and we have added several new results.

W1 and Q1: Selection and evaluation on the same data

We thank the reviewer for this point and agree. Since each threshold is chosen to maximize the FVE on ERA5, the ERA5 leaf statistics are in-sample, and the reference shift is somewhat inflated. As suggested, we evaluated on held-out winters (new Appendix F):

  • Cross-fitted thresholds. We keep the precursor variables and split directions of the 672 relationships, refit their thresholds on 70% of the winters, and score the models on both these training winters and the remaining 30%, over 1000 splits. On the held-out winters, Pangu-Weather has the lowest EμE_\mu in every subset of Table 1 and the lowest Eσ2E_{\sigma^2} in all but one (Table 3).
  • Full cross-validation. Since the precursor variables were still selected on all winters, we also reran the full discovery in a five-fold cross-validation over winters. Pangu-Weather has the lowest held-out EμE_\mu in all five folds and the lowest Eσ2E_{\sigma^2} in all but one (Table 4).

Updates of the manuscript: Section 4.3 states that the ERA5 leaf statistics are in-sample, the Conclusion lists the selection of the precursor variables on all winters as a limitation, and the held-out tests are reported in the new Appendix F.

W2 / Q2: Conventional forecast-error baseline

We thank the reviewer for this comment. We agree that errors averaged over the globe per variable are a simplified view, and have expanded them in the new Appendix G with the TT, uu and vv RMSE at each of the 13 pressure levels and lead days 1–10. They tell the same story as the global errors. Averaged over the 772 initializations, FengWu has the lowest error in 385 of these 390 scores (3 variables, 13 pressure levels and 10 lead days). Comparing the models on each initialization separately, FengWu has the lowest error in 95% of the 772×390772 \times 390 comparisons. By the metrics on which these models are usually compared, FengWu is therefore clearly the preferred model.

Our framework is meant to challenge this way of evaluating models, which says little about process-level fidelity. Physicists spend considerable effort establishing how a quantity such as the NAM should evolve given the recent history of other representations, and pointing out where models fail to reproduce it. Our tests are such conditions, found automatically in the reference, and on them a different model comes out as the best.

We agree that the forecast error on the NAM itself, on the same subsets, belongs in the comparison, and have added it to Table 1 (removing vv, which closely follows uu). As expected for an error on the same index, Pangu-Weather also has the lowest NAM error in 15 of the 16 rows. The main comparison of the paper remains the one between the conventional forecast errors and the conditional errors, because the conventional errors are the metrics by which these models are ranked in practice. The NAM error does not replace the conditional errors either, as on individual dates it rewards forecasts that are damped toward the average. What a faithful model should get right is how the NAM evolves on average after a precursor and how much it varies around that, which EμE_\mu and Eσ2E_{\sigma^2} test. FengWu, for example, has a lower NAM error than GraphCast in every row of Table 1, yet a higher EμE_\mu in several of them, most clearly for strong negative shifts (NAM error 0.46 against 0.59, EμE_\mu 0.33 against 0.27).

Updates of the manuscript: Table 1 now includes the NAM forecast error and no longer vv, the new Appendix G reports the TT, uu and vv RMSE by pressure level and lead time for the 772 initializations.

AuthorsOpenReview ↗

Response to reviewer y2b6 (2/2)

W3 / Q2: Overlapping tests and uncertainty

We thank the reviewer for this comment. We agree that the 672 tests are not independent, and we do not treat them as such. Neighbouring tests, for instance the same precursor averaged over 7 or 10 days, select overlapping but different initializations, and can test different aspects of the same relationship. This dependence is also why we group the tests: each subset of Table 1 collects all tests using the same representation, across averaging windows, levels and latitudes. We use it to ask whether a model responds better to conditions in some representations than in others, for example in zonal means compared with wave-1 EP fluxes. We agree that FengWu and GraphCast cannot be ranked overall, as their EμE_\mu nearly coincides with the motif weighting (0.169 against 0.171). Within subsets they do differ. FengWu has the significantly lower NAM error in every subset, whereas in EμE_\mu FengWu is significantly lower in eight subsets and GraphCast in three, and these three hold with the motif weighting. Section 6 now reports this.

The main message of the paper is the ranking on these tests, which differs from the ranking by conventional forecast errors (see W2). We agree, however, that Table 1 should report uncertainty. Since each subset collects tests of the same representation, we want the result to be stable across the tests of a subset. Table 1 now marks the lowest error in dark green and the second lowest in light green. The lowest is underlined where it is significantly lower than both other models, and the second lowest where it is significantly lower than the highest, in a paired tt-test over the tests of the subset (p<0.05p<0.05). Pangu-Weather's lowest error is significant for EμE_\mu in 15 of the 16 rows and for Eσ2E_{\sigma^2} in 12 out of the 16. We prefer this to resampling winters as the tests are defined by thresholds fitted on the full record, and resampling winters without refitting changes which initializations enter each leaf, so that the resampled tests are no longer the tests we discovered. Refitting the thresholds on part of the winters and evaluating on the others is what we do in W1, where the ranking holds on the held-out winters.

Updates of the manuscript: Table 1 marks the lowest and second-lowest error and underlines them where they are significant in a paired tt-test over the tests of the subset, and Section 6 reports the subset-level differences between FengWu and GraphCast with the same test.

W4 / Q3: Comparison with hand-chosen tests

We thank the reviewer for this comment. Our aim is not to compete with hand-chosen tests, and an expert can always design a specific test for a specific question. The goal is to automate the construction of such tests, so that this kind of fidelity evaluation becomes available without the knowledge of an atmospheric physicist, and can be applied wherever a catalogue of physical representations exists.

However, this comment made us realize that one of the results we are most satisfied with is not properly presented in the paper, namely how physically meaningful the discovered evaluation tests are. The highlighted maxima of Figure 1 all select initializations that follow a strong wave-1 pulse in the troposphere at 45–82°N, after which the NAM decreases by about one standard deviation. This is the classical route to vortex weakening, through upward-propagating wave activity, with the heat flux over 45–75°N as the classical precursor. The search finds this latitude band, a meaningful wavenumber and sign without being given any of them.

Updates of the manuscript: Section 5 relates the maxima of Figure 1 and two second-level examples to established precursors, and a new Appendix H compares retained relationships with the precursor literature.

W5: Mathematical exposition

We thank the reviewer for this comment and agree that the mathematical part can be tightened. As this weakness is related to the comments of Reviewer 1H2r, we address it together with our response to them, and will revise the notation consistently throughout the manuscript.

Reviewer_1H2rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes evaluating ML models of physical systems for "dynamical fidelity". Predictive relationships are discovered from reference trajectories within an expert-defined representation space, frozen, and used to test whether models reproduce them. Section 2 and Appendix A argue that physical-consistency checks (invariants, budgets) cannot establish correct dynamics and are complementary to process-level diagnostics PkP_k. The framework is applied to stratospheric wave–mean-flow dynamics. Precursors built from Eliassen–Palm flux, zonal-mean circulation, and past NAM are searched with winter-level stability filtering and permutation-FDR calibration. This yields 672 conditional relationships for the 10-day evolution of the 50–100 hPa NAM from near-neutral ERA5 initializations. GraphCast, FengWu, and Pangu-Weather are scored by conditional-mean and conditional-variance errors on the frozen subsets, and the authors report that these rankings differ from rankings by conventional TT/uu/vv MAE.

Critical strengths

The motivation is sound. State-space error and physical-consistency checks leave room for incorrect dynamics, and separating an expert-defined vocabulary from data-discovered relationships is a reasonable way to make process-level evaluation more systematic. The discovery pipeline is carefully designed: resampling is done at the winter level, the permutation null reruns the full search, and all relationships are frozen before any model is evaluated. The appendices are thorough and candid about limitations.

Critical weaknesses
  1. "Dynamical fidelity" is never defined. Formalizing dynamical fidelity is listed as a contribution, but no definition is given. The term is used in three different senses:
  • (a) equality of learned and reference dynamics, F^=F\hat F = F, which the paper acknowledges is untestable;
  • (b) agreement of a diagnostic Pk(xt:t+L)P_k(x_{t:t+L}) between a predicted and a reference trajectory from a common initial state (Section 2, Appendix A.3);
  • (c) agreement of conditional distributions p(Y∣X)p(Y \mid X) over a cohort of initializations, where XX is a precursor computed from ERA5 history. The experiments measure only (c). The paper states that each discovered relationship acts as a process diagnostic PkP_k in the sense of (b), but no link is established between (c) and either (b) or (a). In particular, the paper shows neither that F^=F\hat F = F implies zero measured error under the experimental conditions nor that the error decreases as F^\hat F approaches FF. The measured quantity also depends on the cohort, the representation space, the discovery procedure, the discrepancy SS, and the aggregation over relationships. It is therefore a property of a model relative to a test suite, which the paper concedes when it says dynamical fidelity "is not a single model property". This does not fit well with the title's suggestion of a property of the learned dynamics.
  1. The diagnostic actually used is never written down. The only fully specified diagnostic in the paper is the oscillation period in Appendix A.4. In the application, no retained relationship is reported explicitly. From Sections 4.2–4.3 and Appendix D, each retained relationship rr appears to define a frozen leaf
  • Domain. PkP_k is a function of one trajectory. PrP_r is a statistic over a set of trajectories. Proposition 1 and the rest of Appendix A, including the ordering of tests, are stated for single-trajectory comparisons from a common initial state and do not apply to PrP_r.
  • Dependence on the reference. Membership in LrL_r is determined by ERA5 histories up to 30 days before tct_c, including levels above the models' 50 hPa lid (the catalogue contains a 1–50 hPa NAM region). Only the outcomes come from the model. PrP_r applied to a model is therefore not the same diagnostic applied to the model's trajectories, and errors may reflect information the model cannot represent rather than incorrect dynamics. F^=F\hat F = F is not well posed when the model and reference state spaces differ.
  • Selection. The retained leaf is the one whose ERA5 days 6–10 mean deviates most from its parent. The diagnostic is thus chosen using the reference values it is later compared against, so even a perfect model would likely have nonzero error. No such floor is reported.

I would ask the authors to (i) state an explicit definition, for example that a model is (R,S,ε)(\mathcal{R}, S, \varepsilon)-faithful on an initial-state distribution μ\mu if S(pref(Yr∣Xr), pF^(Y^r∣Xr))≤εS\big(p_{\mathrm{ref}}(Y_r \mid X_r),\, p_{\hat F}(\hat Y_r \mid X_r)\big) \le \varepsilon for all r∈Rr \in \mathcal{R}; (ii) write down PrP_r and relate it formally to Section 2, or revise Appendix A to cover it; (iii) show necessity, i.e. that F^=F\hat F = F with full state access gives zero discrepancy up to a quantified finite-sample and selection floor, and state non-sufficiency; and (iv) report several retained relationships in full (sources, thresholds, leaf sizes, and a physical interpretation), so that readers can judge whether the discovered structure corresponds to recognizable wave–mean-flow processes. With these changes the method would be a necessary-condition test relative to a chosen suite, parallel to physical consistency relative to a constraint set JJ, which is arguably the most interesting conceptual point of the paper.

  1. Section 6 compares conditional NAM errors with global, unweighted, all-level TT/uu/vv MAE; the appropriate comparison is NAM forecast error on the same subsets (Appendix E.6), where Pangu-Weather is best on both measures. Since EμE_\mu is a stratified forecast bias, the contribution lies mainly in how the subsets are discovered, and connections to slice and subgroup discovery in ML evaluation should be discussed. Model differences in Table 1 are reported without confidence intervals; a paired bootstrap over winters is needed.
Questions for authors
  1. What is the precise definition of dynamical fidelity?
  2. Is the reconstruction of LrL_r and PrP_r above correct, and does "terminal leaf" in Figure 3 and Table 1 always mean the selected second-level child of Appendix D.3?
  3. How does PrP_r relate to PkP_k in Section 2, and does any part of Appendix A apply to it?
  4. What error would a near-perfect model achieve under your leaf-selection procedure? For example, ERA5-initialized IFS ensemble members, or discovery and evaluation on disjoint sets of winters.
  5. How do the results change when precursors are restricted to information within the models' vertical domain and input window?
2 comments
AuthorsOpenReview ↗

Response to reviewer 1H2r (1/2)

Response

We thank the reviewer for a very careful and constructive review that really helped us to strengthen Section 2.

W1 and Q1. Definition of dynamical fidelity

The reviewer is right that we used "dynamical fidelity" with three different meanings and never defined it.

We added Definition 1 to the manuscript, which states that a model is dynamically faithful if the dynamics it has learned are the same as those of the reference system, F^=F\widehat F=F, for every state. Definition 2 states when a model, which only maps one gridded state to the next, is consistent with such a continuous vector field.

We agree that dynamical fidelity cannot be tested directly, and the rest of Section 2 therefore asks what the tests we do have can tell us about it. We consider the three kinds of evaluation: state-space accuracy, physical consistency, and process-level diagnostics. For each of them we now prove the same two statements (Propositions 1 to 3). If a model is dynamically faithful, its error is exactly zero and a zero error on any finite set of tests does not imply that the model is dynamically faithful. Each evaluation is thus a necessary but not sufficient test, which is the reading the reviewer suggests at the end of W2.

In the original manuscript, the process-level error was defined from a single trajectory, while our experiments compare statistics over many initializations. A process-level diagnostic is now any quantity computed from the trajectories that start from a chosen set of initial times, where one trajectory is the special case of one initial time. The reviewer notes that we do not show that the error decreases as F^\widehat F approaches FF. This is correct, and we do not claim it. The size of the error does not say how far the model is from FF.

The reviewer also notes that what we measure is a property of a model relative to a set of tests, and suggests a definition of this kind, where a model is (R,S,ε)(R,S,\varepsilon)-faithful if its error is at most ε\varepsilon on every relationship in RR. We agree, and the reviewer's (R,S,ε)(R,S,\varepsilon) condition is exactly what we compute for each retained relationship. We do not use it as the definition of dynamical fidelity, since it would then change with every new set of tests, and instead keep dynamical fidelity as a property of the model alone, F^=F\widehat F=F, for which a zero error on the tests is a necessary condition.

Changes to manuscript

  • Section 2 now defines dynamical fidelity (Definition 1) and the consistency of a gridded model with a vector field (Definition 2).
  • Propositions 1 to 3 state that state-space accuracy, physical consistency and process-level agreement are each necessary but not sufficient for dynamical fidelity, with proofs in Appendix A.
  • Process-level diagnostics are defined on sets of trajectories, and Section 3.2 writes the diagnostic of our experiments in this form.
  • Section 6 no longer says that dynamical fidelity "is not a single model property". It now says this of process-level agreement.

W2 (Diagnostic), Q2 and Q3

The reviewer's reconstruction is correct (Q2) where each retained relationship consists of two threshold conditions before initialization, and the initializations that meet both form a frozen subset. "Terminal leaf" in Figure 3 and Table 1 always means the selected second-level child defined under "Split criterion and eligibility" (Appendix D.3 in the submitted version, now D.4).

Section 3.2 now writes the diagnostic down where it is the distribution of the NAM response over the initializations of this subset, compared between the model forecasts and ERA5, and in practice we compare its mean and variance. This is a process-level diagnostic over a set of trajectories as described under W1.

As requested, we now report eight retained relationships in full in a new table in Appendix H. For each we give both conditions with their thresholds, the number of initializations and winters, the ERA5 NAM response and the errors of the three models. Two of them are the examples of Section 5, and six are selected by their conditional FVE and not by how well they can be interpreted. The rest will be released together with the code.

Changes to manuscript

  • Section 3.2 writes down the diagnostic used in the experiments.
  • Appendix H has a new table of eight retained relationships in full, with a short reading of each.
AuthorsOpenReview ↗

Response to reviewer 1H2r (2/2)

W2 (Selection) and Q4. Error of a near-perfect model

The reviewer is right that each leaf is selected on the same ERA5 response that the models are later compared with, so the reference response is somewhat inflated. Reviewer y2b6 raised the same point, and we refer to our reply there for details. In short, the new Appendix F evaluates on held-out winters, both with thresholds refitted on 70% of the winters and with the whole discovery rerun in a five-fold cross-validation. The conditional-mean error on held-out winters is on average no larger than on the winters used for selection, and Pangu-Weather has the lowest EμE_\mu in every subset of Table 1 and in all five folds. We have not evaluated IFS ensemble members, but agree that it would be interesting to compare ML models with physics-based models under this evaluation framework.

Changes to manuscript

  • Section 4.3 states that the ERA5 leaf statistics are in-sample.
  • The new Appendix F reports both held-out evaluations.
  • The Conclusion lists the selection on the same winters as a limitation.

W2 (Dependence) and Q5

The reviewer is right that the subsets are selected from the ERA5 history, including levels above the models' top, while the model only produces the outcome. As we also wrote to Reviewer 2E4y, this is intentional. The idea is to identify initialization times at which the preceding physical evolution provides strong evidence about what should happen next, and to test whether a model initialized from those states reproduces the expected response. A discrepancy may therefore reflect either missing input information or error in the learned dynamics, which the present evaluation does not by itself distinguish. The end user cares mostly about whether the models' predictions agree with what we expect from physics, whereas addressing the cases where they do not is then a task for model developers. The Conclusion states this as a limitation, and Appendix A now has a remark on the case where the gridded state does not describe the system completely. A non-zero error is then partly due to the missing information and does not show that the model could have done better.

Changes to manuscript

  • Appendix A has a new remark on incomplete gridded states, with a pointer after Definition 2.

W3. Conventional errors, subgroup discovery, and uncertainty

We agree that the forecast error on the NAM itself, on the same subsets, belongs in the comparison, and have added it to Table 1. As expected for an error on the same index, Pangu-Weather also has the lowest NAM error in 15 of the 16 rows. We still keep the conventional TT and uu errors next to it, because these are the metrics by which the models are ranked in practice, and on them FengWu is clearly the preferred model. The NAM error also does not replace the conditional errors. FengWu has a lower NAM error than GraphCast in every row of Table 1, yet a higher EμE_\mu in several of them, most clearly for strong negative shifts (NAM error 0.46 against 0.59, EμE_\mu 0.33 against 0.27).

We thank the reviewer for the reference to subgroup and slice discovery and agree that it was missing. A new subsection of Appendix D now discusses it. Our search is a form of subgroup discovery. Subgroup discovery finds subsets of the data where a target behaves unusually, and describes each subset by conditions on a set of variables. In our case, both the target and the variables are expert-defined representations of the reference system, the target of its future and the variables of its past. Slice discovery is different. It finds subsets on which a given model performs poorly, described by input features or learned embeddings. Our subsets are found without any model, so all models are tested on the same subsets. Each subset is also described by conditions on physical quantities, so it can be interpreted physically.

On uncertainty, Table 1 now marks where the lowest error is significantly lower than both other models in a paired tt-test over the tests of each subset (p<0.05p<0.05), which holds for Pangu-Weather in 15 of the 16 rows for EμE_\mu and in 12 for Eσ2E_{\sigma^2}. Our reply to Reviewer y2b6 explains why we prefer this to a bootstrap over winters, and Appendix F shows that the ranking holds on held-out winters.

Changes to manuscript

  • Table 1 now includes the NAM forecast error on the same subsets, and marks the differences that are significant in a paired tt-test.
  • A new subsection of Appendix D relates the method to subgroup and slice discovery, with a pointer in Section 4.3.
  • Appendix F reports the evaluation on held-out winters.
Reviewer_2E4yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for evaluating ML models of physical systems using predictive relationships discovered within expert-defined physical representation spaces. The framework is instantiated for stratospheric circulation using ERA5 and applied to GraphCast, FengWu, and Pangu-Weather. The main empirical finding is that conditional NAM-response diagnostics reveal differences among the models that are not captured by conventional forecast errors.

Critical strengths

1. A useful and well-motivated evaluation framework.

The separation between expert specification of physically meaningful representations and data-driven discovery of predictive relationships is appealing and potentially useful beyond weather forecasting. Freezing the ERA5-discovered relationships before evaluating individual models is also an important design choice that avoids model-specific selection of favorable diagnostics.

2. Careful empirical implementation.

The relationship discovery is supported by repeated winter-level held-out stability checks, permutation-based calibration of the complete search procedure, and a sensitivity analysis for redundancy among retained relationships. The resulting case study provides meaningful evidence that conventional forecast accuracy alone can miss differences in physically structured forecast behavior.

Critical weaknesses

1. The headline "dynamical fidelity" interpretation appears stronger than what the empirical evaluation directly establishes.

This is my main concern.

At the framework level, the paper appears to motivate dynamical fidelity as preservation of a predictive relationship of the form

\quad\text{versus}\quad p_{\mathrm{model}}(Y \mid X).$$ However, this is not exactly what is evaluated in the atmospheric experiment. The precursor variables, thresholds, branch memberships, terminal leaves, and corresponding initialization subsets are all determined from ERA5 and then frozen. For a retained relationship $r$, the empirical evaluation can therefore be viewed as first defining a reference-based subset $$A_r = \\{t_c : X^{\mathrm{ERA5}}_{t_c} \text{ satisfies the retained conditions for } r\\},$$ and then comparing $$\mathcal{L}\\!\left(Y^{\mathrm{ERA5}} \mid t_c \in A_r\right) \quad\text{with}\quad \mathcal{L}\\!\left(Y^{\mathrm{model}} \mid t_c \in A_r\right).$$ This is a meaningful diagnostic: it asks whether a model reproduces the subsequent response on cases belonging to a physically meaningful predictive regime discovered in the reference system. However, it does not directly establish that the model itself preserves the precursor-response relationship, because the conditioning event is never evaluated using the model's own precursor trajectory $X^{\mathrm{model}}$. For example, a model could in principle reproduce the future NAM distribution on the ERA5-selected cases while representing the precursor process incorrectly or relying on a different predictive shortcut. Conversely, as the paper itself notes, some precursor variables summarize histories longer than those available to the forecast models, so disagreement can reflect unavailable conditioning information rather than incorrect subsequent dynamics. I therefore find the current experiments more directly supportive of **fidelity of the future response conditional on reference-discovered physical regimes** than of preservation of the predictive relationship or underlying dynamical structure itself. Importantly, I do not think this necessarily requires substantial new experiments. If the intended estimand is the reference-conditioned quantity above, then clarifying this interpretation and narrowing the corresponding claims would substantially address my concern. If the stronger predictive-relationship interpretation is intended, then the paper needs to explain why the current evaluation is sufficient to support it. The paper also does not externally validate the selected relationships on an independent period or reanalysis, but I view this primarily as a limitation on generality rather than a critical flaw. The extensive within-cohort stability and permutation analyses are reassuring, and I do not consider additional external-validation experiments necessary for my assessment.
Questions for authors

1. Please clarify precisely which notion of "dynamical fidelity" the proposed evaluation is intended to measure. This is the main issue that could change my assessment.

At the framework level, I understand the intended comparison as

\quad\text{versus}\quad p_{\mathrm{model}}(Y \mid X).$$ In the actual experiment, however, the conditioning structure appears to be fixed entirely from ERA5. More precisely, for each retained relationship $r$, the procedure defines $$A_r = \\{t_c : X^{\mathrm{ERA5}}_{t_c} \text{ satisfies the ERA5-discovered conditions for } r\\},$$ and evaluates $$\mathcal{L}\\!\left(Y^{\mathrm{ERA5}} \mid t_c \in A_r\right) \quad\text{versus}\quad \mathcal{L}\\!\left(Y^{\mathrm{model}} \mid t_c \in A_r\right).$$ These seem to represent two different notions: 1. **Reference-conditioned response fidelity:** the model reproduces the future response on physical regimes identified from the reference system. 2. **Predictive-relationship fidelity:** the model itself reproduces the conditional structure $$p_{\mathrm{model}}(Y \mid X) \approx p_{\mathrm{ref}}(Y \mid X),$$ with the conditioning variable also evaluated from the model's own trajectory. Which of these is the intended claim? If the intended notion is **(1)**, I would be largely satisfied if the paper stated this explicitly and revised claims such as "preservation of predictive structure/relationships" so that they consistently describe preservation of the **reference-conditioned dynamical response**. If the intended claim is **(2)**, please explain why evaluating $$\mathcal{L}(Y^{\mathrm{model}} \mid t_c \in A_r)$$ with $A_r$ defined using $X^{\mathrm{ERA5}}$, rather than evaluating the corresponding conditioning event using $X^{\mathrm{model}}$, is sufficient to establish this stronger statement. I am not requesting a broad set of additional experiments. A clear definition of the intended estimand and a consistent alignment between that estimand, the empirical procedure, and the paper's claims could substantially improve my assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

All responses posted

All responses posted

We have now replied to all three reviewers and uploaded the full revision of the manuscript. We thank the reviewers again for their thorough and constructive feedback, which has substantially improved the paper. We would be glad to discuss any remaining questions or concerns, and hope the reviewers will engage with us during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.