acceptodds
Under review as a conference paper at ICLR 2027

SciDynBench-R: Benchmarking Error-Validity Disagreement in Scientific Dynamical Forecasting

Abstract

Scientific dynamical forecasts are usually selected by aggregate point error, but scientific use often depends on regimes, events, oscillations, and constraints. We introduce SciDynBench-R, a benchmark that connects scientifically motivated distribution shifts with task-specific validity criteria across controlled ODEs, Palk inspired biological transport, and leakage safe reduced WeatherBench2 Z500 climate indices. Its reproducible protocols jointly evaluate point error and scientific validity and diagnose disagreement between their model rankings. Across 187 frozen baseline model/split means, SciDynBench-R finds 44 low error scientific failures: ODE forecasts in the wrong regime, biological forecasts with large calcium peak count errors, and climate forecasts selected by NRMSE that behave as event smoothers, missing high variability and amplification events despite low aggregate error. A climate ablation illustrates improvements in event fidelity at higher NRMSE among the evaluated configurations. The benchmark and frozen results provide a reference for examining when favorable point error coexists with failures of task-specific scientific properties.

Primary area:
datasets and benchmarks
Keywords:
Evaluation, Benchmarking, Datasets & Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x6kJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SciDynBench-R, a benchmark for evaluating disagreement between conventional forecast errors and scientific validity in dynamical systems. The benchmark covers three settings: four canonical ODE systems, a mechanistic biological transport model, and six reduced climate indices derived from WeatherBench2 Z500 data. It evaluates forecasting models under IID and distribution shifts using standard error metrics alongside regime preservation, oscillation fidelity, event detection, and physical constraint violations.

Across 187 model/split configurations, the authors identify 44 cases where low prediction error coincides with poor scientific validity. In particular, the climate experiments demonstrate that models optimized for NRMSE can suppress variability and miss extreme events. A supplementary experiment shows that improving event fidelity is possible, although at the expense of higher forecast error.

Critical strengths
  • Important evaluation problem. I agree with the motivation that low forecast error does not necessarily imply scientifically meaningful predictions. The distinction between trajectory accuracy and dynamical properties, particularly regime transitions and oscillations, is important for scientific machine learning. The benchmark provides a systematic way to evaluate this discrepancy across different dynamical systems.

  • Careful evaluation protocol across different settings. The inclusion of controlled ODE systems, mechanistic biological dynamics, and reduced climate observations provides complementary perspectives. I particularly appreciate the explicit treatment of distribution shifts and temporal leakage in the climate experiments, including disjoint training and testing years and training-only preprocessing.

Critical weaknesses
  • Scientific validity is largely defined through heuristic metrics. The benchmark treats regime preservation, peak counts, and event detection as indicators of scientific validity. While these are meaningful diagnostics, they do not necessarily establish physical consistency. For example, the climate evaluation relies on threshold-based labels and Z500-derived proxies rather than actual physical constraints or governing equation residuals. I think the distinction between scientific validity and task-specific forecast properties needs to be better justified.

  • Limited complexity and realism of the evaluated dynamical systems. Most experiments rely on relatively simple ODEs, while the biological system is a reduced, Palk-inspired model rather than a complete physiological representation. The climate component is also limited to six reduced Z500 indices instead of full atmospheric fields. It is unclear whether the observed disagreement between error and validity extends to higher-dimensional PDE systems or realistic weather prediction models. This limits the broader applicability of the proposed benchmark.

  • The baseline models are relatively limited. The experiments primarily compare conventional sequence forecasting architectures trained using MSE. Consequently, the observed failures may largely reflect the known tendency of point predictors to smooth oscillations and extreme events. The benchmark would benefit from including physics-informed, structure-preserving, or probabilistic forecasting methods. These would help establish whether the reported disagreement remains significant when models are explicitly designed to preserve dynamical properties.

  • Limited robustness analysis of the proposed validity metrics. The Metric Disagreement Index is based on Spearman rank correlation between error and validity rankings. However, these rankings involve a relatively small number of models, and the validity scores themselves depend on predefined thresholds, normalization, and weighting choices. Similarly, the 44 reported low-error scientific failures depend on family-specific criteria. I would like to see sensitivity analyses demonstrating that the main conclusions remain robust under alternative validity definitions and thresholds.

  • The climate ablation does not isolate the source of improved event fidelity. The fidelity-aware GRU changes the training objective and uses additional calibration, making it difficult to determine which component drives the improvement. For example, under temporal OOD, event fidelity improves substantially, but NRMSE increases from 0.906 to 1.134. The paper demonstrates a trade-off between the evaluated configurations but does not establish whether better model selection, calibration alone, or alternative objectives could improve both metrics. A more controlled ablation would strengthen the practical contribution of the benchmark.

Questions for authors

See above.

2 comments

Thank you for the detailed review.

W1: Validity defined by heuristic metrics. We agree that regime and event scores are not physical consistency, and the revision makes the distinction explicit (Section 3.5). In this paper, scientific validity is an umbrella term, defined per task, for the properties a scientist relies on when using the forecast for a given task, not physical consistency in the strict sense. We report three kinds separately and call only the second physical validity. (i) Task properties: regime labels, oscillation statistics and event scores, whose thresholds are scientific choices perturbed in E0. (ii) Physical consistency checks grounded in the governing equations: nonnegativity and population conservation (SIR), positivity (Lotka–Volterra), physiological bounds (biology), the exact mean invariant of KS and the residual of the known flow equations (Lorenz). (iii) Dynamical diagnostics: zonal energy spectra and FSS (E5), KS band power (E6).

The disagreement is not confined to the task properties. Among the physical checks, KS neural operators never violate the mean invariant in distribution, but violate it in 26–74% of windows under parameter extrapolation with no error penalty, and in 97–100% under regime shift (E6). SIR forecasts with low error have large constraint violations that a projection removes at almost no error cost (E10). Among the diagnostics, the Z500 forecast with the lowest RMSE keeps only 13% of the power at small scales at day 5. For climate, the reviewer is right that spectra, event F1 and FSS are diagnostics rather than constraints from the governing equations; the climate tier tests event fidelity, and the revision describes it that way.

W2: System complexity. The concern is whether the disagreement is specific to small ODE systems. It is not. On full 5.625° Z500 fields (E5; 1–20 days, WeatherBench 2 metrics, spectra and FSS), error and fidelity rankings reverse as well. In a spatiotemporally chaotic PDE (E6; Kuramoto–Sivashinsky with neural operators and physically defined shifts), three mechanisms separate: error over long horizons that rewards decay to zero, invariant violations with no error penalty, and regime shifts that defeat every model. Both are still research settings rather than operational weather models.

The biological model is a reduced mechanistic model on purpose. It is based on the salivary acinar cell model of Palk et al. (2010), with cytosolic and ER calcium, IP3, six ions and cell volume. Its regimes (sustained and damped calcium oscillations, baseline, secretion responses) and physiological constraints follow from known mechanisms, so the labels need no annotation and shifts can be set through physiological parameters. A complete physiological model would add realism but not this ground truth. The protocol itself does not depend on the simulator, which is how the KS task was added.

W3: Limited baselines. Besides the stronger forecasters (E4), probabilistic models (E2) and alternative objectives including a residual of the known equations, as in PINNs (E3), E9 adds models built to preserve structure: output layers with hard constraints, SINDy fitted per trajectory, and operators that conserve the mean for KS. They keep what they are built for. Constrained forecasts have zero violations, and Lotka–Volterra networks with a positivity constraint reach regime agreement 0.69–0.94 (trained unconstrained networks: 0.00–0.62), at 1.2–2.7× the error winner's NRMSE. For SIR, a simple post hoc projection already does as well (E10). SINDy is best or second best on regime agreement and amplitude in 7 of 8 Lotka–Volterra/Van der Pol groups, yet its mean NRMSE ranks anywhere from first to last, because a few identified models diverge. With these models in the pool, error and validity winners coincide in 3/26 groups. Under regime shift, enforcing an invariant is not enough: the KS operators that conserve the mean fail like the unconstrained ones.

We did not run a coordinate-based PINN. With known equations and parameters it acts as a solver rather than a forecaster, and its physics-informed loss is what our residual objective already applies to forecasters. Hamiltonian networks assume energy conservation, so they do not fit these dissipative systems.

W4: Robustness of MDI and thresholds. The question is whether the disagreement depends on particular thresholds, models or seeds. It does not, in the following sense (E0 and finding 2 in the general response). The pattern holds across 2,000 joint specifications and the perturbations of the label thresholds, and also with up to 13 models per group after adding E4 (same winner 4/26, negative correlation 12/26). Under a seed bootstrap and leave-one-model-out (Appendix H), the sign of the correlation is stable in 22 of 25 groups, 9 of the 10 negative groups have their whole 95% interval below zero, and six stay negative whichever single model is left out. What is less robust is the count of low error failures, which has a high base rate in biology and climate; the revision reports that base rate and treats the count as descriptive.

W5: The climate ablation does not isolate the source. Agreed: the original ablation changed the objective and the calibration together. E8 separates them. We swept the weight λ of the variance objective ∈ {0, 0.1, 0.3, 1, 3, 10} with and without validation calibration (5 seeds). No configuration improves both NRMSE and event scores, and NRMSE rises almost monotonically with λ. Calibration alone gives high variability F1 0.46 at NRMSE 1.09, the objective alone 0.40 at 1.14, and combining them raises amplification SEDI to 0.56 at the cost of many false alarms. Checkpoint selection by validation event score (E3) adds little in climate. Probabilistic members give the best event ranking (amplification ROC-AUC 0.77–0.81, vs 0.64–0.71 for deterministic forecasts scored on continuous event strength). The revision presents this as an observed tradeoff among the tested configurations, not as evidence that event fidelity requires higher error in general.

Reviewer_s6fhOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
1(−2)
Read review
Paper summary

The paper introduces SciDynBench-R, a benchmark for testing whether forecasts with low aggregate point error (MSE/NRMSE) also preserve the scientifically meaningful properties of a dynamical system. The standard protocol defines three evidence tiers. Standard forecasters are evaluated under scientific shifts then scored on both point error and scientific validity. Finally a ranking comparison is performed, highlighting 44 low error scientific failures across 187 configurations. A climate ablation complement the study, reporting that it is possible to better forecast F1 as the cost of higher NRMSE.

Critical strengths

The motivation is sound as model selection by aggregate error alone remains common, and standard protocols pairing error with validity across different domains would be useful. The paper provide strong materials and evidence for reproducibly. The study covers a broad and well-chosen varieties of baseline coming from various scientific domains.

Critical weaknesses

The conclusion is expected. MSE-trained deterministic forecasts converge to the conditional mean and thus smooth out oscillations and events under uncertainty, a well-known effect closely related to the double penalty in forecast verification. The protocol design goes beyond predictability limits. And the best climate candidate barely beats climatology. The paper omits forecast-verification theory and the literature on MSE-induced smoothing and the double penalty It also omits benchmarks that already address this issue: ClimateBench, ExtremeWeatherBench, ChaosBench and OceanBench. Well-known already established solutions to that issue are not evaluated: diagnostic variables (spectra, neighbourhood/event scores), ensemble forecasts, and the CRPS.

Questions for authors

A. Could you report skill scores relative to climatology and persistence for all tiers? For the climate tier in particular, how do you interpret baselines with NRMSE ≈ 0.9 and ablation variants with NRMSE > 1?

B. How do the results change if forecasts are restricted to horizons within the predictability limit, or if metrics are reported per lead time?

C. Could you add at least one probabilistic or ensemble baseline evaluated with the CRPS and spread–skill diagnostics, and report whether the error–validity disagreement persists?

D. How does SciDynBench-R relate to, and go beyond, the forecast-verification literature on the double penalty and existing benchmarks such as ClimateBench, ExtremeWeatherBench, ChaosBench and OceanBench?

2 comments

Thank you for a frank review, and for raising the ethics question openly.

Ethics flag. An earlier version of this work was submitted to AAAI-27 and rejected on September 24, 2026, before the ICLR full paper deadline and before the review period; it has not been under review anywhere since. The full timeline, including the abstract registration date, is in a confidential comment to the AC and Program Chairs. The leftover script and directory names in Appendix E have been renamed in the revised PDF.

"The conclusion is expected." For part of the benchmark, we agree. Forecasts trained with MSE approach the conditional mean, and the double penalty is classical (Mass et al., 2002; Ebert, 2008; Gilleland et al., 2009); WeatherBench 2 (Rasp et al., 2024), Bonavita (2024) and Price et al. (2025) attribute the smoothness of AI weather forecasts to MSE. The revision now cites this work (Section 2.4). It fits our climate tier well. Deterministic forecasts still rank events by their continuous strength (amplification ROC-AUC 0.64–0.71), but their damped amplitude never crosses the event thresholds (F1 = 0); probabilities from members rank events better (0.77–0.81, E2); and the full field Z500 forecast with the lowest RMSE keeps 25% of the anomaly energy at day 10 (E5).

The other 26 of the 44 cases behave differently. They occur at median skill 0.72–0.99 against climatology and 0.66–0.99 against persistence, and the usual remedies for smoothing do not remove them. Biological ensembles whose spread matches their error in distribution become overconfident under regime shift (spread–skill 0.20). Other objectives and selection rules leave the forecasts nearly flat, and stronger architectures fix the regime labels but not the oscillation (E2–E4). In Lotka–Volterra, clipping forecasts to positivity leaves the regime errors in place, while networks with a positivity constraint fix them at 1.2–2.7× the error winner's NRMSE (E9, E10). In the KS PDE, learned operators break an exact conservation law under parameter shift with no error penalty (E6). The value of the benchmark is in telling these cases apart and showing which fix applies to which (table in G1).

A. Skill against climatology and persistence; NRMSE ≈ 0.9 and > 1. E1 reports skill at each lead against both references for every tier. Median climate skill is positive against both at every lead: against climatology 0.80 at day 1 and 0.04 at day 20, against persistence 0.03 at day 1 and 0.43 at day 20 (positive in 97% of all model, split and lead cells for each reference). The ≈ 0.9 average NRMSE is therefore dominated by long leads, where skill against climatology is small. On NRMSE > 1: the forecasts above 1 are the ones that restore variance (the calibrated ablation variants and probabilistic members), and a forecast with realistic variance but imperfect phase is penalised heavily by RMSE. Under equal mean and variance, NRMSE² = 2(1 − ρ), so NRMSE exceeds 1 when the correlation ρ < 0.5 (derivation in Appendix H). This shows how phase errors inflate RMSE; it is not the full double penalty, which concerns displaced events counted as both a miss and a false alarm, the problem neighbourhood verification addresses (Ebert, 2008). The conclusion we draw is limited: members with NRMSE ≈ 1.2 represent the distribution and the events better than the ensemble mean, but that does not make them better deterministic point forecasts.

B. Horizons within the predictability limit, per lead. We agree that part of the disagreement comes from forecasting past the predictability limit. A forecast can serve two goals: an accurate state at short horizons, and plausible dynamics at long horizons, where a chaotic trajectory cannot be predicted. The revision states the two sets of conclusions separately, with chaotic horizons in Lyapunov times (Section 5.7). Our Lorenz forecasts span 18–21 Lyapunov times. Within about one, error and amplitude rankings agree in three of four splits (ρ = 0.9); over the full horizon they do not (ρ −0.1 to 0), and at least part of that separation is the expected effect of chaos. In the KS PDE, error averaged over ~4.7 Lyapunov times picks a forecast that decays to zero, while error within one Lyapunov time ranks models in line with their physical properties (ρ −0.15 → +0.47, E6). Two things this does not explain. The SIR, Lotka–Volterra and biological cases occur at high skill at every lead (above). And at long horizons, validity is not about the trajectory: the forecast should still keep invariants, amplitude distributions and regime statistics, which the decaying KS forecast and the operators that break the invariant fail. In climate, median skill stays positive through day 20, yet event F1 is 0 at every lead (E1).

C. Probabilistic baseline with CRPS and spread–skill. In short: probabilistic forecasting repairs part of the smoothing of deterministic forecasts, but under the biological distribution shift a clear calibration failure and the loss of the oscillation remain. E2 reports CRPS, spread–skill, rank histograms and event scores from members in every tier. In climate, members bring back variance and event F1 (0.43–0.82), with spread–skill 0.80–0.97. In biology, spread–skill is ≈ 1.0 in distribution but 0.20 under regime shift, and members do not bring back the calcium oscillation. The disagreement remains for ensemble means; scoring members removes part of it in climate and Lorenz in distribution.

Established remedies. We agree these are the right tools, and spectra (E5, E6), neighbourhood verification (FSS, E5) and ensembles with CRPS (E2, E5, E6) are now all in. What they show is that the verdict depends on the forecast product and its scoring rule, so we score means, ensembles and event probabilities each with their own rule rather than ranking them on one number. For example, the Z500 ensemble mean, best by RMSE, has a trough FSS of only 0.06 at day 10, against 0.44 for neighbourhood probabilities. This is why the scoring choice should be made explicit for each task, which is what SciDynBench-R does. These tools do not remove the failures under regime shift in biology, Lorenz and KS (table in G1).

D. Relation to verification and existing benchmarks. Section 2.4 of the revision now covers the double penalty and spatial/neighbourhood verification (Mass et al., 2002; Ebert, 2008; Roberts & Lean, 2008; Gilleland et al., 2009), proper scoring rules and ensemble verification (Hersbach, 2000; Gneiting & Raftery, 2007; Ferro, 2014; Fortin et al., 2014), and the spectral smoothing of ML weather models (Rasp et al., 2024; Bonavita, 2024; Price et al., 2025). ClimateBench (Watson-Parris et al., 2022) benchmarks emulation of climate projections; ChaosBench (Nathaniel et al., 2024) targets subseasonal-to-seasonal prediction with physics-based metrics; ExtremeWeatherBench (McGovern et al., 2026, preprint) evaluates weather models on high-impact events; "OceanBench" names a benchmark for interpolating sea surface height (Johnson et al., 2023) and a global ocean forecasting benchmark (El Aouni et al., 2025). Each of these evaluates a specific geophysical problem with domain metrics. We do not propose a new verification score. The contribution of SciDynBench-R is to bring verification of scientific properties, scientifically motivated distribution shifts, disagreement between model rankings and the evaluation of remedies into one experimental framework across dynamical systems: controlled systems where the property is known by construction, a mechanistic biological model, a PDE and a full field climate task. The revised introduction states it this way.

Presentation. The revision now leads with the rank statistics and the mechanisms (abstract, introduction, Section 5.1) and includes the table from the general response (Section 5.7, Table 2).

Reviewer_6rTkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SciDynBench-R, a benchmark for testing whether scientific forecasting models that achieve low pointwise error also preserve scientifically meaningful properties such as regimes, oscillations, events, and physical constraints. It evaluates canonical ODEs, a mechanistic biological transport model, and reduced WeatherBench2 Z500 climate indices under IID and scientifically motivated distribution shifts. Across 187 model/split means, the authors identify 44 cases where favorable prediction error coexists with poor scientific validity, showing that error-based and validity-based model rankings can substantially disagree.

Critical strengths
  1. The paper introduces SciDynBench-R, a benchmark for testing whether scientific forecasting models that achieve low pointwise error also preserve scientifically meaningful properties such as regimes, oscillations, events, and physical constraints. It evaluates canonical ODEs, a mechanistic biological transport model, and reduced WeatherBench2 Z500 climate indices under IID and scientifically motivated distribution shifts. Across 187 model/split means, the authors identify 44 cases where favorable prediction error coexists with poor scientific validity, showing that error-based and validity-based model rankings can substantially disagree.
  2. Using controlled ODEs, a mechanistic biological system, and real-data-derived climate indices gives the benchmark a useful progression from interpretable synthetic evidence to a more realistic external validation setting.
  3. In particular, the climate evaluation assigns years before window construction, fits preprocessing only on training years, and audits source-date overlap. The paper also provides a frozen experimental snapshot and rerun instructions.
Critical weaknesses
  1. Regime definitions, event thresholds, metric groupings, and particularly the weighted validity score can materially influence model rankings. The paper acknowledges that these metrics encode scientific judgment, but there is limited sensitivity analysis showing that the central conclusions are robust to reasonable alternative definitions.
  2. The climate component uses six reduced Z500-derived indices and proxy diagnostics rather than full-field weather forecasting or exact physical diagnostics. This makes the real-world evidence useful as a stress test but weaker than the paper's controlled evidence in establishing broad relevance to modern weather forecasting.
  3. Most neural models are trained and selected using MSE, so it is perhaps unsurprising that they perform better on error than on task-specific validity. The paper includes a useful validity-aware climate ablation, but it does not systematically evaluate alternative training objectives, probabilistic forecasters, or validity-aware model-selection strategies.
Questions for authors

q1. For example, if regime thresholds, event thresholds, SVS weights, or the low-error cutoff are varied over reasonable ranges, do the qualitative and quantitative conclusions remain similar? q2. Can the authors more clearly separate an inherent error–validity mismatch from a consequence of training and selecting almost all models using MSE? q3. How well do the conclusions survive with stronger or more realistic forecasting models/tasks? In particular, could the authors add at least one stronger contemporary forecasting baseline, probabilistic model, or a richer climate task?

1 comment

Thank you for the constructive review. For each question we say what the new evidence shows and what it leaves open.

q1: Thresholds, SVS weights and the low error cutoff. The question is whether our conclusions are produced by our scoring choices. We varied these jointly over 2,000 specifications and separately perturbed the label thresholds with relabeling (E0). The counts move, but the pattern holds: in 95% of specifications at most 7 of 26 groups share the error and validity winner (paper: 5), and at least 6 have a negative rank correlation (paper: 10).

The rank correlation itself, with 95% seed bootstrap intervals (1,000 replicates; all groups in Appendix H), splits by system. It is clearly negative in SIR (IID −0.79 [−0.79, −0.61], interpolation −0.75 [−0.82, −0.71]) and biology (boundary shift −0.61 [−0.82, −0.43], regime shift −0.46 [−0.71, −0.29]), and clearly positive in Van der Pol (interpolation 0.86 [0.82, 0.89]), Lorenz (0.32–0.61) and the climate indices (0.52–0.69), with every lower bound above 0.25; Lotka–Volterra lies in between. For 9 of the 10 negative groups the whole interval is below zero, the median interval width is 0.25, and six groups stay negative whichever single model is left out.

What is less robust is the count of low error failures (median 44; 5–95%: 18–71), which also has a high base rate in biology and climate. The revision therefore presents it as descriptive and leads with the rank statistics.

q2: Inherent, or a consequence of MSE training and selection? It depends on the failure type, so we now answer it case by case (table in G1). In climate, the picture is consistent with smoothing toward the conditional mean: probabilistic members recover variance and event frequency, and variance objectives or calibration trade off against NRMSE (E2, E8). In biology under regime shift, the picture is different. Ensembles whose spread matches their error in distribution (spread–skill ≈ 1.03) become overconfident (0.20, E2). Changing the objective or the selection rule leaves the forecasts nearly flat (regime agreement 0.13–0.19, amplitude ≤ 0.09× the target, E3). Stronger architectures do fix the regime labels (PatchTST/iTransformer: 0.19 → 0.84, E4), but not the oscillation: peak count error stays at 24–26 and amplitude at 0.44–0.60× the target. So regime classification improves while oscillation fidelity stays poor. Selection itself matters: choosing by validation validity within a 10% error budget improves test validity in 9/26 groups, by up to 0.53 (E7).

Together these results point to a more useful conclusion than "MSE smooths". Different failure mechanisms respond differently to probabilistic forecasting, selection by validity, alternative objectives and physical constraints. Some discrepancies can be substantially reduced, while others remain unresolved under the methods we tested. The disagreement between error and validity is therefore a family of evaluation problems with different causes, and the benchmark is what tells them apart.

q3: Stronger and probabilistic models, a richer climate task. The question is whether the disagreement is an artefact of weak deterministic models and reduced indices. It is not, although which forecast product looks best does change:

  • Stronger models (E4). PatchTST, iTransformer, TCN, DLinear and zero-shot Chronos-2, Chronos-Bolt and TimesFM-2.5. They take over as error winners in 16 of 26 groups, yet the error winner is also the validity winner in only 4 of 26, and the rank correlation is negative in 12 of 26.
  • Probabilistic models (E2). GRU/Transformer ensembles trained with Gaussian NLL or CRPS, scored with CRPS, spread–skill and rank histograms. In climate, ensemble means behave like deterministic forecasts (event F1 0), while members recover variance and event F1 (0.43–0.82) at NRMSE ≈ 1.2. Event probabilities from members rank amplification events best (ROC-AUC 0.77–0.81 vs 0.64–0.71 for deterministic forecasts).
  • Full field Z500 (E5). U-Net, FNO and a CRPS U-Net ensemble on 5.625° daily fields, 1–20 days. The forecast with the lowest RMSE, the ensemble mean, keeps only 25% of the anomaly energy at day 10 and has high variability F1 0.05; its members keep 98% of the energy and reach F1 0.40.

Means, members and event probabilities are different forecast products, so we do not claim one is better by a single number. Each is scored with its own rule (RMSE for the mean, CRPS and spread–skill for the ensemble, event scores for members and probabilities). The point is that the product a leaderboard ranked by error selects is the one that misses the events. What none of these model classes fixes is the biological regime shift: the oscillation stays wrong (q2).

Reduced climate indices. We agree. E5 shows the same reversal on full fields, so the index task stays as a controlled entry point next to the full field one.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Comment G1 — General response (1/3): main findings

We thank the reviewers for careful and, in places, challenging reviews. Three concerns came up more than once: whether the results depend on our scoring choices, whether the disagreement is just the known effect of MSE training (smoothing toward the conditional mean, the double penalty), and whether it holds for stronger, probabilistic and more realistic forecasters. We ran new experiments on the frozen data and splits. This comment summarises what we found, the next one lists the changes to the paper, and the third the experiments. Code and the outputs of every run will be added to the supplement.

1. The disagreement is not one phenomenon. It comes from different mechanisms, and these respond differently to probabilistic forecasting, alternative objectives, selection by validity and physical constraints: some discrepancies can be substantially reduced, whereas others remain unresolved under the methods we tested. In 18 of the 44 cases (the 10 climate cases and the 8 Lorenz cases beyond about one Lyapunov time), skill against climatology is low; there, at least part of the disagreement is the expected consequence of limited predictability, and the revision says so. The other 26 (SIR, Lotka–Volterra, biology) occur where median skill is 0.72–0.99 against climatology and 0.66–0.99 against persistence at every lead (E1), so predictability does not explain them. Across all cases, the new experiments show which remedy works where:

CaseWhat helpsWhat does not
Climate eventsEvent probabilities from members (amplification ROC-AUC 0.77–0.81 vs 0.64–0.71 for deterministic forecasts); calibration, at an NRMSE costAny deterministic model: event F1 = 0
Lorenz, IIDEnsemble members (regime agreement 0.28 → 0.86); spectral loss (0.40 → 0.87 at +0.02 NRMSE)–
Lorenz, regime shiftNothing we triedMembers, 7 objectives × 2 selection rules
SIR, constraintsPost hoc projection (agreement ≥ 0.98, NRMSE change ≤ +3%)Members, objectives incl. a physics residual, stronger models
Lotka–Volterra, regimesNetworks with a positivity constraint (0.69–0.94), at 1.2–2.7× the error winner's NRMSEPost hoc clipping (0.34 → 0.34 for the IID GRU)
Biology, regime shiftPartly: PatchTST/iTransformer fix regime labels (0.19 → 0.84), but the oscillation stays wrong (peak error 24–26, amplitude 0.44–0.60×)Members (spread–skill 1.03 → 0.20); objectives and selection (agreement 0.13–0.19, amplitude ≤ 0.09×)
KS PDE, parameter extrapolationScoring error within ~1 Lyapunov timeError averaged over the horizon, which picks a forecast that decays to zero
KS PDE, regime shiftNothing we triedCNN, FNO, spectral loss, CRPS ensemble, operators that conserve the mean

Ensemble means, members and event probabilities are different forecast products, so we score each with its own rule (RMSE for the mean, CRPS and spread–skill for the ensemble, event scores for members and probabilities) and keep deterministic and probabilistic evaluation apart.

2. The pattern holds across scoring choices and seeds, and the correlation splits by system. Over 2,000 random specifications of validity weights, error metric, the fraction counted as low error, and the cutoff (E0), in 95% of them at most 7 of 26 groups share the error and validity winner (paper: 5), and at least 6 have a negative rank correlation between error and validity (paper: 10). Perturbing the label thresholds and relabeling gives 2–3 and 10–13 (3 and 12 unperturbed on the same pool).

The correlation itself is not uniformly weak; it depends on the system. It is negative in all five biological groups and in three of four SIR groups (ρ from −0.79 to −0.18), and positive in every Van der Pol, Lorenz and climate index group (0.32 to 0.86); Lotka–Volterra lies in between. Under a seed bootstrap (1,000 replicates), 9 of the 10 negative groups are negative in every replicate, and the sign is stable in 22 of 25 groups. Six groups stay negative whichever single model is left out. In the climate indices every deterministic baseline misses the events (F1 = 0), so the ranking among them says little; there the disagreement is between deterministic and probabilistic forecasts (E2).

3. Selecting for validity pays off. Selecting by validation validity while allowing up to 10% extra validation error changes the chosen model in 13 of 26 groups. Test validity improves in 9 of them, by up to 0.53 (median 0.14), and is marginally worse in 1; test NRMSE changes by −15% to +10% (E7).

AuthorsOpenReview ↗

Comment G2 — General response (2/3): changes to the paper

We have uploaded a revised PDF; all changes are in blue.

  1. Base rates and what we lead with. Under the paper's rules every biological and climate baseline fails at least one criterion (35/35, 40/40), so in those tiers the count of 44 cases means that no baseline meets the criteria; in the canonical tier, models with low error fail at 75% vs 61% for the rest. We now report these base rates. The abstract, introduction and Section 5.1 now lead with the rank statistics and the mechanisms, and the count of 44 cases is reported as descriptive.
  2. Calcium peaks. Detection used a prominence relative to each trajectory's own amplitude, so ripples on flat forecasts counted as peaks. With a threshold tied to the target (5% of its amplitude, 3 nM floor), the peak error of the Transformer under regime shift falls from 36.7 to 8.8 (persistence 7.6) at 0.05× the target amplitude, so the example is a lost oscillation. All statistics here use the corrected metric (44 cases; same winner 5/26; negative correlation 10/26).
  3. Definition of scientific validity (Section 3.5): an umbrella term defined per task, with three kinds reported separately (task properties, physical consistency checks, dynamical diagnostics); only checks grounded in the governing equations are called physical validity.
  4. Related work on forecast verification (Section 2.4), and a contribution statement that places the benchmark relative to it (Section 1).
  5. The remedies table from G1 with its discussion (Section 5.7, Table 2), and conclusions for short and long horizons stated separately, with chaotic horizons in Lyapunov times.
  6. All new experiments (Appendix H), including the distribution of the rank correlation, seed bootstrap and leave-one-model-out analysis.
  7. Renamed leftover files in Appendix E.

Rerun. All baselines except the Neural ODE were rerun with saved forecasts. They reproduce the frozen tables (median relative NRMSE difference 0.000 in every tier; the biology MLP differs by 3–11% because of a different compute backend) and the headline statistics.

AuthorsOpenReview ↗

Comment G3 — General response (3/3): new experiments

E0 (scoring choices) and E7 (selection by validity) are summarised in G1.

E1. Skill by lead. Against persistence and climatology, every tier. Median climate skill is positive against both at every lead (climatology: 0.80 at day 1, 0.04 at day 20; persistence: 0.03 at day 1, 0.43 at day 20). Lorenz forecasts span 18–21 Lyapunov times and lose skill against climatology after about one; within it, error and amplitude rankings agree in 3 of 4 splits (ρ = 0.9), over the full horizon they do not (−0.1 to 0).

E2. Probabilistic forecasters. Forecasters trained with Gaussian NLL or CRPS (GRU/Transformer, 3 seeds), scored on ensemble means, single members and event probabilities from members. Climate: ensemble means behave like deterministic forecasts (PC1 variance ratio ≈ 0.15, event F1 0), while members bring back variance (1.0–1.8) and event F1 (0.43–0.82) at NRMSE ≈ 1.2. Scored on continuous event strength, deterministic forecasts do rank events (amplification ROC-AUC 0.64–0.71), but their damped amplitude never crosses the event thresholds; probabilities from members rank amplification events best (0.77–0.81). Biology: spread–skill 1.02–1.04 in distribution, 0.20 under regime shift; members do not bring back the calcium oscillation.

E3. Objectives × checkpoint selection. Seven objectives (MSE, MAE, Huber, variance matching, spectral, validity proxy, equation residual), each with two checkpoint rules. 21 of 218 configurations improve validity at no NRMSE cost. The spectral loss is the clearest case: biological extrapolation regime agreement 0.45 → 0.85 at equal NRMSE, Lorenz IID 0.40 → 0.87 at +0.02. Nothing materially helps biology or Lorenz under regime shift.

E4. Stronger baselines. PatchTST, iTransformer, TCN, DLinear and zero-shot Chronos-2, Chronos-Bolt, TimesFM-2.5. They become error winners in 16/26 groups, yet error and validity winners coincide in 4/26 and rank correlation is negative in 12/26. Under biological regime shift, PatchTST/iTransformer cut NRMSE 3–4× and raise regime agreement from 0.19 to 0.84, but peak count error stays at 24–26 (persistence 7.6) and amplitude at 0.44–0.60× the target; zero-shot Chronos-2 has 2.3× their NRMSE, yet agreement 0.97 and peak error 0.8.

E5. Full field Z500. 5.625° daily fields, 1–20 days, U-Net, FNO and a CRPS U-Net ensemble, scored with RMSE/ACC as in WeatherBench 2, zonal spectra and the paper's events derived from predicted fields. The forecast with the lowest RMSE (ensemble mean; 701 vs 781 m²/s² for climatology) keeps 25% of the anomaly energy at day 10 and 13% of the power at small scales (k = 10–20) at day 5, with high variability F1 0.05; its members keep 98% of the energy and reach F1 0.40. Fractions Skill Score for anomalies beyond the training 10th/90th percentiles (boxes up to 9×9 cells): neighbourhood ensemble probabilities score best at every scale and lead, at day 10 the ensemble mean's trough FSS is 0.06 (members 0.31, neighbourhood probabilities 0.44).

E6. Kuramoto–Sivashinsky. Domain length as the physical parameter; ridge, MLP, CNN, FNO, FNO with spectral loss, a CRPS ensemble of FNOs, and CNN/FNO variants that conserve the mean. IID: rankings agree (ρ = 0.79). Parameter extrapolation: NRMSE averaged over the horizon picks ridge, whose forecasts decay to zero (energy ratio 0.007), while the neural operators stay chaotic but break the exact mean invariant in 26–74% of windows (0% in IID, 97–100% under regime shift); scoring error within one Lyapunov time removes the disagreement (ρ −0.15 → +0.47). The FNO that conserves the mean has the best property score but is 6th of 10 by NRMSE. Regime shift to travelling waves: every model fails, and the spread–skill of the CRPS ensemble drops from 1.04 to 0.31.

E8. Climate ablation. Variance objective weight λ ∈ {0, 0.1, 0.3, 1, 3, 10} × calibration: no setting improves NRMSE and event scores together; calibration alone (F1 0.46 at NRMSE 1.09) does better than the objective alone (0.40 at 1.14).

E9. Models that preserve structure. Output layers with hard constraints (SIR as N·softmax, Lotka–Volterra through softplus) have zero violations; in Lotka–Volterra they reach regime agreement 0.69–0.94, against 0.00–0.62 without it, at 1.2–2.7× the error winner's NRMSE. SINDy fitted per trajectory is best or second best on regime agreement and amplitude in 7 of 8 Lotka–Volterra/Van der Pol groups. Its mean NRMSE ranks anywhere from first to last, because a few identified models diverge. With all new models in the pool, same winner 3/26 and negative correlation 12/26.

E10. Post hoc projection. Clipping SIR forecasts at zero and rescaling them to the population lifts every model's regime agreement to 0.98 or more (IID GRU: 0.28 → 1.00), with NRMSE changing by at most +3%. Clipping Lotka–Volterra forecasts at zero removes the violations but not the regime errors. With projected forecasts in the pool, same winner 4/26 and negative correlation 10/26.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.