- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SciDynBench-R, a benchmark for evaluating disagreement between conventional forecast errors and scientific validity in dynamical systems. The benchmark covers three settings: four canonical ODE systems, a mechanistic biological transport model, and six reduced climate indices derived from WeatherBench2 Z500 data. It evaluates forecasting models under IID and distribution shifts using standard error metrics alongside regime preservation, oscillation fidelity, event detection, and physical constraint violations.
Across 187 model/split configurations, the authors identify 44 cases where low prediction error coincides with poor scientific validity. In particular, the climate experiments demonstrate that models optimized for NRMSE can suppress variability and miss extreme events. A supplementary experiment shows that improving event fidelity is possible, although at the expense of higher forecast error.
Critical strengths
-
Important evaluation problem. I agree with the motivation that low forecast error does not necessarily imply scientifically meaningful predictions. The distinction between trajectory accuracy and dynamical properties, particularly regime transitions and oscillations, is important for scientific machine learning. The benchmark provides a systematic way to evaluate this discrepancy across different dynamical systems.
-
Careful evaluation protocol across different settings. The inclusion of controlled ODE systems, mechanistic biological dynamics, and reduced climate observations provides complementary perspectives. I particularly appreciate the explicit treatment of distribution shifts and temporal leakage in the climate experiments, including disjoint training and testing years and training-only preprocessing.
Critical weaknesses
-
Scientific validity is largely defined through heuristic metrics. The benchmark treats regime preservation, peak counts, and event detection as indicators of scientific validity. While these are meaningful diagnostics, they do not necessarily establish physical consistency. For example, the climate evaluation relies on threshold-based labels and Z500-derived proxies rather than actual physical constraints or governing equation residuals. I think the distinction between scientific validity and task-specific forecast properties needs to be better justified.
-
Limited complexity and realism of the evaluated dynamical systems. Most experiments rely on relatively simple ODEs, while the biological system is a reduced, Palk-inspired model rather than a complete physiological representation. The climate component is also limited to six reduced Z500 indices instead of full atmospheric fields. It is unclear whether the observed disagreement between error and validity extends to higher-dimensional PDE systems or realistic weather prediction models. This limits the broader applicability of the proposed benchmark.
-
The baseline models are relatively limited. The experiments primarily compare conventional sequence forecasting architectures trained using MSE. Consequently, the observed failures may largely reflect the known tendency of point predictors to smooth oscillations and extreme events. The benchmark would benefit from including physics-informed, structure-preserving, or probabilistic forecasting methods. These would help establish whether the reported disagreement remains significant when models are explicitly designed to preserve dynamical properties.
-
Limited robustness analysis of the proposed validity metrics. The Metric Disagreement Index is based on Spearman rank correlation between error and validity rankings. However, these rankings involve a relatively small number of models, and the validity scores themselves depend on predefined thresholds, normalization, and weighting choices. Similarly, the 44 reported low-error scientific failures depend on family-specific criteria. I would like to see sensitivity analyses demonstrating that the main conclusions remain robust under alternative validity definitions and thresholds.
-
The climate ablation does not isolate the source of improved event fidelity. The fidelity-aware GRU changes the training objective and uses additional calibration, making it difficult to determine which component drives the improvement. For example, under temporal OOD, event fidelity improves substantially, but NRMSE increases from 0.906 to 1.134. The paper demonstrates a trade-off between the evaluated configurations but does not establish whether better model selection, calibration alone, or alternative objectives could improve both metrics. A more controlled ablation would strengthen the practical contribution of the benchmark.
Questions for authors
See above.
2 comments
W4: Robustness of MDI and thresholds. The question is whether the disagreement depends on particular thresholds, models or seeds. It does not, in the following sense (E0 and finding 2 in the general response). The pattern holds across 2,000 joint specifications and the perturbations of the label thresholds, and also with up to 13 models per group after adding E4 (same winner 4/26, negative correlation 12/26). Under a seed bootstrap and leave-one-model-out (Appendix H), the sign of the correlation is stable in 22 of 25 groups, 9 of the 10 negative groups have their whole 95% interval below zero, and six stay negative whichever single model is left out. What is less robust is the count of low error failures, which has a high base rate in biology and climate; the revision reports that base rate and treats the count as descriptive.
W5: The climate ablation does not isolate the source. Agreed: the original ablation changed the objective and the calibration together. E8 separates them. We swept the weight λ of the variance objective ∈ {0, 0.1, 0.3, 1, 3, 10} with and without validation calibration (5 seeds). No configuration improves both NRMSE and event scores, and NRMSE rises almost monotonically with λ. Calibration alone gives high variability F1 0.46 at NRMSE 1.09, the objective alone 0.40 at 1.14, and combining them raises amplification SEDI to 0.56 at the cost of many false alarms. Checkpoint selection by validation event score (E3) adds little in climate. Probabilistic members give the best event ranking (amplification ROC-AUC 0.77–0.81, vs 0.64–0.71 for deterministic forecasts scored on continuous event strength). The revision presents this as an observed tradeoff among the tested configurations, not as evidence that event fidelity requires higher error in general.
Thank you for the detailed review.
W1: Validity defined by heuristic metrics. We agree that regime and event scores are not physical consistency, and the revision makes the distinction explicit (Section 3.5). In this paper, scientific validity is an umbrella term, defined per task, for the properties a scientist relies on when using the forecast for a given task, not physical consistency in the strict sense. We report three kinds separately and call only the second physical validity. (i) Task properties: regime labels, oscillation statistics and event scores, whose thresholds are scientific choices perturbed in E0. (ii) Physical consistency checks grounded in the governing equations: nonnegativity and population conservation (SIR), positivity (Lotka–Volterra), physiological bounds (biology), the exact mean invariant of KS and the residual of the known flow equations (Lorenz). (iii) Dynamical diagnostics: zonal energy spectra and FSS (E5), KS band power (E6).
The disagreement is not confined to the task properties. Among the physical checks, KS neural operators never violate the mean invariant in distribution, but violate it in 26–74% of windows under parameter extrapolation with no error penalty, and in 97–100% under regime shift (E6). SIR forecasts with low error have large constraint violations that a projection removes at almost no error cost (E10). Among the diagnostics, the Z500 forecast with the lowest RMSE keeps only 13% of the power at small scales at day 5. For climate, the reviewer is right that spectra, event F1 and FSS are diagnostics rather than constraints from the governing equations; the climate tier tests event fidelity, and the revision describes it that way.
W2: System complexity. The concern is whether the disagreement is specific to small ODE systems. It is not. On full 5.625° Z500 fields (E5; 1–20 days, WeatherBench 2 metrics, spectra and FSS), error and fidelity rankings reverse as well. In a spatiotemporally chaotic PDE (E6; Kuramoto–Sivashinsky with neural operators and physically defined shifts), three mechanisms separate: error over long horizons that rewards decay to zero, invariant violations with no error penalty, and regime shifts that defeat every model. Both are still research settings rather than operational weather models.
The biological model is a reduced mechanistic model on purpose. It is based on the salivary acinar cell model of Palk et al. (2010), with cytosolic and ER calcium, IP3, six ions and cell volume. Its regimes (sustained and damped calcium oscillations, baseline, secretion responses) and physiological constraints follow from known mechanisms, so the labels need no annotation and shifts can be set through physiological parameters. A complete physiological model would add realism but not this ground truth. The protocol itself does not depend on the simulator, which is how the KS task was added.
W3: Limited baselines. Besides the stronger forecasters (E4), probabilistic models (E2) and alternative objectives including a residual of the known equations, as in PINNs (E3), E9 adds models built to preserve structure: output layers with hard constraints, SINDy fitted per trajectory, and operators that conserve the mean for KS. They keep what they are built for. Constrained forecasts have zero violations, and Lotka–Volterra networks with a positivity constraint reach regime agreement 0.69–0.94 (trained unconstrained networks: 0.00–0.62), at 1.2–2.7× the error winner's NRMSE. For SIR, a simple post hoc projection already does as well (E10). SINDy is best or second best on regime agreement and amplitude in 7 of 8 Lotka–Volterra/Van der Pol groups, yet its mean NRMSE ranks anywhere from first to last, because a few identified models diverge. With these models in the pool, error and validity winners coincide in 3/26 groups. Under regime shift, enforcing an invariant is not enough: the KS operators that conserve the mean fail like the unconstrained ones.
We did not run a coordinate-based PINN. With known equations and parameters it acts as a solver rather than a forecaster, and its physics-informed loss is what our residual objective already applies to forecasters. Hamiltonian networks assume energy conservation, so they do not fit these dissipative systems.