- Rating
- 2
- Confidence
- 1(−2)
- Informativeness
- 1(−1)
- Presentation
- 1(−1)
- Soundness
- 1(−1)
Read review
Paper summary
This paper proposes a unified evaluation framework for dynamical system models, encompassing prediction, generalization to new initial conditions, state perturbations, interventions on fixed variables, and causal graph recovery. It introduces simple baselines as well as "oracle" benchmarks based on known equation forms. The authors compare various methods using ODEBench and two sets of sparse dynamical systems, finding that strong predictive performance does not guarantee reliable responses to interventions; furthermore, Figure F1 may mask deficiencies in structural recovery.
Critical strengths
The unified interface allows for the assessment of diverse dynamical system models under consistent prediction and intervention tasks. The inclusion of simple baselines, oracle benchmarks, and analyses of response magnitude and shape enhances the interpretability of the results.
The authors clearly distinguish between a priori assumptions and post-hoc analyses. They transparently report results that fail to meet predefined criteria within the stochastic system sets, demonstrating a commendable level of rigor and openness.
Critical weaknesses
-
The "oracle" described in Section 3 has access to the true graph structure, fixed coefficients, and the signs of parameters. Table 11 only varies the initialization range, failing to disentangle the specific contributions of these different a priori pieces of information. Please include a comparison where only the true functional terms are known (while coefficients are estimated freely) to avoid attributing performance gaps solely to the functional form.
-
In Section 5.3 and Table 16, the comparison between time-step dependencies in discrete-time models and the direct causal graphs of continuous-time ODEs may lead to "false positives"—where correct, indirect causal propagation is misidentified as an error. Please incorporate an evaluation based on generator readouts or comparisons against the true discrete-time flow graph, rather than merely noting this mismatch in Appendix C.
Questions for authors
-
In Table 25, the prediction accuracy of "Weak SINDy" drops from 77% to 2% upon the addition of 1% noise. Is this result highly sensitive to the specific configuration? Please re-evaluate key methods using validation sets drawn from the training intervals and a standardized hyperparameter-tuning budget to distinguish between inherent methodological limitations and suboptimal default parameter settings.
-
Section 6 notes that each configuration is evaluated using only a single random seed and one noise realization, while the bootstrapping mentioned in Section 3 primarily reflects variance across different systems. Could you perform independent repetitions and report hierarchical confidence intervals to verify whether the conclusions regarding interventions—presented in Tables 2 and 26—are robust?
-
In Section 5.1 and Table 15, the dual-trajectory experiments simultaneously increase the number of training samples from 105 to 210; thus, it is not yet possible to disentangle the gains resulting from increased state coverage versus those from a larger sample size. Could you include a comparison between single-trajectory and dual-trajectory setups while keeping the total sample budget constant, in order to clarify the source of the improvements in generalization?
3 comments
Response to Reviewer r6SB (2/3)
(3) Is Weak SINDy's collapse a configuration effect? Not within the 24 configurations we searched (Table 25 of the submission is now Table 28). Eight methods (the four SINDy variants, the two neural ODEs, PySR and ODEFormer) each have one predeclared grid of 24 configurations, default included (Appendix G.3), used in three settings: (a) defaults; (b) for methods added after our first ODEBench analysis, tuned on the development suite as the original nine were; (c) for the SINDy variants and neural ODEs, per system from the training window alone: fit on the first 55%, validate the forecast on the next 15%, refit on all 105 samples. PySR and ODEFormer are in (a) and (b) only: per-system validation of PySR would take about eight core-days per suite, and our plan left ODEFormer out without a stated reason. Their per-system validation is untested, which Appendix C lists as a limitation. A bound tuned on the test questions is shown only as a diagnostic: it picks the best of 24 after seeing the test, so it is optimistic by construction.
Weak SINDy, random suite, σ = 0.01: forecast accuracy 0% by default (a), which is also the configuration that development-suite tuning selects (b), 5% (c), and 10% even for the test-tuned bound; its clamps diverge in every setting. No method in (b) or (c) beats no response on a sparse clamp with noise, on any suite, so Finding 2 holds under tuned settings. Some configuration often does better, but the training data cannot pick it: for the Neural ODE on the random suite at σ = 0.01, some configuration has a clamp error below 1 on 49 of 60 systems, the validated choice on 24, the default on 27.
A check we ran before fixing the plan (two repeated Weak SINDy fits) exposed a seeding defect: PySINDy draws Weak SINDy's test functions from NumPy's global generator when it builds the library, and our harness seeded the generator only afterwards. It is fixed; all Weak SINDy numbers in the revision come from seeded runs (Table 32 compares them with the submitted rows). The collapse is unchanged: over ten replicates, random-suite forecast accuracy is 76% without noise and 1% at σ = 0.01 (submitted: 77% and 2%).
(4) Replicates and hierarchical intervals. Every sparse-clamp verdict in the paper holds (Appendix G.4, Tables 42 and 43; Tables 2 and 26 of the submission are now Tables 11 and 29). Each method ran on five noise draws, the submitted one and four new, shared by all methods and with new seeds; Weak SINDy, both oracles, the Neural ODE and C-NODE ran on ten. Intervals are a two-stage bootstrap over systems (families on ODEBench) and then replicates; we also report how often a system's verdict changes between replicates, and leave-one-family-out medians.
With noise, no learned method's sparse-clamp interval lies below 1 on any suite (e.g. C-NODE, random suite: 1.00 [0.98, 1.03]), and none beats no response in any single random-suite replicate. Without noise, the horseshoe operator and Weak SINDy still do. Leaving out any ODEBench family keeps the neural ODEs' sparse-clamp medians between 1.00 and 1.01. Single systems are less stable: 20 to 30% of random-suite systems change verdict between replicates for the neural ODEs, which Section 5.5 now states. Finding 4: the Neural ODE's AUROC lower bound is at least 0.57 on the random suite at every noise level (0.51 to 0.55 on ODEBench, descriptive with six families). The one single-replicate exception, C-NODE on the hand-written suite in 1 of 10 replicates (upper bound 0.999 at σ = 0.01), is noted in Section 5.2.
Table 47 (Appendix G.7) gives the same intervals for new-IC accuracy, the perturbation error and the clamp on ODEBench's fully coupled systems. The first two come from the predeclared aggregation; the last we computed after the other results were known, and it is labelled so. Finding 1 holds: at every noise level the oracle's interval for new-IC accuracy lies above every learned method's. Perturbation verdicts are those of the submitted run with one addition. On the fully coupled systems, where several learned methods do beat no response, two single-draw verdicts change: PySR at σ = 0.01 now beats it (upper bound 0.51, was 1.01) and ODEFormer at σ = 0.05 no longer does (1.01, was 0.91). Two bounds that equalled 1.00 in the submitted run move apart: the Neural ODE's without noise to 0.99, C-NODE's at σ = 0.05 to 1.05.
Response to Reviewer r6SB (3/3)
(5) Coverage or sample size? The ablation did confound the two. A 2 × 2 on ODEBench at σ = 0.01 (one or two trajectories; 105 or 210 samples; Δt fixed) separates them (Appendix G.5; the ablation, Table 15 of the submission, is now Table 18). IC 3 lies between the training initial conditions, IC 4 beyond them.
- At 210 samples, two 7-unit trajectories against one 14-unit trajectory hold samples, Δt and total observed time fixed. The second trajectory raises accuracy on IC 3 for seven of eight learned methods (e.g. SINDy +22 points [+8, +35]) and on IC 4 for three.
- At 105 samples, each trajectory of the pair lasts only 3.5 time units and the gain largely disappears: on IC 4, our predeclared test, only standardised SINDy gains (+14 [+4, +26]).
- With the 7-unit window fixed (Δt changes), a second trajectory raises IC-3 accuracy for seven of eight methods. Sampling one trajectory twice as densely raises it for none on IC 3, and on IC 4 only for the Neural ODE (+7 [+2, +15]).
So a second region of state space, observed long enough, is what helps; denser sampling of the same orbit does not, with that one exception. Section 5.1 now says this and that, for most methods, the original ablation could not separate the two causes.
We hope this resolves your five points. If it does, we would be grateful if you would consider raising your score; if any point remains open, please tell us which, and we will address it during the discussion.
Response to Reviewer r6SB (1/3)
Thank you for five precise requests. All five are answered in the paper (Section 5.5, Appendix G), each with a decision rule written down before the analysis ran (prereg_rebuttal.md and its SHA-256 hash are in the supplement; the file is self-attested, and deviations are listed in Appendix G). No finding changed, Finding 1 is sharper, and the reading of the two-trajectory ablation is corrected. Numbering follows your review.
(1) What the oracle's prior knowledge contributes. The functional terms carry the advantage; the graph alone does not. Table 11 of the submission (now Table 14) varied only the starting values, so we added a nested ladder on ODEBench (Table 5; Appendix G.1):
R2 to R4 are each fitted both to SINDy's derivative estimate and to the trajectory, so fit type is not confounded with prior knowledge. New-IC accuracy at σ = 0.01:
R1 is one graph-aware estimator: knowing the parents lifts SINDy from 17% to 25% at σ = 0.01 and leaves it below the best learned method at every noise level. This does not show that no estimator could use the graph better; Appendix G.1 says so.
Under the trajectory fit, the full oracle's lead over R2 is +3 points [0, +9] at σ = 0 and +5 [−2, +12] at σ = 0.01. The fixed coefficients and near-true starts matter at σ = 0.05 (+10 [+1, +21]), and under the derivative fit already at σ = 0.01 (+13 [+6, +22]); the true signs add at most 3 points. Finding 1 now says that at low noise the advantage lies in the functional terms, not the graph.
(2) Discrete-time read-outs. We compared them with the true Δt-step graph; indirect dependences do not explain their false positives (Section 5.3; Appendix G.2; Table 16 of the submission is now Table 19). In the Δt-step map, a pair joined by a directed path of length k enters at order Δtᵏ, so the exact flow-map graph is, generically, the reachability closure of the ODE graph. Its only absent edges are the pairs with no directed path, so specificity against it is specificity on those pairs. Along the training orbit, two-step entries are about 20 to 25 times smaller than true edges; no-path entries are exactly zero. The random suite has 120 no-path pairs among its 411 absent pairs (226 two-step, 65 longer), the hand-written suite 37; ODEBench has 3, in one system, too few to test.
If indirect dependences drove the false positives, specificity would be clearly higher on no-path pairs. It is not (random suite, σ = 0.01, no-path vs all absent): PCMCI+ 0.97 vs 0.98, Granger 0.62 vs 0.61, horseshoe Koopman 0.44 vs 0.43. The predeclared comparison is by interval overlap, which is less sensitive than a paired test; the point estimates differ by at most 0.04. The hand-written suite agrees; across suites and noise levels, the two intervals separate only for one new read-out (horseshoe log K, hand-written suite, σ = 0).
Generator read-outs: the continuous-time methods (SINDy variants, PySR, ODEFormer, the neural ODEs) already read the graph off an estimated vector field. For the Koopman operators we added log(K)/Δt; it is undefined in most noisy fits on the sparse suites (EDMD on the random suite: 292 of 300 fits at σ = 0.01) because K has eigenvalues on the non-positive real axis, so it is not usable here. Section 5.3 and Appendix C now say so.