acceptodds
Under review as a conference paper at ICLR 2027

Do Learned Models of Dynamical Systems Answer Interventional Questions? A Benchmark with Oracle References

Abstract

Models of dynamical systems learned from data are used to predict the effects of interventions and to infer which variables drive which, yet they are evaluated mainly on forecasts or recovered equations. We pose five questions to every fitted model: a held-out forecast, a new initial condition, a state perturbation, a clamp do-intervention that replaces one variable’s equation, and the causal graph. Each trajectory and intervention score is placed between a trivial reference and an oracle that knows the true functional form and fits only its constants to the same data; graph scores are compared with the complete graph and with chance. Twelve methods that see only 105 noisy samples of one trajectory (sparse regression, Koopman operators, symbolic regression, neural ODEs and a conditional-independence test) are evaluated on all 63 ODEBench systems and on two new sparse-coupling suites, one generated under an analysis plan fixed before any method was run on it. Given the form, the data suffice at low noise: the trajectory-fit oracle generalises to a new initial condition on 98/88/63% of the predictable ODEBench systems at noise 0/1/5%, whereas the best learned method reaches 62% without noise and at most 37% with it; without noise, the vector fields of SINDy, Weak SINDy and PySR fit the training orbit to within 1% (median) yet miss the truth by 11–181% along the new trajectory. With noise, on the 82 systems with an absent edge, all with three to five variables, no learned method’s median error under a clamp at the end of the training window is detectably below 1, the error of predicting no response, while the oracle’s lies between 0.01 and 0.30. Graph F1 cannot separate any method from the complete graph on ODEBench, which scores 0.955; balanced accuracy and a threshold-free AUROC can, and AUROC exposes read-outs that rank edges well but threshold badly.

Primary area:
learning on time series and dynamical systems
Keywords:
benchmark, dynamical systems, system identification, interventions, causal discovery, symbolic regression, Koopman operator, neural ODE

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r6SBOpenReview ↗
Rating
2
Confidence
1(−2)
Informativeness
1(−1)
Presentation
1(−1)
Soundness
1(−1)
Read review
Paper summary

This paper proposes a unified evaluation framework for dynamical system models, encompassing prediction, generalization to new initial conditions, state perturbations, interventions on fixed variables, and causal graph recovery. It introduces simple baselines as well as "oracle" benchmarks based on known equation forms. The authors compare various methods using ODEBench and two sets of sparse dynamical systems, finding that strong predictive performance does not guarantee reliable responses to interventions; furthermore, Figure F1 may mask deficiencies in structural recovery.

Critical strengths

The unified interface allows for the assessment of diverse dynamical system models under consistent prediction and intervention tasks. The inclusion of simple baselines, oracle benchmarks, and analyses of response magnitude and shape enhances the interpretability of the results.

The authors clearly distinguish between a priori assumptions and post-hoc analyses. They transparently report results that fail to meet predefined criteria within the stochastic system sets, demonstrating a commendable level of rigor and openness.

Critical weaknesses
  1. The "oracle" described in Section 3 has access to the true graph structure, fixed coefficients, and the signs of parameters. Table 11 only varies the initialization range, failing to disentangle the specific contributions of these different a priori pieces of information. Please include a comparison where only the true functional terms are known (while coefficients are estimated freely) to avoid attributing performance gaps solely to the functional form.

  2. In Section 5.3 and Table 16, the comparison between time-step dependencies in discrete-time models and the direct causal graphs of continuous-time ODEs may lead to "false positives"—where correct, indirect causal propagation is misidentified as an error. Please incorporate an evaluation based on generator readouts or comparisons against the true discrete-time flow graph, rather than merely noting this mismatch in Appendix C.

Questions for authors
  1. In Table 25, the prediction accuracy of "Weak SINDy" drops from 77% to 2% upon the addition of 1% noise. Is this result highly sensitive to the specific configuration? Please re-evaluate key methods using validation sets drawn from the training intervals and a standardized hyperparameter-tuning budget to distinguish between inherent methodological limitations and suboptimal default parameter settings.

  2. Section 6 notes that each configuration is evaluated using only a single random seed and one noise realization, while the bootstrapping mentioned in Section 3 primarily reflects variance across different systems. Could you perform independent repetitions and report hierarchical confidence intervals to verify whether the conclusions regarding interventions—presented in Tables 2 and 26—are robust?

  3. In Section 5.1 and Table 15, the dual-trajectory experiments simultaneously increase the number of training samples from 105 to 210; thus, it is not yet possible to disentangle the gains resulting from increased state coverage versus those from a larger sample size. Could you include a comparison between single-trajectory and dual-trajectory setups while keeping the total sample budget constant, in order to clarify the source of the improvements in generalization?

3 comments

Response to Reviewer r6SB (1/3)

Thank you for five precise requests. All five are answered in the paper (Section 5.5, Appendix G), each with a decision rule written down before the analysis ran (prereg_rebuttal.md and its SHA-256 hash are in the supplement; the file is self-attested, and deviations are listed in Appendix G). No finding changed, Finding 1 is sharper, and the reading of the two-trajectory ablation is corrected. Numbering follows your review.

(1) What the oracle's prior knowledge contributes. The functional terms carry the advantage; the graph alone does not. Table 11 of the submission (now Table 14) varied only the starting values, so we added a nested ladder on ODEBench (Table 5; Appendix G.1):

  • R1, graph: SINDy restricted to the library terms of each variable's true parents;
  • R2, terms: the true terms, every coefficient free (fixed ones included), from random-sign starts over four orders of magnitude;
  • R3: R2 started at the true signs;
  • R4: the submitted oracle.

R2 to R4 are each fitted both to SINDy's derivative estimate and to the trajectory, so fit type is not confounded with prior knowledge. New-IC accuracy at σ = 0.01:

  • derivative fit: graph 25%, terms 63%, full oracle 78%;
  • trajectory fit: terms 83%, full oracle 88%;
  • best learned method: 37%.

R1 is one graph-aware estimator: knowing the parents lifts SINDy from 17% to 25% at σ = 0.01 and leaves it below the best learned method at every noise level. This does not show that no estimator could use the graph better; Appendix G.1 says so.

Under the trajectory fit, the full oracle's lead over R2 is +3 points [0, +9] at σ = 0 and +5 [−2, +12] at σ = 0.01. The fixed coefficients and near-true starts matter at σ = 0.05 (+10 [+1, +21]), and under the derivative fit already at σ = 0.01 (+13 [+6, +22]); the true signs add at most 3 points. Finding 1 now says that at low noise the advantage lies in the functional terms, not the graph.

(2) Discrete-time read-outs. We compared them with the true Δt-step graph; indirect dependences do not explain their false positives (Section 5.3; Appendix G.2; Table 16 of the submission is now Table 19). In the Δt-step map, a pair joined by a directed path of length k enters at order Δtᵏ, so the exact flow-map graph is, generically, the reachability closure of the ODE graph. Its only absent edges are the pairs with no directed path, so specificity against it is specificity on those pairs. Along the training orbit, two-step entries are about 20 to 25 times smaller than true edges; no-path entries are exactly zero. The random suite has 120 no-path pairs among its 411 absent pairs (226 two-step, 65 longer), the hand-written suite 37; ODEBench has 3, in one system, too few to test.

If indirect dependences drove the false positives, specificity would be clearly higher on no-path pairs. It is not (random suite, σ = 0.01, no-path vs all absent): PCMCI+ 0.97 vs 0.98, Granger 0.62 vs 0.61, horseshoe Koopman 0.44 vs 0.43. The predeclared comparison is by interval overlap, which is less sensitive than a paired test; the point estimates differ by at most 0.04. The hand-written suite agrees; across suites and noise levels, the two intervals separate only for one new read-out (horseshoe log K, hand-written suite, σ = 0).

Generator read-outs: the continuous-time methods (SINDy variants, PySR, ODEFormer, the neural ODEs) already read the graph off an estimated vector field. For the Koopman operators we added log(K)/Δt; it is undefined in most noisy fits on the sparse suites (EDMD on the random suite: 292 of 300 fits at σ = 0.01) because K has eigenvalues on the non-positive real axis, so it is not usable here. Section 5.3 and Appendix C now say so.

Response to Reviewer r6SB (2/3)

(3) Is Weak SINDy's collapse a configuration effect? Not within the 24 configurations we searched (Table 25 of the submission is now Table 28). Eight methods (the four SINDy variants, the two neural ODEs, PySR and ODEFormer) each have one predeclared grid of 24 configurations, default included (Appendix G.3), used in three settings: (a) defaults; (b) for methods added after our first ODEBench analysis, tuned on the development suite as the original nine were; (c) for the SINDy variants and neural ODEs, per system from the training window alone: fit on the first 55%, validate the forecast on the next 15%, refit on all 105 samples. PySR and ODEFormer are in (a) and (b) only: per-system validation of PySR would take about eight core-days per suite, and our plan left ODEFormer out without a stated reason. Their per-system validation is untested, which Appendix C lists as a limitation. A bound tuned on the test questions is shown only as a diagnostic: it picks the best of 24 after seeing the test, so it is optimistic by construction.

Weak SINDy, random suite, σ = 0.01: forecast accuracy 0% by default (a), which is also the configuration that development-suite tuning selects (b), 5% (c), and 10% even for the test-tuned bound; its clamps diverge in every setting. No method in (b) or (c) beats no response on a sparse clamp with noise, on any suite, so Finding 2 holds under tuned settings. Some configuration often does better, but the training data cannot pick it: for the Neural ODE on the random suite at σ = 0.01, some configuration has a clamp error below 1 on 49 of 60 systems, the validated choice on 24, the default on 27.

A check we ran before fixing the plan (two repeated Weak SINDy fits) exposed a seeding defect: PySINDy draws Weak SINDy's test functions from NumPy's global generator when it builds the library, and our harness seeded the generator only afterwards. It is fixed; all Weak SINDy numbers in the revision come from seeded runs (Table 32 compares them with the submitted rows). The collapse is unchanged: over ten replicates, random-suite forecast accuracy is 76% without noise and 1% at σ = 0.01 (submitted: 77% and 2%).

(4) Replicates and hierarchical intervals. Every sparse-clamp verdict in the paper holds (Appendix G.4, Tables 42 and 43; Tables 2 and 26 of the submission are now Tables 11 and 29). Each method ran on five noise draws, the submitted one and four new, shared by all methods and with new seeds; Weak SINDy, both oracles, the Neural ODE and C-NODE ran on ten. Intervals are a two-stage bootstrap over systems (families on ODEBench) and then replicates; we also report how often a system's verdict changes between replicates, and leave-one-family-out medians.

With noise, no learned method's sparse-clamp interval lies below 1 on any suite (e.g. C-NODE, random suite: 1.00 [0.98, 1.03]), and none beats no response in any single random-suite replicate. Without noise, the horseshoe operator and Weak SINDy still do. Leaving out any ODEBench family keeps the neural ODEs' sparse-clamp medians between 1.00 and 1.01. Single systems are less stable: 20 to 30% of random-suite systems change verdict between replicates for the neural ODEs, which Section 5.5 now states. Finding 4: the Neural ODE's AUROC lower bound is at least 0.57 on the random suite at every noise level (0.51 to 0.55 on ODEBench, descriptive with six families). The one single-replicate exception, C-NODE on the hand-written suite in 1 of 10 replicates (upper bound 0.999 at σ = 0.01), is noted in Section 5.2.

Table 47 (Appendix G.7) gives the same intervals for new-IC accuracy, the perturbation error and the clamp on ODEBench's fully coupled systems. The first two come from the predeclared aggregation; the last we computed after the other results were known, and it is labelled so. Finding 1 holds: at every noise level the oracle's interval for new-IC accuracy lies above every learned method's. Perturbation verdicts are those of the submitted run with one addition. On the fully coupled systems, where several learned methods do beat no response, two single-draw verdicts change: PySR at σ = 0.01 now beats it (upper bound 0.51, was 1.01) and ODEFormer at σ = 0.05 no longer does (1.01, was 0.91). Two bounds that equalled 1.00 in the submitted run move apart: the Neural ODE's without noise to 0.99, C-NODE's at σ = 0.05 to 1.05.

Response to Reviewer r6SB (3/3)

(5) Coverage or sample size? The ablation did confound the two. A 2 × 2 on ODEBench at σ = 0.01 (one or two trajectories; 105 or 210 samples; Δt fixed) separates them (Appendix G.5; the ablation, Table 15 of the submission, is now Table 18). IC 3 lies between the training initial conditions, IC 4 beyond them.

  • At 210 samples, two 7-unit trajectories against one 14-unit trajectory hold samples, Δt and total observed time fixed. The second trajectory raises accuracy on IC 3 for seven of eight learned methods (e.g. SINDy +22 points [+8, +35]) and on IC 4 for three.
  • At 105 samples, each trajectory of the pair lasts only 3.5 time units and the gain largely disappears: on IC 4, our predeclared test, only standardised SINDy gains (+14 [+4, +26]).
  • With the 7-unit window fixed (Δt changes), a second trajectory raises IC-3 accuracy for seven of eight methods. Sampling one trajectory twice as densely raises it for none on IC 3, and on IC 4 only for the Neural ODE (+7 [+2, +15]).

So a second region of state space, observed long enough, is what helps; denser sampling of the same orbit does not, with that one exception. Section 5.1 now says this and that, for most methods, the original ablation could not separate the two causes.

We hope this resolves your five points. If it does, we would be grateful if you would consider raising your score; if any point remains open, please tell us which, and we will address it during the discussion.

Reviewer_UMaiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents what they claim is a new set of benchmarks that prioritize predicting perturbation effects. The idea is fine. The paper is impossible to read and a clear reject. These submissions is why there should be an Editorial rejection phase at conferences.

Critical strengths

There is some merit to finding alternative benchmarks for dynamical systems.

Critical weaknesses

This is AI slop. Full stop. I wasted time on this that I will never get back. Please go and read this. Most of the sentences can barely be called sentences let alone carry a thought. It's numbers and words jumbled together. I could have put this through AI myself and gotten some potentially less ornery response, I was moved by the ICLR statement at the bottom of the first page:

"If you used AI to expand, edit, or polish your review, please provide the input text to the LLM. Better still, consider skipping the LLM and submitting your original text: we,and the authors, are much more interested in your unedited thoughts than in what an LLM has to say. AI-assisted or not, you are putting your name and reputation behind your review: LLM-generated falsehoods, hallucinations or misrepresentations are subject to disciplinary action, which may include desk-rejecting all papers you have authored."

Therefore this review is 100% human generated with all the potential flaws and lack of ability to parse borderline random strings that come with it.

Questions for authors

None. There is no redemption for any of us here.

2 comments

Response to Reviewer UMai

Thank you for your review.

Presentation. Readability was the concern common to all three reviews. In the revision each subsection of Section 5 opens with its finding in words, and the supporting numbers have moved into tables and figures (Table 1, Figures 2 and 3, Table 3); the running text of Section 5 now carries fewer than half as many numbers as before. Added or changed text is blue. The general response gives the four findings in four short items.

Soundness. The review rates soundness 1 without naming a specific concern. Reviewer r6SB raised five concrete technical questions; we answered each with a new analysis, added a sixth, and no finding changed (Section 5.5, Appendix G). If you have a specific concern about the protocol, a claim or a number, we will address it directly.

AI use and the ethics flag. The ICLR 2027 AI policy for authors asks authors to disclose LLM use and holds them responsible for the content; the violation it names is "a substantial falsehood, instance of plagiarism, or misrepresentation produced by an LLM". Our AI use statement (p. 11 of the revision) lists every research subtask in which a language model was used, including drafting and editing parts of the text, and covers the revision. The authors designed the conceptual framework, the five questions and the oracle bracket, reviewed all model-assisted code and text, and take responsibility for every claim. Every number quoted in the text is checked by script against the result files, and every reference was matched against its Crossref, arXiv or proceedings record; check_paper_numbers.py in the supplementary material reruns the number check. That check compares the text with the results; it could not catch a seeding defect in our Weak SINDy wrapper, which our own repeated-fit check found during the revision. It is corrected, and Table 32 compares the numbers before and after.

Reviewer_UMaiOpenReview ↗

With all due respect

This work was impenetrable. There were no specific points since the writing was so bad I could not adequately assess any significant part of it. I urge the ACs to please try to parse this paper. See examples below:

"Blind methods fit the training orbit but do not identify the form; given the form, the data suffice at low noise. At 1% noise the trajectory-fit oracle predicts a new initial condition on 88% of the 60 predictable ODEBench systems and the best blind method on 37%. At low noise the advantage lies in the functional terms, not in the graph."

"Graph F1 rewards the complete graph; balanced accuracy and AUROC do not. Every two-dimensional ODEBench system is fully coupled, so the complete graph scores F1 0.955 and no read-out beats it by more than 0.014; balanced accuracy and pooled AUROC still separate read-outs from it, and on the generated suite they do so at every noise level."

"Thetest suite is ODEBench (d’Ascoli et al., 2024): 63 systems (23 one-, 28 two-, 10 three- and 2 four-dimensional), generated with ODEBench’s own code and defaults (LSODA, t ∈ [0, 10], 150 samples, multiplicative noise ̃x = x(1 + σε) at σ ∈ {0, 0.01, 0.05}, first sample exact); it shares Lorenz, Van der Pol and the damped harmonic oscillator with the development suite, which can only favour the blind methods. Every method is fitted to the first 70% (n = 105 samples) of ODEBench’s first initial condition and to nothing else. Two further suites use ODEBench’s format and code: a hand-written suite of 12 systems (d = 3–5; food webs, a repressilator, a toggle switch, coupled oscillators, linear cascades) written after a first ODEBench analysis, and a random suite of 60 systems (d = 3–5, 20 each) drawn by a fixed generator and seed (one or two parents per damped variable, through linear, bilinear, quadratic, sinusoidal or Hill terms) and kept iff they stay bounded, are not forecast by persistence without noise, have an absent and a present edge, and are predictable. Its generator, acceptance rule and analysis plan were fixed before any method was run on it (prereg_random_suite.md; self-attested, but regenerable from its seed and hash)."

The level of clarity, and thus the ability to understand what is actually presented has not changed in revision. Perhaps AI can read this and form an opinion. When ICLR starts catering to AI readers and not human readers then I'd be happy to revisit my assessment.

Reviewer_2QmzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a new benchmark that evaluates models for learning dynamical systems. In contrast to traditional evaluations, this benchmark looks at five different aspects of performance: a held-out forecast, a new initial condition, a state perturbation, a clamp do-intervention, and the causal graph. The benchmark includes three noise settings, 12 methods, and 63 ODEBench systems. The evaluations provide insights about the capabilities and limitations of the current methods.

Critical strengths

The proposed benchmark introduces a much finer-grained and thorough evaluation of the current methods for modeling dynamical systems. By doing so, the scores represent a more meaningful assessment linked to real-world tasks for which those systems might be used.

The insights from this evaluation point to gaps in the literature and might encourage new developments in unexplored directions.

Critical weaknesses

I think the presentation of the paper could be improved. Section 5 is very dense; paragraphs contain many different numbers. It would be better if the results were presented in tables and figures, and the paragraphs themselves discussed only the general, interesting insights. The big Table 2 is itself quite hard to parse. Maybe some figures that would show how the performance changes at different noise levels? Or figures where the methods are grouped by type? Something that would be easier to parse than a dense table and paragraphs.

Questions for authors

Could you improve the presentation so it's easier to understand the key takeaways?

3 comments

Response to Reviewer 2Qmz

Thank you. We adopted each suggestion:

  • Figures by noise level and method type. Figures 2 and 3 group every learned method by class with one marker per noise level, so each method's change with noise reads along its row; the oracle is a shaded band and the trivial reference a grey line. Figure 2 covers forecasts, new initial conditions and perturbations on ODEBench. Figure 3 covers the clamp and the graph on the random suite, and its lower panel splits each clamp response into size and shape.
  • Table 2. Replaced in the main text by Table 3: for each question, the trivial reference, the best learned method and the oracle. The full table is Table 11 (Appendix E).
  • Section 5. Each subsection opens with its finding in one sentence and keeps only the numbers that support it; the rest moved to tables and Appendix E. Its running text now carries about 125 numbers, down from about 250, and its densest paragraph 15, down from 40.
  • Key takeaways. Table 1 lists the four findings and the reporting practice each implies; the general response states them in four short items.

If a passage still reads densely, please point us to it and we will revise it. If the revision resolves your presentation concern, we would be grateful if you would consider raising your presentation score and rating.

Thank you for your response and the updates to the manuscript. I think the presentation has improved, but it still reads like a really dense, technical report rather than a conference paper. It is difficult to point out specific paragraphs because the issue is present throughout the paper. There are at least three different levels where improvements would need to be made:

  1. Information selection and organization: which information is included in the main text, which is omitted, and which is in the appendix
  2. Sentence-level writing. Some sentences just sound awkward
  3. Word-level writing: Some word choices are not appropriate for the context

Examples for 1:

  • Some information in lines 176-186 does not seem to belong in the main text
  • Similarly for lines 220-226

Examples for 2:

  • Lines 49-51: "Interventions have been posed to regularised neural ODEs (Aliee et al., 2021) and to temporal causal models (Thumm et al., 2026); what is missing is to pose them, and the graph, on an external equation-discovery benchmark, across method classes, against references that show what knowing nothing, and knowing the equation’s form, would score."
  • Lines 63-64: "; and (iv) four findings, which methods designed against these failures, per-system tuning and repeated noise draws leave in place, each with the reporting practice it implies (Table 1)."

Examples for 3:

  • Lines 408-409: "no blind method answers a clamp on a system"
  • Line 426: "The verdict survives three changes:"

The paper is not simply optimized for science communication. That is why I will not be raising my score. Please take a look at other benchmarking papers, such as SRBench and DynaDojo (which I have now realized should also be cited), and see how much easier and nicer they are to read.

Also, I find it very odd that you have a script that "checks every number we quote". Ideally, you should have deterministic scripts that take CSV files and generate LaTeX tables. You should verify that the table-generation code is correct. That is all. A separate number checker would only be needed if the tables themselves are generated by an LLM from raw CSV files containing the results, which does not sound like a good solution.

Response to Reviewer 2Qmz (follow-up)

Thank you for the examples. We said we would revise any passage you pointed to, and the latest version uploaded does so. Blue still marks all text that differs from the submission, so the latest edits are not marked separately; the line numbers below are those of the new PDF.

Your six examples

  • Lines 176–186 (what was fixed when) are now three sentences (l. 182–188); the four tiers are listed in Appendix A.
  • Lines 220–226 (ablations and variants) are now one sentence (l. 225–227); the definitions are in Appendix A.
  • Lines 49–51 are now two sentences (l. 50–54): "Predictions under intervention have been tested for regularised neural ODEs and for temporal causal models. What is missing is such a test across method classes on an established equation-discovery benchmark, scored together with the graph and against two references: a trivial prediction, and a model that is given the functional form of the equations."
  • Lines 63–64: each contribution is now its own sentence (l. 59–67).
  • Lines 408–409 now read: "With noise, on systems with an absent edge, no blind method predicts the response to a clamp detectably better than predicting no response."
  • Line 426 now reads: "This result holds under three changes to the protocol: ..."

Beyond the examples. You note that the problem runs through the paper, so we made the same three kinds of edit elsewhere in the main text. The data paragraph of Section 3 is shorter (l. 151–161), with the generator's details in Appendix A. The four findings in the Introduction (l. 69–89) are restated in shorter sentences. "Answers a clamp" and "verdict" no longer occur in the main text. The rest of Sections 3 to 5 is as before, and we do not claim that the paper now reads as easily as SRBench or DynaDojo.

Citations. SRBench was already cited in the Introduction and in Section 2 (l. 46 and 103 of the previous version). DynaDojo was missing; it is now cited in Section 2 (l. 109–111).

Number checking. We agree with the principle, and the tables follow it. Deterministic scripts write 43 of the 47 tables, and every figure, from the saved results (aggregate.py, figures.py and rebuttal/paper_tables.py in benchmark/). No entry in those tables is typed by hand or produced by a language model. The other four tables (1, 2, 4 and 6) are written by hand; they hold text and settings and quote a few results, most of them in Table 4, which sets four predeclared criteria beside their results. The separate checker is for what a script does not generate: results quoted in the running text and in the hand-written tables. The scripts themselves were written with the coding assistance disclosed in our AI use statement; Appendix B describes the checks on the scoring code. Contribution (iii) described this poorly and is reworded (l. 63–65).

We respect your decision on the score, and thank you for the time you have given this paper.

Comments on the paper

2 comments

General response to all reviewers

Thank you to all three reviewers. 2Qmz found the benchmark "a much finer-grained and thorough evaluation" of these methods, and r6SB called its open reporting of planned criteria, including those that failed, "a commendable level of rigor and openness". The revision is uploaded: added or changed text is blue, and deleted text is not shown. It addresses the two concerns the reviews raised, presentation and robustness. Table numbers have shifted; we give the revised numbers and, where a review cites one, the submitted number as well. The abstract is shortened: it states the same findings with fewer numbers and adds the results of the oracle ladder, the tuning and the repeated noise draws. Three things left the main text and so carry no blue: the full results table (Table 2 of the submission, now Table 11 in Appendix E), the sign tests and secondary numbers of Section 5 (now Appendix E), and the Discussion's paragraph on what to report, which became Table 1.

The four findings

  1. Fitting is not identifying. Learned ("blind") models reproduce the training trajectory but not the vector field off it: at 1% noise the best one predicts a new initial condition on 37% of the 60 predictable ODEBench systems, while a reference that knows only which terms appear, with every coefficient fitted to the same 105 samples, does so on 83%.
  2. Interventions. With noise, on systems with an absent edge ("sparse" systems), no learned method's median error under a clamp intervention is detectably below that of predicting no response; the trajectory-fit oracle's stays at or below 0.30.
  3. Graph metrics. On ODEBench the complete graph scores F1 = 0.955 and no method beats it by more than 0.014; balanced accuracy and AUROC do separate methods.
  4. Read-outs. A threshold-free AUROC separates a bad threshold from a bad model: the Neural ODE's Jacobian ranks true edges above absent ones although its thresholded graph is nearly complete.

Presentation (all reviews). Each subsection of Section 5 now opens with its finding in words, and the supporting numbers sit in tables and figures: Table 1 (the findings and the reporting practice each implies), Figures 2 and 3 (every method grouped by class, one marker per noise level, oracle and trivial reference drawn in), and Table 3 (ODEBench at a glance; the full table is now in Appendix E). The running text of Section 5 carries fewer than half as many numbers as before (about 125, down from about 250), in shorter sentences.

Robustness: six new analyses (Section 5.5, Appendix G). Five were requested by Reviewer r6SB. Each design, and the rule by which its result would change a finding, was written down before any was run (prereg_rebuttal.md in the supplement; self-attested, like the random suite's plan). All results are reported.

  1. Oracle ladder: at low noise the functional terms carry the oracle's advantage; sparse regression given only the causal graph stays below the best learned method.
  2. Discrete-time read-outs against the true Δt-step graph: indirect dependences do not explain their false positives.
  3. Per-system tuning on the training window: no tuned method beats no response on sparse systems with noise, and the best tuned new-IC accuracy at 1% noise is 38%, against 37% by default.
  4. Five to ten noise draws with hierarchical intervals: every sparse-system clamp verdict holds (the other questions: Table 47).
  5. Sample budget versus coverage: a second trajectory helps when each trajectory is observed long enough.
  6. Zero-response test (new): where the true response is exactly zero, the oracles, which know the graph, predict none by construction (AUROC at least 0.96 on the random suite); with noise the best learned method reaches 0.70.

What changed. No finding changed under its predeclared rule. Three statements were refined:

  • Finding 1 now places the oracle's advantage, at low noise, in the functional terms, not the graph.
  • The two-trajectory ablation had confounded coverage with sample size; Section 5.1 now separates them (item 5).
  • A seeding defect in our Weak SINDy wrapper is fixed, and all Weak SINDy numbers come from seeded runs (Table 32). Its accuracy on a new initial condition is 55/15/10% rather than 62/22/3% (σ = 0/0.01/0.05), and over the replicates its ODEBench AUROC with noise no longer excludes chance. Finding 4 therefore cites Weak SINDy only without noise, and rests on the random suite, with ODEBench's ten sparse systems as consistent but descriptive evidence.

We welcome further questions.

Area_Chair_NnqdOpenReview ↗

Please join the author-reviewer discussion

Dear all reviewers,

Our author and reviewer discussion will end soon in 18 November (AOE). If you have not yet joined the discussion, now it is the time to start it. We highly appreciate your time on discussing and exchanging thoughts to keep our ICLR standards. Many thanks!

Bests,
Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.