- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PDEDiscoveryBench, a benchmark for evaluating differential-equation discovery from spatiotemporal observations while explicitly controlling the amount of prior scientific information available to each method. The benchmark contains 337 systems, including 332 systems with reference equations and five unlabeled CFD blood-flow fields. The labeled collection spans ODEs and PDEs from multiple families, although a substantial fraction consists of 1D reaction–diffusion and conservation-law systems
Critical strengths
- Controlling scientific prior is a genuinely useful benchmark contribution. Existing benchmarks typically fix the prior implicitly, whereas PDEDiscoveryBench makes it an experimental variable and translates the same conceptual prior into each method’s native representation. This is a useful framework for comparing fundamentally different discovery paradigms.
- Large-scale evaluation across methods, priors, and noise. The benchmark contains 337 systems and evaluates seven methods spanning sparse regression, genetic/RL search, and LLM-based agents. More importantly, the paper does not merely report an aggregate leaderboard; it studies how results change as both prior information and noise change.
Critical weaknesses
- The claimed cross-method comparison is substantially confounded by unequal side information and budgets. This is my largest concern. SINDy and WSINDy receive the ground-truth number of terms K_{GT} as a support cap and undergo family/noise/prior-specific hyperparameter tuning, including clean inner-validation information for noisy tiers. Meanwhile, the LLM agents must infer temporal order and derivative estimators themselves, use retries, and have agent-specific budgets; DISCOVER and LLM-PDESR run once. The authors explicitly state that budgets are “method-native, not equalized.”
- Dataset diversity is considerably smaller than “337 systems” initially suggests.
Questions for authors
Could you please explain or address the weaknesses raised above?
4 comments
Looking forward to your feedback
Dear Reviewer nMZ7,
Thank you again for your constructive review. We hope our response and the revised PDF (changes in blue) have addressed your concerns. In particular, every sparse-regression and search method is now tuned on inner validation, the agents are scored on their first attempt, the noisy tiers are benchmarked without the reference term count and without clean validation (Table 12), and the Results now give family-weighted results next to the pooled ones.
As the discussion period ends on November 18 (AoE), please let us know if any concern remains, and we will answer it before then. If our response has resolved your concerns, we would be grateful if you would consider updating your score.
Best regards, The Authors
A note on the shared concerns
Dear Reviewer nMZ7,
We wanted to share one update before the discussion closes. Reviewer Ufr1 raised the same concern you described as your largest: SINDy and WSINDy receive the reference term count and clean inner validation, and the agents' retries are counted. After reading our response and the revision, Reviewer Ufr1 wrote that all their concerns had been well addressed and raised their score.
The changes that answered this concern apply equally to your review: every sparse-regression and search method is now tuned on inner validation within the training data, the agents are scored on their first attempt, and the noisy tiers are benchmarked without the reference term count and without clean validation (Table 12). On dataset diversity, the Results now give family-weighted results next to the pooled ones.
If any part of your concern remains, we would be glad to answer it before November 18 (AoE). Otherwise, we would be grateful if you would consider updating your score.
Best regards, The Authors
Final note before the discussion closes
Dear Reviewer nMZ7,
A brief final note, as the discussion closes today (November 18, AoE). Both other reviewers, whose reviews also questioned the side information and the fairness of the comparison, have now said that the revision addresses their concerns and raised their scores.
We understand this is a busy period. If you have any remaining question about the side-information ablation (Table 12), the tuning of every method on inner validation, or the family-weighted results, we would be glad to answer it today. If our response has resolved your concerns, we would be grateful if you would consider updating your score.
Best regards, The Authors
Response to Reviewer nMZ7
Thank you for recognizing the value of controlling prior information, and for pointing out the remaining differences between methods. We agree that these need to be clearer in the paper.
We checked whether our main findings remain after correcting these advantages. Prior information still improves recovery in most comparisons, with exceptions:
W1. Unequal side information and budgets. We agree that these differences matter, but we think it is helpful to distinguish between method-specific settings and additional information provided by the benchmark.
For example, SINDy uses sparsity thresholds and regularization, WSINDy also needs to select weak-form integration settings, SGA-PDE uses population size and generations, while DISCOVER involves reinforcement-learning and PINN training. LLM agents, on the other hand, choose their own numerical procedures within a tool-call budget.
These settings do not have direct equivalents across methods. Even if we set every method to run for 100 iterations, one STLSQ iteration is very different from one evolutionary generation or one RL training iteration. We could impose a common time budget, but that would be a separate resource-controlled comparison rather than a common hyperparameter configuration.
Our goal is therefore to evaluate each method using its native procedure while making these computational differences explicit.
To make the tuning consistent, every sparse-regression and search method is now tuned on inner validation within the training data, and these tuned runs are our main results:
We also agree that some of our original settings provided additional information or attempts, beyond the prior cards:
The submission removed the term count only on clean data (Table 11); the revision benchmarks the missing noisy setups (Table 12). WSINDy's lead under noise depends on the cap and clean validation, so we no longer claim that the original gaps establish relative noise robustness.
W2. Diversity of equation families. We agree that the family distribution matters. The 332 labeled systems form 19 families, and 175 are 1D reaction-diffusion or viscous conservation-law cases. We already report the family counts in Table 1 and family-weighted results in Table 10, but we agree that these findings were not sufficiently emphasized in the main discussion. The revised Results now give the family-weighted count next to the pooled one: L3 exceeds L1 in 21 of 25 method-tier cells with the 19 families weighted equally (22 of 25 pooled). The family-weighted L3−L1 interval excludes zero in only 8 of 25 cells (17 pooled), and the limitations now say so.