- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies discovery when the current model class omits a mechanism shared by all its candidates. A learned policy jointly proposes an edit from a predefined grammar and an experiment intended to distinguish the revised class from the current one. Class-level testing requires a shared parameterization across experiments, and ideal sequential evidence governs revision. On 400 held-out dynamical environments, exact recovery is 89.5%, compared with 79.5% for falsification with independent edit and experiment proposals. Additional evaluations cover distribution shifts, out-of-grammar detection, public ODE collections, and simulated physics environments. The experiments demonstrate model revision in executable environments rather than newly conducted physical experiments.
Critical strengths
-
The shared-parameter class-level formulation addresses a concrete discovery failure. Requiring one parameterization to explain multiple experiments prevents separate refits from absorbing evidence of structural misspecification. Coupling this criterion to an edit and a diagnostic experiment gives a substantive framework for reasoning about when revision is necessary.
-
The empirical evaluation extends beyond one controlled task family. Recovery, predictive error, discovery costs, distribution shifts, and external dynamical collections are all examined. These results support the tested pipeline’s utility and provide more information than a single recovery endpoint.
Critical weaknesses
-
The novelty should be positioned more precisely within a predefined edit grammar. The paper studies revision beyond the initial class, while the candidate revisions remain expressible in a supplied grammar. The shared-parameter criterion and the joint-versus-independent comparison provide a meaningful contribution in this setting. A sharper comparison with active model selection, structural search, and sequential class testing would clarify which elements supply the new knowledge and prevent a broader interpretation as unrestricted mechanism invention.
-
Component-level attribution requires clearer training-resource accounting. The joint policy uses teacher-search supervision followed by return optimization. The reported experiment counts, candidate fits, and discovery-time compute do not establish equivalence of teacher supervision, training resources, and fitting configurations for the relevant learned controls. This limits attribution of the recovery gain specifically to joint decision learning, although the recovery and transfer results remain evidence for the complete tested pipeline.
Questions for authors
- Which aspect of the proposed objective or policy goes beyond active selection over the same expanded model library with the same sequential class test? Can you isolate that aspect while holding the edit grammar and available experiments fixed?
- For the joint and independent policies, please compare teacher-search access, supervision volume, training objectives and compute, architecture capacity, and fitting settings. Does the advantage persist when these are matched?
- I encourage the authors to relate the proposed class-revision loop to interactive biological model-reconstruction environments such as SciGym[1]. Explaining what is added by an explicit revision grammar, shared-parameter class test, and coupled edit–experiment policy would make the contribution and its intended experimental use clearer :)
- For one external benchmark task, what practical experimental operations, measurements, and costs do the available queries represent? Which restrictions of that application are encoded in the benchmark, and which remain simulator assumptions?
[1] Duan, H., Lu, S., Harrigan, C. F., Desai, N., Lu, J., Koziarski, M., ... & Maddison, C. (2026). Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab. Advances in Neural Information Processing Systems, 38.
Thank you for your careful reading of our response and for raising your score. We sincerely appreciate your time and constructive feedback, which has helped us improve the paper.