- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Model Discovery Agent (MDA), a framework for autonomously discovering mathematical models that describe unknown scientific mechanisms from limited experimental data. MDA integrates model structure generation using large language models (LLMs), model evaluation through Bayesian inference, and information-theoretic active experimental design.
Conventional Bayesian model selection assumes a predefined, fixed set of candidate models and therefore cannot identify the correct model when it is absent from the candidate set. To address such M-open model discovery problems, MDA dynamically expands the hypothesis space by using an LLM to propose new model structures when existing models fail to adequately explain the observed data.
MDA represents an unknown system using a hierarchical state-space modeling framework consisting of three components: (1) model structure, (2) model parameters, and (3) the temporal evolution of latent states. For candidate models proposed by the LLM, parameter estimation and model evaluation are performed using numerical optimization, sequential Monte Carlo (SMC), and Bayesian information criterion (BIC) approximations, as appropriate. The posterior probabilities of the candidate models are then updated. MDA also selects experimental conditions that are expected to most effectively discriminate among competing models, using an approximation to expected information gain based on disagreement between model predictions. Based on the experimental outcomes, MDA iteratively expands or prunes the candidate model set and updates parameter estimates, thereby improving predictive performance under a limited experimental budget. The effectiveness of the proposed framework is evaluated on three scientific model discovery tasks of varying complexity.
- CHEMBENCH: This benchmark involves discovering enzyme reaction rate laws from seven experimental variables. Across 57 reaction mechanisms, MDA demonstrates greater sample efficiency than existing LLM-based symbolic regression and in-context learning approaches.
- GLUCOSEBENCH: This benchmark involves learning dynamical models of blood glucose regulation from partially observed data generated by a physiological simulator of type 1 diabetes. MDA introduces additional compartments representing latent states and delayed responses, achieving better predictive performance than ARX, SINDYc, and direct LLM-based prediction.
- NEURONBENCH: This benchmark focuses on discovering neuronal dynamics governed by stochastic Hodgkin–Huxley models. MDA models ion-channel mechanisms and stochastic latent-state dynamics, outperforming deterministic ODE-based models and direct LLM-based prediction. In particular, the combination of stochastic differential equations (SDEs) and SMC achieves high accuracy in predicting neuronal firing characteristics. The main contribution of this study is the development of a unified framework that combines flexible LLM-based hypothesis generation with inference and experimental design grounded in explicit probabilistic models. This framework accommodates diverse classes of mathematical models, including algebraic models, deterministic ODEs, and stochastic SDEs, within a common model discovery process.
Critical strengths
-
An Integrated Framework for Scientific Discovery Combining LLM-Based Hypothesis Generation, Bayesian Inference, and Active Experimental Design The primary strength of this study lies in its integration of flexible mathematical model generation using LLMs, model evaluation based on Bayesian inference, and information-theoretic active experimental design. In particular, the mechanism for dynamically expanding the hypothesis space using an LLM when existing candidate models fail to adequately explain the observed data represents a promising approach to overcoming the limitations of conventional Bayesian model selection, which assumes a fixed set of candidate models. A distinctive feature of this approach is that LLMs are employed to generate explicit and executable scientific models rather than being used directly as predictors.
-
Applicability to Scientific Systems with Different Levels of Complexity Another strength is the application of the proposed framework to three distinct types of problems: static chemical reaction rate laws, partially observed deterministic ODE systems, and stochastic neuronal dynamics. In particular, its ability to handle complex systems involving latent states and stochastic dynamics demonstrates the broad applicability of the framework as a scientific discovery agent. Furthermore, the reported improvements in sample efficiency over existing methods across multiple benchmarks suggest that the proposed approach is promising for the practical challenge of constructing scientific models under limited experimental budgets.
Overall, this paper has two notable strengths: its integrated framework combining LLM-based model structure exploration with statistical inference and experimental design, and its applicability to a diverse range of scientific systems. These contributions have the potential to advance the development of autonomous scientific discovery systems.
Critical weaknesses
- Insufficient Experimental Validation of the Core Contribution and Potential Contamination from LLM Pretraining Knowledge One of the key features of this study is its ability to dynamically expand the hypothesis space using an LLM, thereby addressing M-open model discovery problems in which the true model is not included in the initial candidate set. However, the current experimental results do not sufficiently demonstrate the extent to which this mechanism contributes to the observed performance improvements. For example, in CHEMBENCH, the initial candidate set includes nine canonical reaction mechanisms, suggesting that the true model may already be present in the candidate set for some tasks.
More fundamentally, the LLM may have already encountered the ground-truth equations or closely related mathematical models during pretraining. This concern is particularly relevant to well-established scientific models, such as Hodgkin–Huxley models and classical enzyme kinetics. Consequently, the reported performance may partly reflect the retrieval or recombination of previously learned scientific knowledge rather than the discovery of genuinely unknown mechanisms from experimental observations.
To address these concerns, the authors should conduct additional experiments in which the ground-truth model is explicitly excluded from the initial candidate set, together with ablation studies that disable LLM-based model space expansion. More importantly, the authors should evaluate MDA on novel synthetic systems with previously unseen governing equations or systematically modified mechanisms that are unlikely to have appeared in the LLM's pretraining data. Such experiments would help distinguish genuine model discovery capabilities from the reproduction of prior scientific knowledge.
- Potential Misalignment Between Experimental Design and the Final Prediction Objective While the ultimate evaluation objective is to minimize prediction errors under unseen intervention conditions, the experimental design strategy relies on an approximation to expected information gain (EIG) aimed at discriminating among competing models. However, experiments that are informative for model discrimination are not necessarily optimal for improving predictive performance. The authors should clarify the interpretation of posterior model probabilities under adaptive model space exploration. Furthermore, it would be valuable to compare the current model-discrimination-based experimental design strategy with alternative approaches that directly aim to improve predictive performance under unseen intervention conditions.
Questions for authors
-
Potential Influence of LLM Pretraining Knowledge on Model Discovery The LLM may have already encountered the ground-truth equations or closely related mathematical models used in the benchmarks during pretraining. This concern is particularly relevant to well-established scientific models, such as Hodgkin–Huxley models and classical enzyme kinetics. Consequently, the reported performance may partly reflect the reproduction or recombination of previously acquired scientific knowledge rather than the discovery of genuinely unknown mechanisms from experimental observations. How do the authors distinguish genuine model discovery from the retrieval or reconstruction of scientific knowledge acquired during LLM pretraining? Have the authors evaluated MDA on novel synthetic systems governed by equations or mechanisms that are unlikely to have appeared in the LLM's training data? Such experiments would help clarify the extent to which MDA can discover previously unseen scientific mechanisms rather than reproduce established models.
-
Objective Function for Active Experimental Design In this paper, the ultimate evaluation objective is predictive performance under unseen intervention conditions. However, the experimental design strategy relies on an approximation to expected information gain (EIG) aimed at discriminating among competing candidate models.
Experiments that are informative for model discrimination are not necessarily optimal for reducing future prediction errors. How do the authors justify this potential mismatch between the experimental design objective and the final evaluation objective? Have the authors compared their model-disagreement-based experimental design strategy with alternative approaches that directly maximize the expected reduction in future predictive loss?
- Accuracy of Scientific Mechanism Discovery Although mechanistic model discovery is presented as a primary goal of this study, the evaluation metrics focus mainly on prediction errors under unseen intervention conditions.
Have the authors quantitatively evaluated how accurately the discovered models recover the true underlying data-generating mechanisms? Since the ground-truth mechanisms are known in the benchmarks used in this study, it should be possible to assess the recovery of model structures, dependencies among latent variables, reaction mechanisms, and other mechanistic components. Furthermore, how do the authors evaluate models that are predictively equivalent but represent different underlying mechanisms? Clarifying this distinction would help establish whether MDA achieves genuine mechanistic discovery or primarily identifies models with strong predictive performance.
- Reproducibility and Computational Cost MDA involves several sources of variability, including LLM-based model proposals, numerical optimization, and SMC inference.
How robust are the reported improvements in sample efficiency across different random seeds and variations in LLM-generated model proposals? It would be helpful to report the variability in predictive performance and model discovery success rates across independent runs.
In addition, could the authors provide the number of LLM calls, model evaluations, and computational time required for each benchmark? Comparisons that account for computational cost, rather than experimental sample efficiency alone, would provide a more comprehensive assessment of the practical advantages of MDA.