- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents GEMOT, an LLM-based framework for automatically developing and refining mechanistic models of biological systems through iterative hypothesis generation and numerical evaluation. The approach is evaluated on 18 published biological modelling tasks, demonstrating competitive model quality and promising predictive capabilities on selected held-out experiments. The manuscript is well written, clearly motivated, and supported by extensive experimentation and detailed methodological documentation.
The main limitations concern the consistency of generalization across benchmarks, the difficulty of establishing genuine mechanistic novelty, and the limited evidence of improvements over standalone frontier LLM agents. Overall, this is a valuable and timely contribution with promising results, although some claims would benefit from stronger experimental support.
Critical strengths
-
The paper is clearly written, well motivated, and supported by extensive methodological details, making the proposed framework easy to follow and its experimental findings easier to assess.
-
The authors evaluate the framework across 18 diverse, real-world biological modelling problems, comparing against established reference models. The inclusion of repeated runs, held-out experimental validation, and ablation studies across different LLMs and agent configurations provides substantial evidence of its practical capabilities.
-
The proposed architecture separates LLM-driven hypothesis generation from numerical evaluation, enabling transparent and traceable model construction. This is particularly valuable for scientific applications, where numerical reliability and interpretability are essential, and provides a promising foundation for automated mechanistic discovery.
Critical weaknesses
-
Although the paper includes held-out experimental evaluations and discusses several biologically plausible hypotheses, the proposed mechanisms are not independently validated through new biological experiments. Moreover, the qualitative analysis focuses on selected representative examples, making it difficult to assess how consistently the framework generates biologically meaningful hypotheses. A more systematic evaluation of mechanistic plausibility would strengthen the scientific claims.
-
The the proposed framework provides valuable architectural improvements, however the ablation experiments show that a standalone GPT-6-Astra agent achieves comparable, and in some cases better, predictive performance. This raises questions about the extent to which the observed improvements can be attributed to the proposed framework rather than the capabilities of the underlying LLM. A more detailed analysis of the individual architectural components would help clarify their contributions.
-
While the authors compare GLM-5.3-Flash and GPT-6-Astra in their ablation experiments, the main evaluation across 18 biological problems relies exclusively on GLM-5.3-Flash. Evaluating the framework more extensively with different LLM families would help establish whether the observed performance generalizes across models and better isolate the contributions of the proposed architecture from those of the underlying LLM.
Questions for authors
-
Could the authors elaborate on the motivation for selecting GLM-5.3-Flash for the main evaluation across 18 biological problems? Was this decision primarily driven by computational cost, or were other factors considered? Given the stronger performance observed with GPT-6-Astra in some experiments, it would be interesting to understand how the choice of LLM affects the framework's overall capabilities.
-
Could the authors further clarify which components of GEMOT provide the most significant improvements over a general-purpose LLM agent? In particular, are the main benefits related to model quality, reliability, computational efficiency, or reproducibility?
3 comments
I appreciate the work done by the authors to address most of my presented concerns. However, I decide to maintain my score.
We thank the reviewer for their engagement and are pleased that our response addressed most of their concerns.
W1: Plausibility shown only for selected examples. We now compare every model with curated knowledge (OmniPath, Reactome) on 13 problems, three of them new, not selected examples (Sec. 4.3, Apps. D.4 and F). Where the agents’ models depart from it, an LLM-assisted audit suggests gaps in the curated networks, justifiable coarse-graining (e.g. EGFR acting on Akt without PI3K), genuine misses and limitations of the analysis; how to judge such departures remains an open problem (Sec. 4.3). The structural comparison (Fig. 2, Table 11) already covers all runs. We agree that experimental validation is the definitive test.
W2: A bare GPT-6-Astra agent performs comparably. In response to the concerns about organisation (Reviewer WwtP, W1), we removed the Fujita bare-agent configurations and the analysis of hardcoded literals to shorten the text and improve the narrative flow. The revision instead tests the harness on the six largest problems with published references (Sec. 4.6), where it matters:
(i) With GLM, barebones agents passed 2/30 runs against 29/30 with gemot, and the harness’s feedback made search more efficient (Fig. 5).
(ii) gemot lets an open-weight model reach 98% of the best frontier improvement at under 3 USD per model, compared with over 6 USD (Fig. 3e).
(iii) gemot computes scores outside the LLM, and the agent cannot alter them, so it cannot fabricate results; this makes results more trustworthy and reduces the need for human audit (App. B.4).
W3: Single LLM family in the main evaluation. GLM-5.3-Flash was the first open-weight model that consistently solved all problems, though no longer the only one. We evaluated Astra on Fujita only because of cost, which grows on larger problems where fits can exceed its 1 h cache timeout. The revision adds Claude Fable 5.2 on Fujita; both frontier LLMs were faster than GLM but not more predictive (6/10 against 10/15 models beat the reference; Sec. 4.4). We added DeepSeek v4.1 Flash (open-weight), GPT-6 luna and Claude Haiku 5.5 on the six largest problems with published references (Fig. 5): DeepSeek and Haiku passed 29/30 runs, as GLM did, and luna 14/30, most of its failed runs exhausting the token budget.
Q1: Why GLM-5.3-Flash? It was the first LLM that consistently solved the original 18 problems, and at under 3 USD per model (Fig. 3e) it made 110 runs across 22 problems affordable. See also W3.
Q2: Which components matter most? The improvement likely results from many small, compounding gains rather than any single component, and the computational cost of each run makes a comprehensive deconstruction of performance difficult. Based on the ablations (Figs. 3 and 5):
Performance: barebones agents passed 2/30 runs against 29/30 with gemot on the six largest problems with published references (Sec. 4.6).
Cost: open-weight models become competitive (98% of the best frontier improvement at less than half the cost).
Auditability: gemot versions every model and computes scores outside the LLM.
Search signals: withholding fix pointers or progress signals left success unchanged but made the search less efficient (Fig. 5): without fix pointers, agents had to derive revisions themselves; without progress signals, they lost track of their best model and branched off stale versions.
Removing the curator had little effect (held-out wins 4/5 vs. 3/5), consistent with the LLM’s implicit knowledge being sufficient for Fujita. Biological context mattered for prediction on some systems but not others (Sec. 4.5).