- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Origo, a multi-agent framework that integrates tools based on expert-designed strategies to automate the construction of genome-scale metabolic model (GEM) for producing target molecules and the identification of enzyme candidates. It also introduces DLBench, a benchmark for evaluating these capabilities.
Critical strengths
-
The motivation for developing the framework is clear. Although the framework consists of numerous components, each component is described in detail.
-
The work involves substantial engineering effort, and the authors develop a dedicated benchmark to evaluate their proposed framework.
Critical weaknesses
-
The Related Work section does not clearly articulate the specific differences between this work and prior studies. Furthermore, the Method section lacks sufficient justification for the design choices behind the individual components of Origo and DLBench. As a result, the paper reads more like a technical report describing Origo and DLBench than a research paper.
-
The paper primarily presents the evaluation results of Origo without providing detailed analyses or ablation studies. Furthermore, to establish DLBench as a meaningful benchmark, the number of benchmark instances should be sufficiently large, and validation by human experts, rather than relying solely on an LLM judge, is necessary.
Questions for authors
-
Although Origo appears to be primarily an engineering effort, what specific research problems does it aim to address, and how does the proposed framework solve them? Could the authors provide comprehensive ablation studies and analyses to justify the design choices behind each component?
-
Could the authors substantially increase the number of benchmark instances and provide a human expert baseline? Could they also provide qualitative comparisons with existing models
4 comments
W2 and Q1
Thank you for requesting ablations to examine the design choices. We added comparisons that change agent organization, knowledge access, MCP availability, and tool exposure. All use the same ten targets sampled from the original set with seed 11062026:
| Experimental Group | XML Outputs | SBML Validity | Growth Success | Reference Production Success | Mean Memote Score | Mean LLM Score |
|---|---|---|---|---|---|---|
| Origo (original paper) | 10/10 | 100% | 100% | 100% | 0.62 | 7.20 |
| Single agent | 7/10 | 70% | 70% | 20% | 0.57 | 6.00 |
| No knowledge | 10/10 | 100% | 100% | 100% | 0.48 | 6.80 |
| No MCP | 2/10 | 10% | 0% | 0% | 0.37 | 2.00 |
| All tools/task | 9/10 | 90% | 90% | 80% | 0.56 | 6.22 |
Success percentages use ten target requests per condition. XML outputs are 10, 7, 10, 2, and 9, respectively. Memote is evaluated only for SBML-valid models. The Origo reference is taken from the original paper for the same sampled targets.
Removing MCP most strongly affects whether a valid, growing model is produced. Removing knowledge leaves the binary functional rates unchanged in this subset but lowers Memote, while the single-agent and all-tools-per-task variants reduce quality. Together, these results show that tool access, knowledge-guided construction, and task organization contribute at different stages of producing an executable and curated network.
We also ran Origo with eight alternative backbones on the same target subset to examine how the results depend on the underlying model:
| Method | N | SBML validity (%) | Growth (%) | Product formation (%) | Memote score (0-1; mean) | LLM overall score (1-10; mean) |
|---|---|---|---|---|---|---|
| Origo-DeepSeek-V4-Flash (Used in This study) | 10 | 100.00 | 100.00 | 100.00 | 0.62 | 7.20 |
| Origo-GPT-6-Astra | 10 | 100.00 | 100.00 | 100.00 | 0.61 | 6.80 |
| Origo-GPT-6.1-Sol | 10 | 100.00 | 100.00 | 100.00 | 0.61 | 6.90 |
| Origo-GPT-6-Luna | 10 | 100.00 | 100.00 | 100.00 | 0.61 | 7.20 |
| Origo-GLM-5.3 | 10 | 100.00 | 100.00 | 100.00 | 0.48 | 6.60 |
| Origo-GLM-5.3-Flash | 9 | 100.00 | 77.80 | 100.00 | 0.38 | 6.33 |
| Origo-Kimi-K3 | 10 | 100.00 | 100.00 | 100.00 | 0.52 | 6.90 |
| Origo-Qwen3.8-Max | 10 | 100.00 | 100.00 | 100.00 | 0.56 | 6.90 |
| Origo-Qwen3.8-Flash | 10 | 100.00 | 100.00 | 100.00 | 0.56 | 6.90 |
Validity and product formation reach 100% among available models for all eight alternatives. Growth is 100% except for GLM-5.3-Flash, at 7/9. Mean Memote ranges from 0.38 to 0.61. The framework therefore supports several backbones, with differences in curation and overall quality.
Manuscript changes: Appendix F adds the ablation study details (Figures S8–S9). Appendix G adds the backbone comparison (Figure S10).
W2 and Q2
Thank you for recommending human evaluation. We invited a biology PhD student and a rater with industry experience to conduct a double-blind assessment of 90 generated models, scoring all eight dimensions. Five additional LLM judges evaluated the same models. Together with the original judge, this gives the following comparison of six LLM evaluators and two human raters:
| Generating method | DeepSeek-V4-Pro(Original) | DeepSeek-V4.1-Flash | Kimi-K3 | Qwen3.8-Max | GPT-6 Luna | GPT-6.1 Sol | Biology PhD Student | Industry Expert |
|---|---|---|---|---|---|---|---|---|
| Origo | 7.20 | 7.00 | 6.80 | 6.60 | 5.70 | 5.20 | 8.20 | 7.20 |
| Biomni | 6.20 | 6.60 | 6.70 | 5.80 | 4.20 | 4.20 | 5.40 | 5.30 |
| STELLA | 2.40 | 2.30 | 2.30 | 2.10 | 1.50 | 1.60 | 1.90 | 1.60 |
| GPT-5.5 | 3.30 | 3.50 | 4.10 | 3.30 | 2.60 | 2.60 | 3.10 | 2.80 |
| DeepSeek-V4-Pro | 5.70 | 5.30 | 5.00 | 4.30 | 3.20 | 3.30 | 4.10 | 3.90 |
| GLM-5.2 | 4.60 | 4.90 | 4.80 | 4.20 | 3.40 | 3.50 | 4.10 | 3.70 |
| Qwen3.7-Max | 3.30 | 2.70 | 3.00 | 2.60 | 2.30 | 1.90 | 2.10 | 1.70 |
| Kimi-K2.6 | 4.60 | 4.40 | 4.10 | 3.10 | 3.10 | 2.70 | 5.30 | 3.20 |
| Gemini-3.5-Flash | 2.50 | 2.10 | 2.00 | 1.90 | 1.40 | 1.50 | 1.80 | 1.40 |
Overall quality on a 1–10 scale; means over ten models per method and judge. Bold entries mark column maxima.
To examine agreement beyond method-level means, we also compared Spearman’s rank correlation between paired scores across all 90 models. The complete correlation matrix is below:
| Judge | DeepSeek-V4-Pro (Original) | DeepSeek-V4.1-Flash | Kimi-K3 | Qwen3.8-Max | gpt-6-luna | gpt-6.1-sol | Biology PhD Student | Industry Expert |
|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4-Pro (Original) | 1.00 | 0.92 | 0.85 | 0.87 | 0.84 | 0.81 | 0.83 | 0.85 |
| DeepSeek-V4.1-Flash | 0.92 | 1.00 | 0.90 | 0.91 | 0.88 | 0.84 | 0.85 | 0.90 |
| Kimi-K3 | 0.85 | 0.90 | 1.00 | 0.91 | 0.84 | 0.81 | 0.82 | 0.87 |
| Qwen3.8-Max | 0.87 | 0.91 | 0.91 | 1.00 | 0.87 | 0.83 | 0.83 | 0.90 |
| gpt-6-luna | 0.84 | 0.88 | 0.84 | 0.87 | 1.00 | 0.82 | 0.84 | 0.87 |
| gpt-6.1-sol | 0.81 | 0.84 | 0.81 | 0.83 | 0.82 | 1.00 | 0.81 | 0.85 |
| Biology PhD Student | 0.83 | 0.85 | 0.82 | 0.83 | 0.84 | 0.81 | 1.00 | 0.88 |
| Industry Expert | 0.85 | 0.90 | 0.87 | 0.90 | 0.87 | 0.85 | 0.88 | 1.00 |
The backbone LLM use in this study ratings correlate 0.828 and 0.848 with the human scores; the five additional LLM judges have human correlations of 0.813 to 0.896. Pairwise correlations between LLM judges range from 0.809 to 0.920. These results support consistent rankings across evaluators.
Manuscript changes: Appendix H adds assessment of the sampled ten-target subset by two human raters and multiple LLM judges (Figures S11–S13).
Thank you for the response. While the differences from prior work are now laid out more clearly, I am still not convinced of how meaningful those differences actually are. I also think the number of benchmark instances remains too small (and the additional experiments were conducted on only ten targets). Regarding the human expert baseline, what I meant was that the performance of human experts performing the same task should be included, and a qualitative comparison still does not appear to have been provided. For these reasons, I will maintain my current score.
W1 and Q1
Thank you for asking us to clarify the research question and the reasons behind Origo's design. We study the extent to which an agent system can understand and reconstruct biological metabolic networks. Origo tests this through target-driven de novo construction: starting from pathway and reaction knowledge, agents assemble the supporting metabolism into an executable SBML model and revise it using network diagnostics. Starting without a pre-existing host GEM requires the system to organize the metabolic infrastructure itself. This provides a computational test of metabolic understanding through the connections, biochemical constraints, and functions represented in the generated network.
The task definition and discussion scope is different from related methods, and that also drives us to build DLBench. ChatGEM analyzes and modifies existing GEMs; MetaGEM reconstructs networks from organism-specific metabolites and candidate proteins; D2Cell connects literature-derived engineering knowledge to predictions using existing GEM structures. SubNetX integrates production subnetworks into a host GEM, while EnzRetro predicts enzymatic routes and EC classifications. Origo examines the combination of target conditioning, pathway-based assembly, and system-level SBML construction without a host GEM as input. Under this definition, its scope differs from that of the five systems compared here.
The scientific contribution is a computational investigation of how far agent systems can understand and reconstruct metabolic organization, with de novo SBML generation making that capability open to inspection and testing. It provides an exploratory foundation for the metabolic modeling infrastructure of digital cell factories and virtual cells.
Manuscript changes: Section 2 and Appendix E discuss the five related systems and distinguish their task requirements from Origo's settings.