- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors focus on using multiple language model agents to replicate the predictions of a genome-scale metabolic model. They propose metabolic knowledge fine-tuning with domain-specific harnesses to model microbial communities. First, they explored fine-tuning of language models on FVA/pFBA outputs to give valid flux distributions adhering to the biochemistry. They then studied multi-agent systems scaled to bacterial communities to capture interactions within the community. They claim that this multi-agent approach correctly modelled the network dynamics, whereas dynamic multispecies metabolic modelling failed in comparison. However, my main concerns are that the writing tends to overclaim where experiments do not cover these claims, in particular, that language models can reproduce the state of an organism and can act as linear program solvers.
Critical strengths
The authors target community metabolic modelling, which is a relevant problem, and evaluate against experimental rather than simulated data.
The paper contains very detailed procedures for constructing the benchmark for reproducibility and makes a great effort in describing the setups of the metabolic models.
Critical weaknesses
-
The framework mainly uses LLM agents with prompting, and LoRA fine-tuned on FBA-based simulation outputs. The methodological novelty is limited to the combination of various components.
-
The main claim is not tested directly. The authors claim that LLMs are trained on literature, so they carry the information needed to predict the state of microorganisms. However, the results in 4.1.1 and Table 1 show that prompted LLMs do not outperform FBA. This may be used to argue that LLMs cannot predict the metabolic state.
-
Using LoRA is not training the model to solve a linear program, and current results have not tested this. Finetuning likely imitates the solver outputs, but this is not equivalent to solving the LP. This requires a dedicated experimental section to validate if scoped this way.
Questions for authors
-
How are the growth fold and metabolites log10 measured in Table 1?
-
The validator is relevant for many results but is not specified. How is the validator implemented exactly?
3 comments
I appreciate the effort the authors have put into their response and revision. My main concern is not addressed. I feel the paper's idea is good but needs to justify the relevance of LLMs.
The main claim is not tested directly. The authors claim that LLMs are trained on literature, so they carry the information needed to predict the state of microorganisms. However, the results in 4.1.1 and Table 1 show that prompted LLMs do not outperform FBA. This may be used to argue that LLMs cannot predict the metabolic state.
Prompted LLMs lose to FBA and outperform it only after fine-tuning. This shows the useful signal actually comes from the metabolic models. Are LLMs just acting as surrogates for FBA? What properties of LLMs make them better surrogates than the metabolic models themselves? The language modeling, more parameters, or the architecture? For example, a non-language surrogate trained on the same corpus, as a control, could show what LLMs bring.
Thank you @RbGd for continuing the conversation. Your initial comment was "LLMs cannot predict the metabolic state." which we clarified by stressing that fine-tuning increased performance beyond that of metabolic networks. Then your follow-up comment was "the useful signal actually comes from the metabolic models", which we agree with; our hypothesis is that fine-tuning allows a form of synergy between modalities, that's why it outperforms both baselines (LLMs+Metabolic models). For example, on single-agent models, fine-tuning allowed LLMs to improve prediction on lactate and butyrate (fermentation pathways).
What properties of LLMs make them better surrogates than the metabolic models themselves? Our initial hypothesis was that LLMs had learned new capacities by solving mixed-integer linear programs for metabolic models which was shown by the increase in feasible solutions proposed by LLMs, but we removed that claim at the request of the reviewer. A plausible hypothesis hinted at by the reviewer is language modeling; the LLMs we tested have reasoning capacity which might be tied to increase in performance.
As requested, we ran a non-LLM baseline on the same corpus, by training an artificial neural network, which receives the same information as the prompt (media composition, species names) as numerical input and was run in the same simulation as the fine-tuned LLM. This setting showed that there is signal that comes from metabolic models for the species in the training set (Table1: accuracy of 0.72/F1 score of 0.62). However, for species and media outside the corpus, the MLP fails (error on metabolites is 1.9 vs 0.6 for LLMs), while the LLM are able to take any medium as input even if not part of training, which brings a notion of transfer learning outside the training set.
The reviewer have also alluded on more parameters being more helpful to learning and our figure S1, shows that larger models are able to recapitulate metabolic knowledge better with an accuracy proportional to popularity of the microorganism (well-studied organisms are better predicted). The fine-tuned model used Llama-3.1-8B, which was less accurate than FBA without fine-tuning. This provides a natural control on the important components that led to the improvements in accuracy.
To sum up, on the reviewers’ question about the increase in performance of fine-tuned LLMs over metabolic models, we showed in the first section of the paper that metabolic models can be incomplete in a new medium where they haven’t been curated (they don’t have a notion of statistical learning). In fact, only 1/25 species grew on our test media DM38, and we had to manually debug those draft models to make them compliant with experiments, which then all 25/25 ended up growing on it, as expected. We mitigated these gaps during fine-tuning by generating a corpus that included 897 species grown on different media with ablated nutrients so the fine-tuned model does not depend on one single gappy reconstruction.
Finally, on the general question on why LLM perform well in recapitulating metabolic knowledge, there have been recent reports that state of the art models are being trained on biomedical literature and genomic sequences, which may explain why Haiku was the only model that recapitulated such a complex behavior as diauxic growth.
We’re glad to answer any other results the reviewer might have and we will upload the amended manuscript with new results shortly.
We thank the reviewer for a careful reading, for their critical feedback to improve our work, and for the opportunity to respond to those critiques. We are glad that the reviewer noted critical strengths in the "high relevance of the problem", the "evaluation against real data", and the "great effort in describing the setups". The reviewer has noted substantial weaknesses that we addressed in the point-by-point response below and in the updated manuscript. We believe these weaknesses relate to supplementary figures and discussion points, and not to core results.
W1, novelty. To our knowledge, no prior work has evaluated LLMs as models of microbial communities nor modeled these communities using agents given bacterial personas. The closest work, Eubiota (Lu et al., 2026), is a biomedical AI agent that acts as a co-scientist with knowledge about gut bacteria, whereas we model the bacteria directly as agents. No prior work has shown that fine-tuning an LLM on a metabolic graph is feasible and provides accuracy higher than either the metabolic graph or the LLM (Table 1), which indicates learned emergent knowledge. We do agree, however, that the basic recipe (LLM + LoRA fine-tuning) is standard, although a novel recipe is not a requirement for the applications to physical sciences track. Our work has generated new knowledge and novel insights into a pertinent basic ecological problem using LLMs.
W2, LLMs are less accurate than metabolic networks. We first tested the ability of prompted LLMs to recover known capabilities quantitatively, and they do so at lower precision than metabolic networks (Table 1). We also show in a new figure (Figure S1) that this is directly proportional to LLM model size, where larger models do better in predicting network fluxes for more common and better-studied organisms (E. coli, yeast, P. putida), reflecting a knowledge bias. All this preliminary data was what drove our work to the next phase, which was fine-tuning LLMs for microbial species. We used this fact as a ramp to come up with a fine-tuning strategy, which then showed that LLMs fine-tuned on metabolic networks now have higher performance than metabolic networks and than prompted LLMs (Table 1 and Table S2).
W3, Fine-tuning does not solve a linear program. We agree and withdraw the claim. This was a discussion point and not a core result. It was merely a hypothesis we came up with to explain the results and model behavior, given that the admissible fraction of LLM-proposed fluxes rose from 0.868 to 0.928 with fine-tuning (Table S5), and given prior work by Nair et al. (2020) solving MILPs using deep neural networks. The revised manuscript now states only that the fine-tuned fluxes are highly consistent with the linear program results.
Q1, growth fold and metabolite errors. These are measurements used as ground truth data for testing, from Clark et al. (2021). Growth fold is each species' final biomass after 48 hours of culture, and the metabolite error is the absolute log10 of predicted over measured concentration in mM for butyrate, acetate, lactate and succinate, averaged over the four. We added the formulas for computing them (Equations 12 and 13) to Appendix C.2, where these results are reported in full (Table S2).
Q2, the validator. A validator is a set of functions that verify that the proposed LLM flux distribution respects a pre-specified output format (a JSON object with a specified format) and that the proposed fluxes respect mass balance (no uptake fluxes are larger than what is available in the medium). It is a basic check step that is required so that downstream analyses, including dynamic simulation, can run consistently. We included a description of the validator in the manuscript (Section 3.4), and Table 1 reports its outcome for each model (rows "mass balanced fluxes" and "unparsable LLM outputs").
Ethics flag. The footnote that the reviewer flagged was added automatically by the ICLR submission system to all submissions. Any other submission has it as well.