acceptodds
Withdrawn from ICLR 2027

Adapting Language Models for Microbiome Prediction and Scientific Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The human gut microbiome plays a vital role in shaping metabolism and immunity. To support research in this domain, AI systems must be able to reason about general microbiome knowledge and provide analysis of individual microbiome measurements, for example predicting disease status from a sample. Current models are suboptimal: microbiome-specific models are inexpensive and can classify individual samples, but are not designed for open-ended knowledge tasks; frontier language models have scientific knowledge and few-shot classification abilities, but are expensive to run. In our work, we address this discrepancy by curating data to help inexpensive pretrained language models support both of these capabilities. Starting with sample classification, we introduce a continued-pretraining (CPT) dataset that combines information from sample metadata and scientific literature. We show that while current language models can be adapted for sample classification, training on our dataset improves performance. To evaluate microbiome knowledge, we introduce an expert-authored benchmark of challenging questions that require critical assessment of scientific situations. Finally, to support both sample classification and scientific knowledge, we curate a multi-task finetuning data mixture using data derived from biological databases, publications, and sample-level classification. A 1.7B model trained on this mixture outperforms few-shot frontier models by 7 percentage points on average on our sample classification tasks and outperforms a 31B model by 10 points on one of our two knowledge tasks. Our results illustrate the feasibility of curating data that supports predicting biologically meaningful sample properties and scientific knowledge for a complex scientific domain within a single, compact model.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
microbiome, biology datasets, multi-task learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZcLkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to adapt a compact language model for both microbiome sample classification and scientific reasoning. The authors introduce MicrobeText, a continued-pretraining dataset combining microbiome abundance profiles, sample metadata, and literature-derived information. They further construct MicrobeTextSFT for multi-task supervised fine-tuning and introduce MicrobeExpertQA, an expert-authored benchmark for evaluating microbiome knowledge and scientific reasoning. Starting from Qwen3-1.7B, the authors develop MicrobiomeLM-base and MicrobiomeLM-MT. Experiments on seven microbiome classification tasks and two scientific knowledge benchmarks show improvements over several language model baselines, although the additional benefits of continued pretraining become smaller after multi-task fine-tuning.

Critical strengths
  1. Practical significance.The paper explores combining microbiome classification and scientific reasoning within a compact language model, achieving promising results on several benchmarks.
  2. Useful evaluation resources.The proposed MicrobeExpertQA benchmark provides expert-authored questions and evaluation rubrics, offering a useful resource for assessing microbiome-related scientific reasoning.
Critical weaknesses

Weaknesses:

  1. Insufficient validation of the multi-task training strategy. The paper builds upon MicrobiomeLM-base and further introduces scientific reasoning capabilities through multi-task supervised fine-tuning. However, microbiome classification and scientific reasoning differ substantially in their learning objectives, supervision signals, and output formats. The paper does not sufficiently demonstrate whether this training strategy can effectively integrate these two capabilities. In particular, No-CPT-MT achieves competitive performance through direct multi-task fine-tuning without CPT. Therefore, the current experiments do not establish a clear advantage of the proposed two-stage training strategy over direct multi-task fine-tuning, nor do they adequately investigate potential task interference or negative transfer.
  2. Limited incremental benefits of CPT for the final multi-task model. As acknowledged by the authors in Section 3.4, the performance advantage of CPT becomes substantially smaller after multi-task fine-tuning. Table 3 shows that MicrobiomeLM-MT improves upon No-CPT-MT by only 1.0 and 0.4 points on MEQA and MR, respectively. Such limited gains raise questions about the necessity of the additional CPT stage. However, the paper does not adequately evaluate the additional computational cost of CPT or systematically analyze the trade-off between performance gains and training overhead, making it difficult to assess the practical value of this extra training stage.
  3. Incomplete statistical reporting. Table 1 reports standard deviations across three independent runs for single-task models but does not provide corresponding variability estimates for multi-task models. Table 3 also lacks such statistics, making it difficult to assess the statistical reliability of the relatively small performance improvements.
  4. Insufficient ablation studies. Table 2 progressively removes synthetic documents and external knowledge rather than independently ablating each component. This makes it difficult to clearly determine the individual contributions of these data components to model performance.
  5. Minor typos and grammatical errors: L141: studies focusing microbiome-human disease interactions → studies focusing on microbiome-human disease interactions L225: representative domain-specific model → a representative domain-specific model L249: Few-shot LMs Next → Few-shot LMs. Next Appendix A.1: commentary commentary → commentary Table 4: Lactococus → Lactococcus.
Questions for authors
  1. What clear advantages does fine-tuning on top of MicrobiomeLM-base offer over direct multi-task fine-tuning? Could the authors further investigate potential task interference or negative transfer between microbiome classification and scientific reasoning, and provide controlled experiments to validate the effectiveness of the multi-task training strategy?
  2. Given the relatively small performance improvements from CPT in Table 3, how necessary is the additional CPT stage for the final multi-task model? Could the authors report its computational cost and further analyze the trade-off between performance gains and training overhead?
  3. Could the authors report standard deviations across multiple independent runs or confidence intervals for the multi-task results in Tables 1 and 3 to clarify the statistical reliability of the reported performance improvements?
Reviewer_jVqXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a data-centric approach to train a small language model for both microbiome sample classification and scientific question answering. The authors construct MicrobeText for continued pretraining by converting microbial abundance profiles into different text formats and combining metadata and literature derived information. They further develop MicrobeTextSFT to train MicrobiomeLM-MT based on Qwen3-1.7B, and introduce MicrobeExpertQA with 50 questions written by experts. The CPT improves average macro-F1 from 66.8 to 70.0 across seven classification tasks, while the final model shows competitive performance in some tasks and even outperforms Gemma-4-31B-it on Microbiome Reasoning. However, the technical novelty is quite limited, as the work mainly relies on data formatting and existing training methods rather than any new modeling approach. The improvement is not very substantial, and the experiments are still not enough to clearly demonstrate the generalization and scientific reasoning ability of the model. The main contribution seems to be mostly about preparing the datasets rather than proposing something technically new.

Critical strengths
  1. The paper attempts to combine microbiome abundance based prediction and scientific question answering in a single language model. The authors construct MicrobeText using different text formats, sample metadata, information from literature. In Table 2, CPT improves the average macro-F1 from 66.8 to 70.0, and removing some data components reduces the performance. This shows some benefit of their data construction strategy, although the improvement is not very large.

  2. The authors also introduce MicrobeExpertQA to evaluate microbiome knowledge and scientific reasoning, including data interpretation, experimental design, causal evidence. The use of weighted evaluation criteria and penalties for scientifically incorrect answers is a useful part of this benchmark. It may help to evaluate scientific reliability of microbiome language models, although the benchmark contains only 50 expert questions.

Critical weaknesses
  1. The contribution of domain CPT to the final multitask model seems quite limited. In Table 3, MicrobiomeLM-MT only improves from 32.2 to 33.2 on MicrobeExpertQA and from 71.2 to 71.6 on Microbiome Reasoning compared to No-CPT-MT. These improvements are quite small considering the additional pretraining stage.

  2. In Table 1, the effect of CPT is also not consistent across different classification tasks. The results suggest that much of the final performance may come from SFT rather than CPT. The paper does not clearly justify why the additional CPT stage is necessary, especially considering its computational cost.

  3. The benefit of using a unified model for microbiome classification and scientific question answering is not sufficiently demonstrated. In Table 1, XGBoost still shows competitive performance on several classification tasks, and MicrobiomeLM-MT does not consistently outperform it. It is difficult to see a clear advantage of the proposed model for classification.

  4. The scientific question answering performance is also not very convincing. In Table 3, MicrobiomeLM-MT achieves 33.2 on MicrobeExpertQA, which is much lower than 51.2 of Gemma-4-31B-it. This raises some questions about how effectively the model actually learns and uses microbiome scientific knowledge.

  5. The paper does not evaluate tasks that require using microbiome abundance profiles and scientific knowledge together. Classification and question answering are mainly evaluated separately. It is unclear what benefit the unified model provides compared to using a specialized classification model and a separate language model. This weakens one of the main motivations of the paper.

Questions for authors
  1. The performance difference between MicrobiomeLM-MT and No-CPT-MT is quite small. What is the main reason for including the additional CPT stage in the final training pipeline?

  2. Considering the additional computational cost of CPT, can the authors explain whether these small improvements are sufficient to justify its use?

  3. The current evaluation measures microbiome classification and scientific question answering separately. Is there any experimental evidence showing the benefit of combining these two abilities in a single model?

  4. Can authors provide results on tasks that require both microbiome abundance profiles and scientific knowledge? Without such evaluation, it is difficult to see the practical advantage of the proposed unified model.

Reviewer_mzj7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses a challenge in microbiome research: specialized, small-scale models excel at cost-effective sample classification but lack natural language question answering and scientific reasoning. Conversely, while large-scale, general-purpose foundation models possess extensive scientific knowledge, they are computationally expensive to run and often underperform on specific classification tasks. To bridge this gap, the authors propose a data construction pipeline and a two-stage training framework. Furthermore, they introduce a novel dataset for disease classification, which consists of 50 complex, long-form scientific questions authored by domain experts. The authors also perform comprehensive ablation experiments to analyse the performance of the training stages and different models.

Critical strengths
  • A high-quality evaluation benchmark: The newly introduced MicrobeExpertQA comprises 50 challenging, long-form questions crafted by microbiology experts. It includes a detailed scoring rubric, specifying both positive criteria for points and negative criteria for deductions.

  • Comprehensive ablation studies and baseline comparisons: Ablation experiments conducted in a single-task setting clearly demonstrate the value of incorporating diverse tabular data and external knowledge from scientific literature for specific tasks, such as disease classification. Additionally, the authors provide an honest assessment of the strong performance of traditional machine learning baselines (e.g., XGBoost) and analyze the trade-offs between cost and performance across varying few-shot settings.

Critical weaknesses
  • Dirichlet noise may obliterate signals from key biomarkers. In Section 2.1 and Appendix A.3, the authors admit that there is "no method to ensure the biological plausibility" of the newly generated abundance tables. Given that microbiome data are characterized by extreme sparsity, zero-inflation, and complex ecological networks of co-occurrence or mutual exclusion among taxa, many disease states are determined by the presence or absence of keystone taxa that exist at very low abundances. Applying a uniform, global pseudo-count smoothing and stochastic noise injection is highly likely to disrupt the authentic ecological structure of the microbiome, potentially erasing the genuine biological signals of low-abundance taxa in the long tail.
  • To increase contextual diversity, the authors use a 1.7B model to rewrite a formal table of microbial abundance into 50 different styles. The authors acknowledge that they only manually spot-checked the output rather than conducting a systematic verification; this approach carries a high risk of injecting significant factual hallucinations and pseudoscientific correlations into the model during the pre-training phase.
  • The 37,000 scientific Q&A entries in MicrobeTextSFT are generated and distilled using the Gemma-4-31B-it model. However, the same model works as an “LLM judge” to assign scores. This practice may introduce scoring bias, as the judge model shares the same origin as the distilled student model.
Questions for authors

In the rebuttal, I would like to see clarifications regarding the data processing pipeline (the effects of noise and the necessity of style rewriting), and the judge results from a different model. Please refer to the weakness part for details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.