- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how to adapt a compact language model for both microbiome sample classification and scientific reasoning. The authors introduce MicrobeText, a continued-pretraining dataset combining microbiome abundance profiles, sample metadata, and literature-derived information. They further construct MicrobeTextSFT for multi-task supervised fine-tuning and introduce MicrobeExpertQA, an expert-authored benchmark for evaluating microbiome knowledge and scientific reasoning. Starting from Qwen3-1.7B, the authors develop MicrobiomeLM-base and MicrobiomeLM-MT. Experiments on seven microbiome classification tasks and two scientific knowledge benchmarks show improvements over several language model baselines, although the additional benefits of continued pretraining become smaller after multi-task fine-tuning.
Critical strengths
- Practical significance.The paper explores combining microbiome classification and scientific reasoning within a compact language model, achieving promising results on several benchmarks.
- Useful evaluation resources.The proposed MicrobeExpertQA benchmark provides expert-authored questions and evaluation rubrics, offering a useful resource for assessing microbiome-related scientific reasoning.
Critical weaknesses
Weaknesses:
- Insufficient validation of the multi-task training strategy. The paper builds upon MicrobiomeLM-base and further introduces scientific reasoning capabilities through multi-task supervised fine-tuning. However, microbiome classification and scientific reasoning differ substantially in their learning objectives, supervision signals, and output formats. The paper does not sufficiently demonstrate whether this training strategy can effectively integrate these two capabilities. In particular, No-CPT-MT achieves competitive performance through direct multi-task fine-tuning without CPT. Therefore, the current experiments do not establish a clear advantage of the proposed two-stage training strategy over direct multi-task fine-tuning, nor do they adequately investigate potential task interference or negative transfer.
- Limited incremental benefits of CPT for the final multi-task model. As acknowledged by the authors in Section 3.4, the performance advantage of CPT becomes substantially smaller after multi-task fine-tuning. Table 3 shows that MicrobiomeLM-MT improves upon No-CPT-MT by only 1.0 and 0.4 points on MEQA and MR, respectively. Such limited gains raise questions about the necessity of the additional CPT stage. However, the paper does not adequately evaluate the additional computational cost of CPT or systematically analyze the trade-off between performance gains and training overhead, making it difficult to assess the practical value of this extra training stage.
- Incomplete statistical reporting. Table 1 reports standard deviations across three independent runs for single-task models but does not provide corresponding variability estimates for multi-task models. Table 3 also lacks such statistics, making it difficult to assess the statistical reliability of the relatively small performance improvements.
- Insufficient ablation studies. Table 2 progressively removes synthetic documents and external knowledge rather than independently ablating each component. This makes it difficult to clearly determine the individual contributions of these data components to model performance.
- Minor typos and grammatical errors: L141: studies focusing microbiome-human disease interactions → studies focusing on microbiome-human disease interactions L225: representative domain-specific model → a representative domain-specific model L249: Few-shot LMs Next → Few-shot LMs. Next Appendix A.1: commentary commentary → commentary Table 4: Lactococus → Lactococcus.
Questions for authors
- What clear advantages does fine-tuning on top of MicrobiomeLM-base offer over direct multi-task fine-tuning? Could the authors further investigate potential task interference or negative transfer between microbiome classification and scientific reasoning, and provide controlled experiments to validate the effectiveness of the multi-task training strategy?
- Given the relatively small performance improvements from CPT in Table 3, how necessary is the additional CPT stage for the final multi-task model? Could the authors report its computational cost and further analyze the trade-off between performance gains and training overhead?
- Could the authors report standard deviations across multiple independent runs or confidence intervals for the multi-task results in Tables 1 and 3 to clarify the statistical reliability of the reported performance improvements?