- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OmicsLM integrates transcriptomic profiles into an LLM through continuous embeddings, allowing multiple samples to be included alongside natural-language instructions. The authors construct a large-scale omics–language training corpus and introduce GEO-OmicsQA benchmark for evaluation. The model is evaluated on annotation, expression interpretation, and perturbation-response direction prediction.
Critical strengths
- The construction of a large-scale omics–language training corpus and a new evaluation benchmark is a valuable contribution to the community.
- Representing expression profiles as continuous embeddings and interleaving them with natural-language instructions makes sense. This provides a compact way to include quantitative measurements and multiple samples without serializing the raw tabular expression data.
Critical weaknesses
- The authors motivate continuous omics inputs partly by noting that ranked gene lists omit expression magnitudes. However, the perturbation evaluation only predicts directions. This leaves it unclear how well the model captures DEG or post-perturbation expression beyond their direction. Moreover, the evaluation coverage seems narrow relative to the diversity of the training tasks. For instance, drug sensitivity and pathway reasoning are not evaluated. Additional evaluations would help demonstrating the broad biological capabilities of OmicsLM.
- The paper does not compare OmicsLM with a tool-augmented agent setup. Tool-augmented systems including Claude Science and Biomni have provided practical alternatives for solving biological tasks. A comparison with such a setup would help clarify why specialized LM training is needed and what additional benefits it provides.
Questions for authors
GEO-OmicsQA free-text answers are evaluated using a Gemini judge. How well do its scores agree with expert judgments and the actual expression data?
4 comments
Thanks for the response. I now understand that the PerturbQA direction task is restricted to genes already known to respond. But I still think OmicsLM would have been more useful in practice if its training objectives had included predicting gene-level expression after perturbation. TxPert [1], for example, uses the same underlying Replogle and Nadig datasets for this kind of prediction.
Regarding W2, a general-purpose LLM agent given the same expression profiles and basic python or deep learning model-based tools is a practical and relevant baseline, but the authors do not provide additional comparison with agents.
Given these two remaining concerns, I’m keeping my score at 2.
References
[1] TxPert: using multiple knowledge graphs for prediction of transcriptomic perturbation effects, Nature Biotechnology '26
Thank you for your continued engagement and for clarifying the remaining concerns.
1. Predicting post-perturbation expression. We agree that predicting full post-perturbation expression profiles would be useful. However, we do not consider its absence a critical weakness given OmicsLM’s focus on flexible interpretation of omics profiles. TxPert is a specialized model for predicting transcriptomic perturbation effects. OmicsLM is designed for interpreting omics profiles through flexible natural-language questions and responses across many tasks. These models therefore address different objectives.
Our training already includes perturbation-response prediction beyond direction: predicting affected genes, the most up- and downregulated genes, and the top-expressed genes after knockout or treatment, as documented in the appendix’s training-task table. These outputs describe the response at the gene level, without generating a complete numerical expression vector. The direction-only PerturbQA evaluation should therefore not be taken to describe the full scope of our training objectives.
The distinction is analogous to image understanding versus image generation: a vision-language LLM can interpret images without generating them, while a specialized diffusion model is trained for generation. The absence of image generation is not, by itself, a critical weakness of an image-understanding model. Likewise, full transcriptome generation would be a useful extension, but it is a different objective from the flexible omics–language interface studied here.
2. Tool-using agents. Thank you for this suggestion. We evaluated Gemini 3.8 Flash with medium reasoning effort in two agent setups on all 1,000 binary and 2,000 free-text GEO-OmicsQA questions. Each agent received the complete expression profiles. In Gemini + Python, the Python tool could read and analyze all measured genes’ numerical expression values (TPM) for every supplied sample. Evaluation used the original references and scoring. Biomni A1 used a basic Python environment. We allowed up to 32 model calls and 65,536 output tokens per call.
On all 1,000 binary and 2,000 free-text questions:
| Model | Binary accuracy | Free-text score | Avg. generated tokens/question | Avg. tool calls/question |
|---|---|---|---|---|
| OmicsLM (without tools) | 76.7% | 0.651 | — | 0 |
| Gemini + Python | 75.3% | 0.689 | 5,210 | 10.81 |
For Biomni, we compare both models on the same 788 binary and 1,989 free-text questions where Biomni returned a valid answer. The remaining 223 attempts did not produce a clear final answer.
| Model | Binary accuracy | Free-text score | Avg. generated tokens/question | Avg. tool calls/question |
|---|---|---|---|---|
| OmicsLM (without tools) | 77.0% | 0.651 | — | 0 |
| Biomni A1 + Gemini | 78.2% | 0.521 | 5,325 | 6.43 |
Token and tool-call means cover the questions in each table. Generated tokens include output and reasoning across model turns, excluding input tokens, judging and recovery attempts. Tool calls count Python executions.
OmicsLM generates answers directly without external tool calls and is competitive with the proprietary Gemini model in both setups: it scores higher on one metric and lower on the other in each comparison.
Thank you for your detailed explanation and providing additional experiments. Reconsidering again, the authors' response regarding the perturbation response prediction makes sense, and the performance seems comparable with a tool-using agent. I raise my score to 3.
Thank you for recognizing the value of our training data, benchmark, and compact interface for multiple profiles.
1. Input values and predicted changes. An up/down prediction can still benefit from the expression values in the input. OmicsLM receives continuous transcriptomic profiles. PerturbQA tests whether a gene's expression increases or decreases after a perturbation. In our controlled 0.8B ablation, the full continuous input achieves 0.607 AUROC on PerturbQA. A ranked list of the 1,024 highest-expressed genes gives 0.488. Both scores are averages over three training runs (Appendix A.7). This supports our input representation even for up/down predictions. PerturbQA includes readout genes already known to change after perturbation. It does not test which genes change or by how much.
2. Task coverage. Our evaluation focuses on tasks that use measured expression profiles: annotation, biological interpretation, and perturbation prediction. Text-only benchmarks could test additional biological abilities, such as pathway reasoning, but they would not directly evaluate our main focus: using measured omics profiles to answer biological questions.
3. Tool-using agents. A central goal of OmicsLM is to learn to interpret omics profiles directly through multimodal training. Our GEO-OmicsQA comparison tests this intended capability: each model answers from the supplied inputs in one call, without external tools or code execution (Appendix A.6.1).
4. Judge reliability. Experts reviewed 50 GEO-OmicsQA model-response traces as a qualitative check and found no errors that would affect scoring in those cases. The judge checks agreement with reference answers supported by the source publications.
Separately, experts reviewed 300 GEO-OmicsQA questions for question–answer validity, sample grounding, and biological coherence. Over 90% of the checked reference claims agreed with the linked expression data.