acceptodds
Under review as a conference paper at ICLR 2027

Toward a Shared Objective for Omics Foundation Models: The PRIMO Benchmark

Abstract

Omics foundation models are trained and promoted as tools for drug development, but they are rarely evaluated in this context. Existing benchmarks mostly operate at the level of the cell, the gene, or the batch, but the decisions that advance a drug program are made at the level of the patient the sample came from: which indication to pursue for a given target, whether a candidate should progress into the clinic, and which patients should be enrolled to observe a treatment effect. In addition, tasks and datasets are often preprocessed and built differently from one study to another, making results difficult to compare. We introduce PRIMO (Patient Representations in Multi-Omics), an open benchmark of sample-level tasks, with a first version focused on bulk and single-cell transcriptomics. We curate 63 tasks from 45 public cohorts on 30 diseases, most of them in four therapeutic areas, covering these decisions through five task categories: treatment outcome, clinical activity, prognosis, endotype and perturbation response, including cross-species and cross-indication transfer. Every task follows the same curation process: a clear drug-development question, a public cohort that can answer it, data cleaning, and tests ensuring the contained signal is linked to the biology rather than to technical artifacts. Under a common evaluation protocol with subject-grouped, label-stratified cross-validation, we evaluate 11 foundation models trained on bulk, single-cell or both modalities in the zero-shot setting: their weights stay frozen, and a shared linear probe reads their embeddings. All entries are rated against a raw-expression baseline, so that every rating reads directly as the value a representation adds over the raw data. Since foundation models are fine-tuned in practice, these zero-shot ratings are a lower bound on their value. Even in this setting, the strongest foundation models reach or exceed the raw-expression baseline, while most remain below it. We release PRIMO as a public leaderboard on curated tasks, with ratings per therapeutic area, giving model developers, drug developers and clinicians a shared reference to track progress and optimize for.

Primary area:
datasets and benchmarks
Keywords:
benchmark, omics foundation models, transcriptomics, single-cell RNA-seq, bulk RNA-seq, drug development, sample-level representations, zero-shot evaluation, leaderboard

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WdN9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the lack of standardized benchmarks for evaluating omics foundation models on clinically relevant drug-development tasks. The authors introduce PRIMO (Patient Representations in Multi-Omics), an open benchmark designed to assess the quality of sample-level representations learned by omics foundation models.

The main contributions of this work include:

  1. Benchmark Construction: The authors curate 63 tasks from 45 public cohorts covering 30 diseases, organized into five categories: treatment outcome, clinical activity, prognosis, endotype, and perturbation response.

  2. Unified Evaluation Protocol: The paper establishes a standardized evaluation framework based on frozen model embeddings, linear probing, subject-grouped cross-validation, and comparisons against a raw-expression baseline.

  3. Comprehensive Model Evaluation: The authors evaluate 11 foundation models across bulk and single-cell transcriptomics and introduce a public leaderboard using Bradley–Terry-based Elo ratings to facilitate comparisons across tasks and therapeutic areas.

Experimental results demonstrate that most evaluated foundation models do not consistently outperform the raw-expression baseline. EVA 300M achieves the strongest overall results among the evaluated bulk foundation models, while EVA 300M and Stack-Large obtain competitive results on single-cell tasks. These findings highlight the challenges of transferring pretrained omics representations to clinically relevant downstream applications.

Critical strengths
  1. Important Research Problem: The paper addresses a meaningful gap between existing omics foundation model benchmarks and real-world clinical applications. Evaluating representations at the patient or sample level, rather than focusing exclusively on cell-level or gene-level predictions, is well motivated.

  2. Extensive Benchmark Construction: PRIMO incorporates 63 tasks spanning multiple diseases, therapeutic areas, and clinical objectives. The systematic data curation procedure, including quality control, label harmonization, and confounder checks, represents a substantial practical contribution.

  3. Standardized Evaluation Framework: The use of frozen representations, consistent downstream probes, subject-grouped cross-validation, and a common raw-expression baseline improves comparability across different foundation models. The benchmark also provides a useful foundation for reproducible evaluations.

  4. Useful Empirical Findings: The results reveal that several sophisticated foundation models perform worse than relatively simple raw-expression representations. This observation challenges assumptions about the general applicability of pretrained omics models and provides useful guidance for future research.

Critical weaknesses
  1. Potential Data Leakage and Reproducibility Concerns: The evaluation raises concerns regarding data contamination and reproducibility. Appendix A.1 explicitly acknowledges that COMPASS was fine-tuned on the IMvigor210 cohort, which is subsequently used for the atezolizumab-response task. Consequently, the corresponding evaluation does not represent a genuinely held-out test. Although the authors disclose this limitation, its potential influence on the overall rankings is not quantified. Furthermore, EVA 300M, the strongest-performing foundation model in the main experiments, relies on a private checkpoint (Appendix A.9), preventing independent reproduction of the central results. The authors should examine potential overlaps between benchmark datasets and model pretraining data, report contamination-adjusted rankings, and improve the accessibility of the evaluated checkpoints.

  2. Limitations in the Cross-Validation Protocol: Although the outer cross-validation procedure groups samples by subject, Section 3.2 explicitly states that the inner hyperparameter search is stratified but not subject-grouped. Consequently, samples from the same subject may appear in both the training and validation subsets during hyperparameter selection, potentially introducing optimistic validation estimates and affecting the selected regularization parameters. This inconsistency is particularly concerning for small clinical cohorts containing repeated measurements. The authors should adopt subject-grouped cross-validation consistently across both inner and outer evaluation procedures and examine whether this modification changes the reported model rankings.

  3. Limited Statistical Power and Generalization Evidence: Despite the relatively large number of benchmark tasks, many individual datasets contain very few samples, particularly in the single-cell setting, where some tasks include only 15–20 samples. Moreover, the reported confidence intervals are relatively wide. For example, EVA 300M achieves an Elo rating of 1048 with a 95% confidence interval of [915, 1211] on single-cell tasks, which does not establish a statistically significant improvement over the baseline rating of 1000. Only three tasks explicitly evaluate cross-cohort or cross-species transfer, while most experiments rely on within-cohort cross-validation. Additional evaluations across independent cohorts, together with stronger statistical analyses, would help establish the robustness and generalizability of the benchmark findings.

  4. Incomplete Analysis of Evaluation Choices and Model Adaptation: The benchmark relies primarily on frozen representations evaluated through linear probes. Although Appendices H and I investigate alternative expression baselines and nonlinear probes, the effects of other important evaluation choices remain insufficiently explored, particularly single-cell pooling strategies, model-specific preprocessing, and representation dimensionality. Moreover, the repeated characterization of zero-shot performance as a lower bound on fine-tuned model performance is not rigorously justified, since fine-tuning does not necessarily improve performance in small-data settings. Finally, the Bradley–Terry rating system considers pairwise wins rather than performance margins, potentially obscuring differences in the practical significance of model improvements. Further sensitivity analyses would strengthen confidence in the benchmark conclusions.

Questions for authors
  1. Data Contamination and Reproducibility: How would the overall model rankings change if tasks overlapping with model pretraining or fine-tuning datasets were excluded? In particular, could the authors report results excluding the IMvigor210 task for COMPASS and clarify whether the EVA 300M checkpoint will be made publicly available?

  2. Cross-Validation Consistency: Why is subject grouping enforced during outer cross-validation but not during inner hyperparameter optimization? Could the authors repeat the experiments using fully subject-grouped nested cross-validation and quantify its impact on performance estimates and model rankings?

  3. Statistical Reliability and Cross-Cohort Generalization: Given the small sample sizes and wide confidence intervals, how stable are the rankings when individual cohorts are excluded? Could the authors provide additional leave-one-cohort-out sensitivity analyses, or expand the evaluation to independent cohorts to assess generalization beyond the original study populations?

  4. Evaluation Sensitivity and Practical Model Utility: How sensitive are the conclusions to different sample-level pooling strategies, preprocessing choices, and representation dimensions? Furthermore, could the authors include a limited fine-tuning comparison to examine whether the relative rankings of foundation models remain consistent when task-specific adaptation is permitted?

Reviewer_BNiDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a benchmark (PRIMO) for evaluating omic foundation models, specifically for drug development usecases. While existing benchmarks focus on a cell/gene/batch level, this contradicts the objective of obtaining a sample level outcome which is more useful for drug development. The authors have a set of 63 tasks from 45 public patient cohorts across 30 diseases. They evaluate 11 foundation models on this and have a public leaderboard.

Critical strengths
  1. The benchmark is diverse, comprising several diseases, outcome categories and tasks.

  2. The authors have ensured a sufficient number of samples in each category/fold and have designed the inclusion criteria well

  3. The use of frozen representations across the foundation models provides a lower bound on performance

  4. Making the leaderboard public would be useful to the community developing omic foundation models and help set a standard evaluation protocol.

Critical weaknesses

The motivation that other benchmarks do not evaluate on a per-sample basis needs more elaboration. For example, Liu et al[1] talk about integrative models which deal with a sample level outcome (like cancer subtype classification, drug response prediction), including models like scGPT which is evaluated in PRIMO. In addition, the differences in PRIMO as compared to benchmarks like SCMBench[2] needs to be elaborated.

Questions for authors
  1. Liu et al[1] describes details on preprocessing the data and batch effect correction. How is this achieved in PRIMO?

  2. PRIMO relies on zero shot inference with existing foundation models. Is this the best approach, given that certain models may have been trained for some of these tasks?

  3. How is the data processing across all cohorts done? Is this in such a way that is compatible to all the foundation models evaluated?

  4. How do you ensure that data leakage is avoided? Is it possible that some of these foundation models have been trained on these datasets?

  5. How trivial or non-trivial is it to include publicly available clinical trial datasets, like those in TrialBench[3]?

  6. Can the benchmark be easily extended to include other omics like proteomics, mutations, copy number variations etc?

  7. How do you handle situations where a different subset of genes are observed across cohorts in the public datasets?

[1] Liu, H., Cai, W., Sun, Y., Liu, H., Zou, Z., Zhao, Q., ... & Sun, J. (2026). Foundation Models in Omics Research: A Comprehensive Survey.

[2] Wang, Y., Fan, Y., Wang, X., Yu, T., Zong, Y., Liu, X., ... & Li, Y. (2026). SCMBench: benchmarking domain-specific and foundation models for single-cell multi-omics data integration. Nature Communications.

[3] Chen, J., Hu, Y., Cai, M., Lu, Y., Wang, Y., Cao, X., ... & Fu, T. (2025). TrialBench: Multi-modal AI-ready datasets for clinical trial prediction. Scientific data, 12(1), 1564.

2 comments
Reviewer_BNiDOpenReview ↗

Response to rebuttal

Thank you for the answers and for checking the data leakage concern. I would encourage the authors to ensure that the leakage issues they identified are reported in the main paper.

AuthorsOpenReview ↗

Leakage reported in the main paper

Thank you, we agree. The revised PDF now reports the leakage in the main text (Section 4.2, paragraph "Pretraining overlap and robustness"), including the two models that saw labels: COMPASS (IMvigor210 cohort) and probably scPRINT-2 (one single-cell task). Appendix J gives the full audit.

Reviewer_p2hCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

PRIMO is a benchmark for omics foundation models, focusing on tasks for drug development and clinical research. It comprises tasks in immunology, cancer and inflammation diseases, and it covers treatment response, disease severity, molecular perturbation, and patient endotypes.

Critical strengths
  1. PRIMO benchmark groups tasks by key decisions relevant for preclinical translation and clinical drug development, which ties evaluation to the use cases that determine the practical value of omics foundation models.

  2. Realistic data regimes and transfer settings. Benchmarks performs test generalization under distribution shifts.

  3. Comparison against statistical baselines identifies task categories where simple models match foundation models.

Critical weaknesses

Data leakage and fair comparison. For example, one of the models, EVA-RNA, pretrains on 545k human and mouse samples focused on immunology and inflammation, and many benchmark cohorts that might originate from the same public repositories as the test data. Another model, COMPASS, is pre-trained on 10k human samples (so, no mouse) in tumor (so, no immunology and inflammation). Unclear how to ensure that model comparison is fair given that pre-training datasets for different models vary in size by orders of magnitude, domains (disease areas and species). Models used different pre-training datasets, different fine-tuning strategies, and different model architectures. Unclear how a model that was designed and developed for disease area A can be compared on disease area B.

Questions for authors

Please address issues of data leakage and ensure fair comparison, matched by data and compute budget, as well as taking into consideration which tasks are in-domain tasks vs out-of-domain tasks for specific models included in the benchmark.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank the three reviewers and the area chair for their time and their careful reading, which improved the paper. The revised PDF adds the audit of pretraining data, the comparison by compute and domain, the subject-grouped inner search and new robustness analyses, as detailed in our answers on this page.

We appreciate your time, and remain available for further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.