- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the lack of standardized benchmarks for evaluating omics foundation models on clinically relevant drug-development tasks. The authors introduce PRIMO (Patient Representations in Multi-Omics), an open benchmark designed to assess the quality of sample-level representations learned by omics foundation models.
The main contributions of this work include:
-
Benchmark Construction: The authors curate 63 tasks from 45 public cohorts covering 30 diseases, organized into five categories: treatment outcome, clinical activity, prognosis, endotype, and perturbation response.
-
Unified Evaluation Protocol: The paper establishes a standardized evaluation framework based on frozen model embeddings, linear probing, subject-grouped cross-validation, and comparisons against a raw-expression baseline.
-
Comprehensive Model Evaluation: The authors evaluate 11 foundation models across bulk and single-cell transcriptomics and introduce a public leaderboard using Bradley–Terry-based Elo ratings to facilitate comparisons across tasks and therapeutic areas.
Experimental results demonstrate that most evaluated foundation models do not consistently outperform the raw-expression baseline. EVA 300M achieves the strongest overall results among the evaluated bulk foundation models, while EVA 300M and Stack-Large obtain competitive results on single-cell tasks. These findings highlight the challenges of transferring pretrained omics representations to clinically relevant downstream applications.
Critical strengths
-
Important Research Problem: The paper addresses a meaningful gap between existing omics foundation model benchmarks and real-world clinical applications. Evaluating representations at the patient or sample level, rather than focusing exclusively on cell-level or gene-level predictions, is well motivated.
-
Extensive Benchmark Construction: PRIMO incorporates 63 tasks spanning multiple diseases, therapeutic areas, and clinical objectives. The systematic data curation procedure, including quality control, label harmonization, and confounder checks, represents a substantial practical contribution.
-
Standardized Evaluation Framework: The use of frozen representations, consistent downstream probes, subject-grouped cross-validation, and a common raw-expression baseline improves comparability across different foundation models. The benchmark also provides a useful foundation for reproducible evaluations.
-
Useful Empirical Findings: The results reveal that several sophisticated foundation models perform worse than relatively simple raw-expression representations. This observation challenges assumptions about the general applicability of pretrained omics models and provides useful guidance for future research.
Critical weaknesses
-
Potential Data Leakage and Reproducibility Concerns: The evaluation raises concerns regarding data contamination and reproducibility. Appendix A.1 explicitly acknowledges that COMPASS was fine-tuned on the IMvigor210 cohort, which is subsequently used for the atezolizumab-response task. Consequently, the corresponding evaluation does not represent a genuinely held-out test. Although the authors disclose this limitation, its potential influence on the overall rankings is not quantified. Furthermore, EVA 300M, the strongest-performing foundation model in the main experiments, relies on a private checkpoint (Appendix A.9), preventing independent reproduction of the central results. The authors should examine potential overlaps between benchmark datasets and model pretraining data, report contamination-adjusted rankings, and improve the accessibility of the evaluated checkpoints.
-
Limitations in the Cross-Validation Protocol: Although the outer cross-validation procedure groups samples by subject, Section 3.2 explicitly states that the inner hyperparameter search is stratified but not subject-grouped. Consequently, samples from the same subject may appear in both the training and validation subsets during hyperparameter selection, potentially introducing optimistic validation estimates and affecting the selected regularization parameters. This inconsistency is particularly concerning for small clinical cohorts containing repeated measurements. The authors should adopt subject-grouped cross-validation consistently across both inner and outer evaluation procedures and examine whether this modification changes the reported model rankings.
-
Limited Statistical Power and Generalization Evidence: Despite the relatively large number of benchmark tasks, many individual datasets contain very few samples, particularly in the single-cell setting, where some tasks include only 15–20 samples. Moreover, the reported confidence intervals are relatively wide. For example, EVA 300M achieves an Elo rating of 1048 with a 95% confidence interval of [915, 1211] on single-cell tasks, which does not establish a statistically significant improvement over the baseline rating of 1000. Only three tasks explicitly evaluate cross-cohort or cross-species transfer, while most experiments rely on within-cohort cross-validation. Additional evaluations across independent cohorts, together with stronger statistical analyses, would help establish the robustness and generalizability of the benchmark findings.
-
Incomplete Analysis of Evaluation Choices and Model Adaptation: The benchmark relies primarily on frozen representations evaluated through linear probes. Although Appendices H and I investigate alternative expression baselines and nonlinear probes, the effects of other important evaluation choices remain insufficiently explored, particularly single-cell pooling strategies, model-specific preprocessing, and representation dimensionality. Moreover, the repeated characterization of zero-shot performance as a lower bound on fine-tuned model performance is not rigorously justified, since fine-tuning does not necessarily improve performance in small-data settings. Finally, the Bradley–Terry rating system considers pairwise wins rather than performance margins, potentially obscuring differences in the practical significance of model improvements. Further sensitivity analyses would strengthen confidence in the benchmark conclusions.
Questions for authors
-
Data Contamination and Reproducibility: How would the overall model rankings change if tasks overlapping with model pretraining or fine-tuning datasets were excluded? In particular, could the authors report results excluding the IMvigor210 task for COMPASS and clarify whether the EVA 300M checkpoint will be made publicly available?
-
Cross-Validation Consistency: Why is subject grouping enforced during outer cross-validation but not during inner hyperparameter optimization? Could the authors repeat the experiments using fully subject-grouped nested cross-validation and quantify its impact on performance estimates and model rankings?
-
Statistical Reliability and Cross-Cohort Generalization: Given the small sample sizes and wide confidence intervals, how stable are the rankings when individual cohorts are excluded? Could the authors provide additional leave-one-cohort-out sensitivity analyses, or expand the evaluation to independent cohorts to assess generalization beyond the original study populations?
-
Evaluation Sensitivity and Practical Model Utility: How sensitive are the conclusions to different sample-level pooling strategies, preprocessing choices, and representation dimensions? Furthermore, could the authors include a limited fine-tuning comparison to examine whether the relative rankings of foundation models remain consistent when task-specific adaptation is permitted?
Response to rebuttal
Thank you for the answers and for checking the data leakage concern. I would encourage the authors to ensure that the leakage issues they identified are reported in the main paper.