- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper benchmarks evaluation methods and foundation models for predicting cancer cell drug responses. It compares cell foundation models and molecular foundation models within a unified framework covering cell representations, drug representations, and downstream prediction. It also investigates generalization under different settings, explicitly distinguishing pair-level, cell-line-level, and drug-level zero-shot generalization.
Critical strengths
The main strength is the systematic evaluation of cell and molecular foundation models within a unified framework, particularly the distinction between different levels of zero-shot generalization. This provides meaningful insights into model performance and represents a useful starting point for benchmarking drug response prediction.
Critical weaknesses
- Limited dataset scale and label quality. The effective dataset size is much smaller than it appears. Although the shared-label dataset is seemingly large, it contains only slightly over 400 independent response labels. Moreover, both datasets contain only 10 drugs, which is insufficient to establish robust differences in molecular representation quality. Drug sensitivity labels are also defined inconsistently across datasets, as acknowledged by the authors, introducing potentially substantial noise. These limitations significantly weaken the benchmark's reliability and impact.
- Limited evaluation of foundation models and downstream architectures. The authors use frozen foundation model embeddings without fine-tuning, which deviates from practical usage. Furthermore, the downstream predictor simply concatenates cell and drug embeddings, a straightforward approach that may not adequately capture cross-modal interactions. These choices limit how comprehensively the benchmark evaluates the capabilities of different foundation models.
Overall, the paper provides a useful starting point, but I do not find it sufficiently impactful or convincing as a benchmark in its current form.
Questions for authors
How might the conclusions change if foundation models were fine-tuned or more expressive architectures were used to model interactions between cell and drug representations?
Critical Weaknesses
We agree these are important limitations and now state them prominently (Section 5). Although the shared-label dataset contains many cells, it provides only 416 distinct cell-line–drug response labels. We report this effective sample size explicitly and evaluate at the pair level rather than treating cells sharing a label as independent observations.
"Each setting includes only 10 drugs, insufficient for robust rankings of molecular encoders": We do not claim that any drug representation is intrinsically superior. Our narrower conclusion is that, on currently available single-cell datasets, pretrained molecular representations show no consistent advantage over structural fingerprints. This is a limitation of the evidence data available for evaluating transfer, not evidence that molecular foundation models cannot succeed with broader drug coverage.
Label definitions are also heterogeneous. We deliberately separate two complementary regimes: shared labels, which are relatively standardized but coarse (assigned at cell-line–drug level but per single-cell transcriptome), and cell-specific labels, which capture cellular heterogeneity but combine different experimental readouts and labeling strategies. The principal patterns are: weak entity-level zero-shot discrimination and limited consistent gains from pretrained encoders. Those persist across both drug response label definitions, despite their differences. We now discuss the resulting uncertainty and limits on cross-study comparisons explicitly.
Our goal is a controlled assessment of released pretrained representations as transferable features, not a comprehensive benchmark of end-to-end training. Frozen inference is a valid practical use case and isolates representation quality from model-specific fine-tuning choices. Fine-tuning could help, but with only 416 distinct response labels in the shared-label setting, adapting models with >100M parameters risks substantial overfitting.
We also clarify that concatenation specifies the input, not the predictor’s expressiveness. All three predictors can learn cross-modal interactions: the MLP through nonlinear hidden layers, LightGBM through splits spanning cell and drug features, and xRFM through a learned full feature matrix encompassing both modalities. Nonetheless, architectures explicitly designed for interactions (e.g., bilinear layers or cross-attention) might capture them more efficiently. We now acknowledge this boundary in Section 5.
We appreciate the reviewer’s assessment that this is a useful starting point. We have substantially strengthened the manuscript in response to all reviewers. The revision clarifies our contributions relative to prior benchmarks and adds threshold-free metrics across all splits. It also states the selection criteria for the representations and expands the discussion of limitations and future directions. The paper’s central contribution is not a definitive ranking of foundation models. It is showing that evaluation choices change the scientific conclusion: the same predictions can look strong when aggregated but fail when scored within a drug or cell line, while prevalence baselines can match or outperform full models. Under cell-line holdout, MCC changes from 0.52 within cell lines to −0.03 within drugs, while a drug-prevalence baseline reaches 0.58.
Questions for Authors
How might fine-tuning or more expressive interaction architectures change the conclusions?
Using foundation models as frozen feature extractors is a common practical usage, and it tests their central promise: transferable representations that can be reused without task-specific retraining. Fine-tuning might benefit the prediction task but would confound representation quality with task adaptation and require per-model tuning, undermining the controlled comparison. Benefits under unseen-cell-line or unseen-drug holdout remain an open empirical question, and we would expect them to be constrained by limited drug diversity and strong marginal effects: single-modality models already track prevalence baselines. With 416 independent labels in the shared-label setting, fine-tuning models with over 100M parameters would also carry a high risk of overfitting.
We expect smaller gains under entity-level zero-shot splits. Our ablations show that much of the signal is marginal (single-modality models track prevalence baselines), and with 9 training drugs per drug zero-shot fold, no architecture has much chemical diversity to extrapolate from. This is consistent with bulk benchmarks, where complex architectures do not reliably outperform tree-based baselines (Bernett et al., 2026). We have clarified these points in Section 5.