acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Cell and Drug Representations for Generalizable Drug Response Predictions

Abstract

Foundation models provide increasingly sophisticated representations of cells and drugs, with promising applications in precision oncology and drug discovery. Yet it remains unclear when these representations improve drug-response prediction or generalize across drugs, cellular contexts, and datasets. We systematically assess the effects of cell and drug representations, predictive architectures, and dataset characteristics by benchmarking five cell representations, six drug representations, and three predictive models across two multimodal single-cell settings: a large, controlled shared-label dataset with uniformly generated transcriptomes and standardized cell line–drug response labels, and a smaller, heterogeneous cell-specific-label setting with finer-grained response annotations. We evaluate increasingly stringent regimes, from random splits to zero-shot prediction of unseen cell line–drug pairs, drugs, and cellular contexts. Across both settings, performance depends more strongly on the cell representation and predictive model than on the drug representation, while pretrained encoders do not consistently outperform simpler alternatives. Under cell line–drug pair zero-shot evaluation, the best MCC reaches 0.36 in the shared-label setting and 0.17 in the cell-specific-label setting. Generalization degrades sharply with stringent splits compared with random splits: in the shared-label setting, the average within-drug MCC decreases from about 0.89 with a random split to 0.18 with cell line–drug pair zero-shot. In the cell-specific-label setting, it decreases from 0.62 to 0.03. Conclusions also depend strongly on the evaluation axis; for example, shared-label cell-line zero-shot yields MCC 0 when evaluated across cellular contexts but 0.52 when aggregated across drugs. Multimodal models likewise do not consistently outperform single-modality baselines, indicating that much of the predictive signal can often be captured by one modality alone. These results highlight the importance of realistic zero-shot evaluation, complementary aggregation views, and strong simple and single-modality baselines.

Primary area:
datasets and benchmarks
Keywords:
Single-cell, drug-response, representation learning, foundation models, molecular representations, evaluation design, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sKrROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper benchmarks evaluation methods and foundation models for predicting cancer cell drug responses. It compares cell foundation models and molecular foundation models within a unified framework covering cell representations, drug representations, and downstream prediction. It also investigates generalization under different settings, explicitly distinguishing pair-level, cell-line-level, and drug-level zero-shot generalization.

Critical strengths

The main strength is the systematic evaluation of cell and molecular foundation models within a unified framework, particularly the distinction between different levels of zero-shot generalization. This provides meaningful insights into model performance and represents a useful starting point for benchmarking drug response prediction.

Critical weaknesses
  • Limited dataset scale and label quality. The effective dataset size is much smaller than it appears. Although the shared-label dataset is seemingly large, it contains only slightly over 400 independent response labels. Moreover, both datasets contain only 10 drugs, which is insufficient to establish robust differences in molecular representation quality. Drug sensitivity labels are also defined inconsistently across datasets, as acknowledged by the authors, introducing potentially substantial noise. These limitations significantly weaken the benchmark's reliability and impact.
  • Limited evaluation of foundation models and downstream architectures. The authors use frozen foundation model embeddings without fine-tuning, which deviates from practical usage. Furthermore, the downstream predictor simply concatenates cell and drug embeddings, a straightforward approach that may not adequately capture cross-modal interactions. These choices limit how comprehensively the benchmark evaluates the capabilities of different foundation models.

Overall, the paper provides a useful starting point, but I do not find it sufficiently impactful or convincing as a benchmark in its current form.

Questions for authors

How might the conclusions change if foundation models were fine-tuned or more expressive architectures were used to model interactions between cell and drug representations?

1 comment

Critical Weaknesses

  1. Limited effective sample size, drug diversity, and label consistency.

We agree these are important limitations and now state them prominently (Section 5). Although the shared-label dataset contains many cells, it provides only 416 distinct cell-line–drug response labels. We report this effective sample size explicitly and evaluate at the pair level rather than treating cells sharing a label as independent observations.

"Each setting includes only 10 drugs, insufficient for robust rankings of molecular encoders": We do not claim that any drug representation is intrinsically superior. Our narrower conclusion is that, on currently available single-cell datasets, pretrained molecular representations show no consistent advantage over structural fingerprints. This is a limitation of the evidence data available for evaluating transfer, not evidence that molecular foundation models cannot succeed with broader drug coverage.

Label definitions are also heterogeneous. We deliberately separate two complementary regimes: shared labels, which are relatively standardized but coarse (assigned at cell-line–drug level but per single-cell transcriptome), and cell-specific labels, which capture cellular heterogeneity but combine different experimental readouts and labeling strategies. The principal patterns are: weak entity-level zero-shot discrimination and limited consistent gains from pretrained encoders. Those persist across both drug response label definitions, despite their differences. We now discuss the resulting uncertainty and limits on cross-study comparisons explicitly.

  1. Frozen representations and simple downstream architectures.

Our goal is a controlled assessment of released pretrained representations as transferable features, not a comprehensive benchmark of end-to-end training. Frozen inference is a valid practical use case and isolates representation quality from model-specific fine-tuning choices. Fine-tuning could help, but with only 416 distinct response labels in the shared-label setting, adapting models with >100M parameters risks substantial overfitting.

We also clarify that concatenation specifies the input, not the predictor’s expressiveness. All three predictors can learn cross-modal interactions: the MLP through nonlinear hidden layers, LightGBM through splits spanning cell and drug features, and xRFM through a learned full feature matrix encompassing both modalities. Nonetheless, architectures explicitly designed for interactions (e.g., bilinear layers or cross-attention) might capture them more efficiently. We now acknowledge this boundary in Section 5.

  1. Overall impact and reliability as a benchmark.

We appreciate the reviewer’s assessment that this is a useful starting point. We have substantially strengthened the manuscript in response to all reviewers. The revision clarifies our contributions relative to prior benchmarks and adds threshold-free metrics across all splits. It also states the selection criteria for the representations and expands the discussion of limitations and future directions. The paper’s central contribution is not a definitive ranking of foundation models. It is showing that evaluation choices change the scientific conclusion: the same predictions can look strong when aggregated but fail when scored within a drug or cell line, while prevalence baselines can match or outperform full models. Under cell-line holdout, MCC changes from 0.52 within cell lines to −0.03 within drugs, while a drug-prevalence baseline reaches 0.58.

Questions for Authors

How might fine-tuning or more expressive interaction architectures change the conclusions?

Using foundation models as frozen feature extractors is a common practical usage, and it tests their central promise: transferable representations that can be reused without task-specific retraining. Fine-tuning might benefit the prediction task but would confound representation quality with task adaptation and require per-model tuning, undermining the controlled comparison. Benefits under unseen-cell-line or unseen-drug holdout remain an open empirical question, and we would expect them to be constrained by limited drug diversity and strong marginal effects: single-modality models already track prevalence baselines. With 416 independent labels in the shared-label setting, fine-tuning models with over 100M parameters would also carry a high risk of overfitting.

We expect smaller gains under entity-level zero-shot splits. Our ablations show that much of the signal is marginal (single-modality models track prevalence baselines), and with 9 training drugs per drug zero-shot fold, no architecture has much chemical diversity to extrapolate from. This is consistent with bulk benchmarks, where complex architectures do not reliably outperform tree-based baselines (Bernett et al., 2026). We have clarified these points in Section 5.

Reviewer_iJDyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors have developed an evaluation framework for cell and drug representations, in the context of drug response prediction. They evaluate 5 cell representations, 6 drug representations and 3 predictive models for this. They also consider 2 does - a shared-label setting and a cell specific setting. The key contribution is the evaluation across these representation types.

Critical strengths

The problem of cell line drug response prediction is a literature-rich field, with most papers opting for specific representation choices like Morgan fingerprint for drugs. This paper provides a systematic framework for varying this. The shared-label and cell specific setting is well motivated.

Critical weaknesses

The problem is relevant, but appears to have been tackled in prior work. Distinguishing the need for the proposed evaluation method needs more motivation and elaboration. The authors talk about multimodal inputs, but the data appears to be only related to single cell RNA-seq data. The models used for evaluation are classical methods, whereas more advanced approaches on cell-line drug response prediction are available. As such, conclusions drawn from these approaches may not hold good for these advanced approaches.

Questions for authors
  1. How does the evaluation framework compare or add over existing frameworks like [1], [2]?

  2. How are the findings from this paper adding over those in [3], [4]?

  3. The authors talk about avoiding highly imbalanced response distributions in section 3.1. How realistic is this given real world data can be imbalanced?

  4. In the cell-specific label setting, do you only train on samples of a specific drug? Otherwise a single cell line may be mapped to different responses if only drug representation is naively removed.

  5. GeneFormer is consistently ranked high across splits and evaluation - is there any intuition for this?

  6. How does this evaluation work on broader datasets, beyond Tahoe? Generalization appears to be a key claim, but is not adequately supported.

[1] Bernett, J., Iversen, P., Picciani, M., Wilhelm, M., Baum, K., & List, M. (2026). Critical evaluation of drug response prediction models with DrEval. Nature Communications, 17(1), 4238. [2] Partin, A., Brettin, T. S., Zhu, Y., Narykov, O., Clyde, A., Overbeek, J., & Stevens, R. L. (2023). Deep learning methods for drug response prediction in cancer: predominant and emerging trends. Frontiers in medicine, 10, 1086097. [3] Abdel-Rehim, A., Tate, E., Soldatova, L. N., & King, R. D. (2026). Drug response profile-based machine learning enables strategic cell line and compound selection for drug development. Bioinformatics, 42(6), btag293. [4] Herbert, W. G., Chia, N., Jensen, P. A., & Walther-Antonio, M. R. (2026). Monotherapy cancer drug-blind response prediction is limited to intraclass generalization. PLoS computational biology, 22(4), e1013232.

2 comments

Critical Weaknesses

Novelty: DrEval [1] benchmarks bulk-screen predictors; [2] reviews this field. Existing single-cell benchmarks largely vary cell encoders, while molecular benchmarks fix or omit cellular inputs. We cross-evaluate five cell and six drug representations under fixed predictors, across two label settings, pair/cell-line/drug holdouts, two scoring axes and prevalence controls. This reveals an evaluation effect missed by single-axis benchmarks: under cell-line holdout, identical predictions yield MCC 0.52 within cell lines but −0.03 within drugs, versus 0.58 for a drug-prevalence baseline.

Modalities and advanced methods. “Multimodal” denotes transcriptomes plus chemical structure, not multi-omics; we clarified this wording. We deliberately fix simple predictors to isolate representations, but include three single-cell foundation models and four pretrained molecular encoders. Our main conclusions concern leakage, marginal effects and scoring choice, rather than architectural superiority: all predictors score near zero within drugs under cell-line holdout (Table 12). We acknowledge that end-to-end models might yield different rankings.

Questions for Authors

  1. "How do we extend [1,2]? ": Unlike bulk-focused work, we evaluate single-cell inputs, including cell-level labels, and jointly vary cellular and chemical representations. In addition to corroborating the importance of simple baselines, we show that (i) scoring axis can reverse conclusions under entity holdout (MCC 0.52 vs −0.03), and (ii) repeated cells across drugs can leak information: without cell blocking, within-drug MCC is higher for 88/90 configurations (Table 11), although training fractions also differ. We now discuss [1,2].

2." How do we extend the work of [3,4]?": In [3] shows that bulk response-profile descriptors outperform expression; [4] finds that drug-blind prediction generalizes primarily within drug classes. Consistently, our single-modality models track prevalence, while drug holdout yields within-cell-line MCC 0.14 on a chemically diverse panel (median Tanimoto similarity 0.08). We extend these observations to single cells and cell-level labels while comparing both representation modalities and scoring axes. We do not directly test [3]’s response-profile descriptors or [4]’s mechanism-of-action dependence.

  1. "How realistic is filtering imbalance?": This is a metric-reliability choice, not an assumption that real responses are balanced. Retained drugs range from 21–85% sensitive; excluded drugs have 2–14% minority prevalence among only 39–43 cell lines, often leaving 1–2 minority cases and unstable within-drug MCC. Cell-specific data remain unfiltered (72.7% sensitive overall, some units >87%). Section 3.1 clarifies that assessing excluded drugs requires class counts, undefined-metric handling and uncertainty estimates.

  2. "How is the cell-only ablation trained?": A single model is trained jointly across drugs and cell lines. Removing drug representations intentionally tests how much can be predicted without drug identity, even if one cell receives different drug-dependent labels. In shared-label data, cell-only MCC matches line-prevalence controls (0.455 vs 0.458; leakage-free pair holdout). In cell-specific data, only 821/60,688 cells (1.4%) appear under multiple drugs, with no conflicting labels, and only 3/13 cell-line units span drugs. Drug and cellular context are therefore largely confounded, helping explain similar cell-only and full-model scores (Section 4.2).

5." Why Geneformer is consistently ranked higher? " Its advantage occurs in the cell-specific setting, not consistently in shared-label data. Expression-rank encoding may be more robust to heterogeneous quantification than binned expression; cancer-domain pretraining may help capture within-line states. These are hypotheses, not proven mechanisms: pretraining, vocabulary and checkpoint differences may contribute. Strong relative rankings also coexist with weak absolute zero-shot results. We added this caveat in Section 4.4.

  1. "How does this evaluation work on broader datasets, beyond Tahoe? " We also evaluate 19 published datasets from 16 sources and 13 biological systems, including patient-derived models and xenografts with different response labels. The same broad patterns persist: limited pretrained-encoder advantages and weak zero-shot discrimination. We distinguish unseen-entity holdout (split) from within-drug/within-cell-line scoring (task). We do not claim cross-dataset generalization, which we have not tested; the Abstract and Section 3.3 now clarify this.
Reviewer_iJDyOpenReview ↗

Response to rebuttal

Thank you for the response. My key concern is the superior performance of Geneformer and whether the hypotheses that the authors have on it will affect the conclusions from this paper.

Reviewer_JnryOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors assemble public and private dataset to test the quality of 5 transcriptomic embedding methods and 6 chemical/druge embedding methods across 4 generalization settings: randomized control, pair zero-shot (unseen cell-drug pairs), cell-line zero-shot (unseen cell lines), and drug zero shot (unseen drugs). Results indicate that current models do not beat simple baselines.

Critical strengths
  • The paper is well evaluated across several settings of generalization: randomized control, pair zero-shot (unseen cell-drug pairs), cell-line zero-shot (unseen cell lines), and drug zero shot (unseen drugs). These are highly relevant to the development of biological and chemical foundation models.
  • The presentation of results is clear and the honest presentation of results is important for the community.
Critical weaknesses
  • The models (scVI, PCA, scGPT, Geneformer, GeneJEPA) are not a complete sampling of the model space -- its unclear how the deep learning models were chosen, especially since they can use different genes and perhaps different training data sources. Retraining using standardized datasets might make this a better aplpes to apples comparison. Perhaps similarly for the drug models.
  • It is highly unclear what the takeaway is or how to fix the issues highlighted. Are the models not good enough? What would improve things? Calibration across multiple axes (including unseen drugs/cells) in a controlled setting would be useful.
  • It is not clear if the benchmarked dataset that was collected will be released. This value of this paper would be significantly decreased as a benchmark without public release of the dataset used to evaluate.
Questions for authors
  • Will you release the dataset?
  • What can be done to improve generalization performance? What are the takeaway from the models chosen? How could we choose better models for evaluation to highlight this?
1 comment

Critical Strengths

We thank the reviewer for recognizing the clarity, soundness, and breadth of our evaluation, particularly the importance of testing distinct generalization settings and reporting negative results transparently. We revised the manuscript to sharpen its central message and practical implications.

Critical Weaknesses

  1. Model selection and comparability.

Our panel is representative, not exhaustive: it evaluates whether released, frozen representations improve upon simple task-specific alternatives, rather than ranking every available architecture. We include PCA and fold-fitted scVI; Geneformer, scGPT, and GeneJEPA, representing distinct transcriptomic pretraining approaches; and structural fingerprints alongside four pretrained drug encoders.

All cell methods receive a common input gene panel before model-specific vocabulary mapping (Appendix C.1). Retraining all foundation models on one corpus would test architectures rather than the pretrained representations available to practitioners, and is beyond our scope. Importantly, scVI is refit on standardized training data within each fold.

Breadth comes from the evaluation design: 90 model configurations, two label settings, five split strategies, two scoring axes, prevalence controls, and single-modality ablations. We added Appendix Fig. 10 and Table 6 to document model architectures and pretraining corpora, and clarified this limitation in Section 5.

  1. Unclear takeaways, remedies, and calibration.

We reorganized the Discussion around three findings.

First, random splits give optimistic estimates: within-drug MCC falls from 0.88 (random) to 0.18 (leakage-free pair holdout).

Second, apparent zero-shot prediction often reflects marginal effects rather than transferable interactions. Under cell-line holdout, within-cell-line MCC is 0.52, below a drug-prevalence baseline (0.58), while within-drug MCC is −0.03. Under drug holdout, within-drug MCC is 0.42 versus 0.44 for a cell-line-prevalence baseline.

Third, pretrained encoders do not consistently outperform PCA, scVI, or fingerprints.

These findings point to data coverage and evaluation design—not simply model capacity—as key bottlenecks. We now recommend more diverse, systematically crossed drug–cell screens; entity-level holdouts scored on the task-relevant axis rather than pooled predictions; and routine marginal/prevalence and single-modality controls.

We agree that the fixed 0.5 threshold makes MCC sensitive to calibration. We therefore added threshold-independent AUROC comparisons across splits and scoring axes (Fig. 3B), ablations (Fig. 4B), and all pair-holdout configurations (Supplementary Figs. 12–13). AUROC tests whether conclusions persist without thresholding, though it is not itself a full calibration analysis.

  1. Dataset availability.

The benchmark is already publicly released under CC BY 4.0. Zenodo (DOI: 10.5281/zenodo.22962632) contains processed single-cell inputs, cell/drug embeddings, and full result tables; the linked GitHub repository (https://github.com/anonymoussinglecell/iclr2027) includes preprocessing, training, evaluation, and figure-generation code. Both are linked in the Reproducibility Statement.

Questions for Authors

  1. Will the dataset be released?

Yes, it is already available, with processed data, embeddings, results, and code, as detailed above.

  1. How can generalization improve, what do the models teach us, and what should be tested next?

Our ablations show that single-modality models often track prevalence baselines; with only 10 drugs per setting, limited chemical diversity restricts learning transferable effects. Larger crossed screens and interaction-aware predictors could help separate true cell–drug interactions from marginal effects.

In the shared-label setting, pretrained cell encoders do not reliably beat PCA/scVI; in the cell-specific setting, Geneformer leads across splits, but absolute zero-shot performance remains low. Pretrained drug embeddings offer no consistent gain over fingerprints, while predictor choice matters more (LightGBM is best in each cell-specific split).

Future comparisons should add (i) fine-tuned encoders, (ii) perturbation-pretrained cell models evaluated without pretraining overlap—GeneJEPA was trained on Tahoe-100M, which includes our control cells—and (iii) mechanism-aware drug representations incorporating targets, pathways, or perturbation signatures.

These extensions are now highlighted in Section 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.