- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors of CREST propose a metric learning framework that learns a shared representation of perturbation-context effects on cells and uses it for retrieval on previously unseen cell lines or to predict perturbations' effects. CREST outperforms simple baselines on the LINCS1000 dataset, and it also shows promising results on the Replogle RPE-1 dataset in a cross-assay experiment.
Critical strengths
The paper is mostly well written, and the abstract and introduction are engaging. The authors clearly define the general problem and provide a convincing rationale for their approach. I like Figure 1, and I think it is a good summary of the work. I would slightly increase the font size and change the orange font to another colour, though - currently it's barely legible. I appreciated appendices E and F, though some of that material should definitely move to the main text to support important claims.
Critical weaknesses
I am not entirely convinced of the novelty of the approach and of its effective contribution to the field. The methodology is indeed not new, as contrastive alignment with all the subsequent machinery and softmax-kNN retrieval/prediction are well documented and established in different domains and applications. As far as the specific application is concerned, I believe there is some adjacent work that predates this study.
Conceding that what is stated above is debatable, I have some other concerns:
- The results are measured only on 1 dataset per type of experiment, on 3 different seeds rather than splits, and no statistical test has been performed. Moreover, the splits for the main results have very limited val and test sizes;
- I believe the empirical attribution is not sufficiently solid: the comparison does not isolate contrastive learning from retrieval-based prediction, dimensionality of latents, or trivial averaging effects. Some elements in the architecture have not been ablated, such as the k elements in the top-k retrieval, or competing baselines' parameters;
- This approach is trivially bound to fail whenever any queried perturbation-context tuple falls outside the convex hull of the measured reference responses. As I understand it, this model has no way to avoid this, and no uncertainty measure could ameliorate it.
Overall, I think this paper is not ready for publication as is, but I am willing to reconsider if persuaded.
Questions for authors
- Q1: Can the authors compare CREST against simpler retrieval-based baselines, such as PCA-kNN and MSE+kNN, and evaluate CREST with an unconstrained regression decoder? This would help disentangle the contributions of contrastive learning, retrieval, and averaging.
- Q2: Can the authors extend the evaluation to additional datasets or, alternatively, explore different train/test splits to assess robustness and out-of-distribution generalization? In particular, I would appreciate confidence intervals across independently held-out cellular contexts, as well as experiments involving unseen compounds or scaffold-based splits.
- Q3: Can the authors disentangle the effects of context diversity and training-set size? For example, could they compare models trained on the same number of observations drawn from different numbers of cellular contexts?
4 comments
First, I really appreciate the care and effort put into the rebuttal. The authors have clearly tried their best to accommodate my requests and concerns.
- Figure 1 is much more readable now - well done.
- Evaluation and statistics have been partially addressed; I would still like to see the statistical tests. The evaluation is overall a bit limited by the fact that the splits involve only five distinct held-out cell lines, and the primary experiments still rely on a single dataset. The new chemical OOD experiment is appreciated, although the results look relatively modest. CREST's Retrieval@10 performance on structurally novel compounds is very close to PCA-kNN and Latent Additive. I believe this deserves more emphasis when discussing generalisation to previously unseen perturbations.
As a general recommendation, I believe empirical ML studies should always quantify the uncertainty associated with reported performance estimates and support comparative claims with appropriate statistical tests. Reporting point estimates alone is rarely sufficient to establish that observed improvements are robust or statistically meaningful. I would encourage the authors to follow this advice in the future, following established practices in statistical learning and model evaluation.
- Discrepancy of results: there is a difference between the original and revised discrimination-rank results. In the revised Table 3, the per-perturbation mean baseline slightly outperforms CREST on discrimination rank in both modalities, while PCA-kNN is also very competitive. Could the authors clarify if these differences are attributable to the revised splits, or whether the evaluation protocol, metric implementation, or retrieval configuration has also changed?
- Convex-hull limit. I appreciate the additional analyses and the explicit acknowledgement of the convex-hull limitation. The fixed-atlas scaling experiment is particularly useful in separating representation quality from reference coverage. Still, the oracle comparison does not resolve the extrapolation limitation, since the oracle selects reference responses using information from the ground-truth test response. I still think introducing a measure of uncertainty or OOD-reliability would be valuable for practical applications, although I recognise that addressing extrapolation directly may be beyond the scope of the proposed retrieval framework.
- learned geometry. Moving the pathway enrichment and MoA-coherence analyses into the main text is a welcome improvement, as is the new gene KO–compound retrieval experiment. I still find the biological interpretation somewhat stronger than the supporting evidence. The MoA-coherence analysis measures similarity within biological mechanisms, but does not establish how effectively distinct mechanisms are separated. I think this would require a set of experiments that perhaps would not fit too well into this narrative, so I would suggest reconsidering the narrative instead.
OVERALL
The rebuttal has substantially improved my assessment of the paper. In particular, the additional retrieval baselines, hyperparameter analyses, multiple holdout splits, and controlled scaling experiments address several of my original concerns convincingly. My remaining reservations concern primarily the degree of methodological novelty, the limited diversity of biological evaluation settings, and the strength of the claims regarding the learned representation's biological meaning and generalisation capabilities.
I recognise the value of the proposed framework and the empirical improvements demonstrated by the authors. I suggest that the other reviewers consider these elements before the final discussion, too.
I believe these remaining limitations prevent me from supporting acceptance at this stage. In recognition of the substantial improvements and the authors' constructive response, I am raising my recommendation from clear rejection to weak rejection.
We thank the reviewer again for carefully re-evaluating our revised manuscript, for recognizing the substantial improvements made during the rebuttal, and for raising your assessment of our work. We briefly address each of your remaining points below:
-
Figure 1 readability. Thank you!
-
Evaluation, statistical tests, and chemical OOD generalization.
- Statistical tests & uncertainty: We have now added paired statistical tests alongside split-level standard deviations across the manuscript to better ground comparative claims.
- Chemical OOD generalization: In Section 4.2 and Table S5, we now explicitly emphasize that while CREST matches or outperforms baselines on structurally novel compounds, absolute retrieval accuracy remains modest across all methods. We also note in Section 5 that incorporating richer molecular and biological priors and/or featurizations is an important next step for out-of-vocabulary and OOD perturbations.
-
Discrepancy in discrimination-rank results. Thank you for catching this difference, and we apologize for forgetting to note it earlier. The change stems from how CREST's predictions were scored rather than from the splits or metric implementation: in the initial submission, CREST's discrimination rank was computed using cosine similarities in CREST's learned embedding space (via ). However, given that the task in Table 3 is transcriptional response prediction—and to ensure a fair, apples-to-apples comparison across methods—we decided it was more appropriate to evaluate CREST using its softmax-kNN-predicted profiles directly in expression space, matching how all other response prediction baselines are scored (Table 3, Appendix C).
-
Convex-hull limitation and OOD reliability. We agree that the empirical oracle does not resolve the convex-hull extrapolation limit, as it selects from the same reference atlas using ground-truth target shifts. Our cell-line holdout and multi-context single-cell evaluations partially probe cross-context generalization (Sections 4.2, 4.4, and 4.6), and our cross-line similarity analyses (Appendices G and M) show that baseline transcriptional similarity to the nearest training line strongly correlates with CREST's retrieval and response prediction accuracy. While this similarity can provide a practical indicator of OOD reliability, we agree that more directly assessing OOD responses for transcriptional response prediction would be valuable directions for future work.
-
Learned geometry and biological interpretation. Thank you for the feedback on moving the pathway enrichment, MoA-coherence, and gene KO–compound matching analyses into the main text. Following your suggestion, we revised Section 4.3 to better align our framing with the supporting evidence, focusing on CREST's preservation of shared mechanistic similarity across cellular contexts rather than claiming separation between distinct biological mechanisms.
Finally, as mentioned in our latest general comment above, we have added a new evaluation on a multi-context single-cell Perturb-seq inverse retrieval benchmark (Section 4.6, Appendix R). We hope that this new single-cell benchmark, together with the added statistical tests and calibrated biological narrative, addresses your remaining concerns, particularly regarding the diversity of biological evaluation settings.
Thank you for working on this and updating the manuscript. I am much happier with this current version; it is now a much stronger and more appropriate comparison and study. I am still relatively unsure of the fairness of some of the comparisons, but I acknowledge that this is a tough task (first-hand experience), and I am willing to reward the excellent responsiveness and work done in the last couple of weeks. I am sorry my colleagues did not interact as much with your work, but I hope my comments helped improve this work. Please finish adjusting the narrative and tidying up the text. I thank you for your work and raise my score to weak accept (cannot, in all fairness, ignore my last doubts). Best, mKBY
We thank the reviewer for the careful reading, the concrete experimental suggestions, and the openness to reconsideration. We have tried to address each point directly in the revised manuscript.
Novelty. We agree that contrastive alignment and softmax-kNN are established building blocks; our intended contribution is how they are formulated and combined for candidate prioritization and perturbation modeling (G1). As detailed in Section 3 (see also G3 and Reviewer 1, Item 1), conditioning perturbation embeddings on measured CCLE expression via gated cross-attention before contrastive alignment, rather than operating on unconditioned representations or text descriptions, yields large gains over our PertOmni-style encoder across Condition Retrieval@10, Context Retrieval@1 (Table 1), and cross-modality gene KO–compound matching (Table 2c).
Evaluation and statistics. Please see G2. All LINCS experiments now use five cell line holdout splits rather than one split with three seeds, reported as mean ± SD across splits (Section 4.1, Tables 1–3) and per held-out cell line across the three splits (Appendices G, L, and M).
Contrastive objective and hyperparameter ablations.
Reference atlas coverage and convex-hull limitations. Please see G4. We agree that non-parametric retrieval cannot extrapolate beyond the convex hull of reference responses, and Section 5 now states this limitation explicitly. Within that same convex hull, however, the Table 3 oracle and fixed-atlas scaling analysis (Figure 3, Appendix P) show that better representations and neighbor weighting steadily improve predictions from the exact same reference set, indicating substantial headroom within existing atlas boundaries.
Presentation. The response-projector pathway enrichment and cross-cell-line MoA-coherence analyses have been moved into the main text (Section 4.3, Figure 2, Table 2a,b). For Figure 1, we have increased the label font sizes and replaced the low-contrast orange text with a higher-contrast color.
Q1. Please see G3 and Contrastive objective and hyperparameter ablations above for the PCA-kNN and matched MSE/wMSE+kNN retrieval controls (Appendix N). For the unconstrained regression-decoder comparison, the matched MSE and wMSE backbones (Table 3) pair the identical condition encoder with a linear decoder trained via reconstruction loss rather than contrastive retrieval.
Q2. Please see G2 for our five cell line holdout splits, per-line mean ± SD across held-out contexts (Appendix L), strict chemical out-of-distribution evaluation on structurally novel compounds (Tanimoto similarity ≤0.4; Appendix F), and context-similarity analyses (Appendices G and M).
Q3. We agree that adding cell lines in the scaling analysis (Section 4.5) also increases total training data, confounding the two effects. We therefore added a controlled ablation (Appendix Q) that varies context diversity while holding training-set size fixed. Specifically, gene-KO CREST models are trained on the same 4,000 knockouts, each observed in r cell lines (N=4,000r total profiles) distributed across L≥r training cell lines, with the held-out test set kept identical across all configurations. Comparing settings with the same N but different L isolates context diversity: spreading a fixed number of observations across more cell lines consistently improves Condition Retrieval@10, whereas increasing the number of observations per knockout (r) yields a substantially larger gain. Thus, context diversity improves generalization beyond data volume alone, though data volume remains the dominant driver at this scale.