acceptodds
Under review as a conference paper at ICLR 2027

Right for the Right Reasons? Testing Whether Cellular Profiles Explain Gene-Drug Retrieval

Abstract

Image-based cellular profiling records how cells change after a gene is modified or a drug is applied. Comparing these responses can reveal gene–drug relationships and help prioritize candidates for drug discovery and repurposing. However, a high retrieval score does not show why a model recovered a known relationship. The model may use the cellular response, favor genes or drugs with many recorded partners, or exploit the relationship graph, in which genes and drugs are nodes and recorded relationships are edges. We introduce a controlled evaluation framework that asks two questions: does a model recover known gene–drug relationships better than randomized relationships, and do cellular features or graph connections support that recovery? The framework combines held-out retrieval with degree-preserving randomized relationships. It separates cellular features, which describe observed cell responses, from graph topology, the pattern of recorded gene–drug connections. We evaluate RxRx3-Core, CPJUMP1, MOTIVE-CRISPR, and a private angiogenesis dataset. CPJUMP1 supports recovery in both directions, RxRx3-Core is stronger from drugs to genes, and the angiogenesis data have no consistently dominant direction across metrics. MOTIVE-CRISPR supports nonlinear feature-based retrieval, while its graph controls expose the same structural cautions. Among the tested models, adding nonlinearity or graph message passing therefore does not consistently produce stronger or more informative alignment. The main contribution is a practical way to determine whether a result is supported by cellular information, graph structure, or both. This provides a clearer basis for selecting gene–drug candidates for experimental follow-up.

Primary area:
datasets and benchmarks
Keywords:
phenotype alignment, image-based profiling, cross-modal retrieval, controlled evaluation, graph neural networks, self-supervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RH7iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether image-based cellular profiles help retrieve known gene–drug relationships, and whether retrieval performance is supported by cellular features or by the topology of the recorded relationship graph. The authors introduce a controlled evaluation framework with two questions: whether a model retrieves known relationships better than degree-preserving randomized relationships, and how cellular features and graph connections affect retrieval. The framework combines query-held-out retrieval and degree-preserving nulls with graph interventions, including disabling message passing and rewiring edges, and feature interventions, including zeroing and shuffling features. The evaluation covers RxRx3-Core, CPJUMP1, MOTIVE-CRISPR, and a private angiogenesis dataset represented by handcrafted features and OpenPhenom embeddings. CCA and MLP serve as non-graph baselines; Sup-GINE and SSL-GINE are used to study graph-based approaches. The results vary by dataset and model: CPJUMP1 shows retrieval above the null in both directions, RxRx3-Core is stronger from drugs to genes, MOTIVE-CRISPR shows positive excess AUROC for the MLP, and the angiogenesis representations show strong early retrieval but little excess AUROC over the null. The interventions also show that features and graph connections can help or hurt retrieval depending on the setting. Overall, the experiments support the authors’ caution that a high retrieval score alone does not establish that cellular profiles explain gene–drug alignment. However, the results do not yet provide a clear account of when cellular profiles reliably help recover these relationships.

Critical strengths

(1)The paper asks an important question: whether retrieval of known gene–drug relationships is supported by cellular information or by the structure of the recorded relationship graph. The degree-preserving null and the separate feature and graph interventions provide a more informative evaluation than retrieval scores alone. (2)The results are heterogeneous rather than uniformly favoring a particular model or modality. This supports the paper’s caution that high retrieval scores should not automatically be interpreted as evidence of phenotype alignment.

Critical weaknesses

(1)The paper’s second research question asks how cellular features and relationship topology contribute to retrieval, with the goal of clarifying when cellular profiles help recover gene–drug relationships. The ablations show that features and graph connections can help, hurt, or have little effect depending on the dataset and model. However, the paper does not synthesize these observations into a clear explanation of when or why cellular profiles provide useful support. As a result, the experiments characterize context-dependent effects but do not fully deliver the stated explanatory contribution. (2)In Table 4, shuffling features lowers Sup-GINE AUROC on the published datasets but raises it for both angiogenesis representations. Since these representations come from the same private dataset, they are not independent replications. The result raises a specific concern about the angiogenesis data or its processing, including feature-to-entity correspondence, preprocessing, or assay-related confounds. The paper should provide evidence that these possibilities have been checked.

Questions for authors
  1. The current description is insufficient to verify that the evaluation is leakage-free. For each fold, whether held-out query nodes retain any incident edges, and whether degree-preserving nulls are generated from the full graph or training edges only. Also state whether each null is evaluated with a retrained model or fixed embeddings.
  2. In Table 4, shuffling features lowers Sup-GINE AUROC on the published datasets but raises it for both angiogenesis representations. The two angiogenesis representations come from the same private dataset, so this is one dataset-specific discrepancy, not an independent replication. Report quality-control checks of the entity–profile–label mapping, profile aggregation and normalization, and assay or batch effects, and explain the improvement under feature shuffling.
  3. Across Sections 4.2–4.4, the paper reports retrieval performance relative to the degree-preserving null (Table 2), topology interventions (Table 3), and feature interventions (Table 4), with additional qualitative observations in Figures 3–4. However, these findings are discussed mainly as dataset- and model-specific outcomes; the paper does not establish when retrieval above the null is supported by the correct cellular features rather than by graph structure. The authors should synthesize the results across these experiments and state what evidence-based conclusion they support about when cellular profiles help recover gene–drug relationships. If the experiments only demonstrate that the effects vary across settings, the paper should limit its claim accordingly.
  4. In panels A and B, the gene branch shows Gene Phenotype → encoder → Gene Vectors/Embeddings, whereas the drug-branch arrows point toward Drug Phenotype and do not show the corresponding output path consistently. State the intended drug-branch computation and correct the figure to show Drug Phenotype → encoder → Drug Vectors/Embeddings, if that is the implemented architecture.
Reviewer_6jaHOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an evaluation framework for studying whether a cellular profile is explained by gene or drug perturbation. The paper evaluates retrieval of gene-drug pairs with linear (CCA), non-linear (MLP), supervised GNN (GINE), and self-supervised GNN (GINE) models using four datasets including image profiles of cells under gene or drug perturbations. The paper also studies how different graph connections and message passing affect gene-drug retrieval. CCA and MLP are considered as baselines to GINE models which are further studied. The evaluation framework is used to study retrieval on four datasets of which three are public and one is private. The results show dataset specific variation between retrieval directions.

Critical strengths
  • The paper presents an interesting approach for examining cellular profiles and associations between gene and drug perturbed samples.
  • The paper evaluates four different datasets (one using two sets of representations) showing that gene-drug retrieval is dataset specific.
Critical weaknesses
  • The focus of the paper is not clear. It presents an evaluation framework, but there is no software available that others could use to test with other datasets. The paper compares GINE methods to basic baselines, but not to other existing methods. Finally, GINE architectures are evaluated but without clear conclusions what causes difference in retrieval AUC and why changes alter directions of better retrieval.
  • Figure 1 (A,B) seems misleading. Those show either separate Linear Projection or MLP Encoder models from Drug/Gene Phenotype to Drug/Gene Embeddings (also in the case of drug perturbations, both arrows point as output from the model). It is not defined what the Drug/Gene phenotypes are. I assume those are the available metadata information. However, the embedding is not created from metadata but from other embedding if I understood the details correctly. Technical description also details that MLP Encoder has two separate encoder branches (supplementary line 74). It would be important to clarify technical architectures in detail.
Questions for authors
  • See weak points above.
  • The paper is missing comparisons to other existing methods.
  • The existing results do not lead to clear conclusions except that gene-drug pair retrieval is dataset specific.
  • Technical description should be clarified.

Minor comments:

  • Table 4: Sup-GINE best results are with zeroed features? Does that mean the profiles are less important for the retrieval than the graph structure?
  • Could the Angio dataset be considered simplistic (it is 1-ch data?) due to the high edge retention in rewiring? Could that explain why retrieval fails with this dataset ie. there is just no information in the current dataset to link gene-drug pairs?
  • Figures 3 & 4 are presented in Discussion though these are Results.
Reviewer_owY1OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether successful gene–drug retrieval reflects cellular-profile information, relationship-graph structure, or both. It combines query-held-out evaluation with a foldwise degree-preserving label-randomization reference for non-graph models and matched feature/topology interventions for graph models. Experiments cover four datasets through five dataset–representation inputs, using CCA, an MLP, supervised GINE, and self-supervised GINE. The results illustrate that high retrieval scores need not imply query-specific cellular correspondence and that the benefit of features or message passing depends on the dataset and model.

Critical strengths
  1. The paper addresses an important evaluation problem and provides a concrete diagnostic example. Recovering recorded gene–drug relationships is not equivalent to showing that the correct cellular profiles contribute to prediction. The CPJUMP1 degree-only baseline reaches 0.903 AUROC, close to the zero-feature supervised GNN's 0.899, demonstrating how a strong score can arise without query-specific discrimination. This is a useful caution for interpreting biological retrieval benchmarks.

  2. The SI makes the controlled evaluation substantially more credible and transparent. It distinguishes fixed-ranking, degree-preserving label randomization from training-graph rewiring; supplies matched interventions and paired statistical tests with multiplicity correction; and reports limitations rather than concealing them. Together with the multi-dataset, bidirectional analysis, these controls provide useful empirical evidence about the dependence of retrieval on features and topology.

Critical weaknesses
  1. The substantive advance over prior evaluation work is not established strongly enough. MOTIVE already compares morphology-only, topology-only, and combined predictors, includes cold-source/cold-target evaluation, and studies the relative contribution of features and graph structure (Arevalo et al., 2024, §§3.4 and 4.1–4.3; its topology-only comparisons are in the random-split setting). The present paper adds a degree-preserving conditional null, more systematic matched interventions under query-held-out evaluation, and a broader dataset panel. These distinctions deserve credit, but the contribution remains primarily an integration of established evaluation tools, and the broad 'first solution' claim needs narrower positioning. A matched demonstration of which conclusions change relative to existing protocols would make the added methodological or empirical value clearer. This limited demonstrated advance is my principal reason for weak rejection, not an assertion that the framework is identical to MOTIVE or requires a new architecture. I would reconsider if the authors establish a substantive new inference, generalizable finding, or practical benefit.

  2. The reproducibility and reusable-artifact commitments are insufficiently specified for a practical evaluation framework. SI §A describes Angio preprocessing, but neither supplied PDF gives a concrete reviewer-access route or release arrangement for the private data. The future code/embedding release does not clearly cover both feature matrices, relationship labels and their provenance, or fixed splits. The provenance of the 5,931 Angio relationships is particularly relevant to the degree-based analyses. It is also unclear whether the release will provide a minimal, documented implementation that others can readily apply. This limits independent verification of the Angio findings and assessment of the framework's practical value. Private data are not inherently invalid, and derived artifacts could support downstream reproducibility even if raw images cannot be released. This concern is based on the supplied PDFs; no separate submission archive was inspected.

Questions for authors
  1. Novelty and significance (clarification central to my recommendation): What precise inference or empirical insight does this framework enable beyond MOTIVE and prior degree-aware evaluation? Can the authors identify a substantive conclusion that existing protocols would miss, preferably using a matched public-data comparison? Such a comparison should distinguish changes in evaluation from changes in the perturbation set, graph construction, and training objective. A demonstrated added contribution, rather than only narrower wording, could change my recommendation.

  2. Reproducibility and practical contribution (clarification central to my recommendation): What reviewer-access or release commitment covers Angio raw images or, where image release is restricted, the derived feature matrices, relationship labels and their provenance, fixed splits, and runnable analysis code? Will the framework be provided as a minimal, documented, reusable implementation with a public-data example? Please distinguish resources available during review from planned future releases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.