- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether image-based cellular profiles help retrieve known gene–drug relationships, and whether retrieval performance is supported by cellular features or by the topology of the recorded relationship graph. The authors introduce a controlled evaluation framework with two questions: whether a model retrieves known relationships better than degree-preserving randomized relationships, and how cellular features and graph connections affect retrieval. The framework combines query-held-out retrieval and degree-preserving nulls with graph interventions, including disabling message passing and rewiring edges, and feature interventions, including zeroing and shuffling features. The evaluation covers RxRx3-Core, CPJUMP1, MOTIVE-CRISPR, and a private angiogenesis dataset represented by handcrafted features and OpenPhenom embeddings. CCA and MLP serve as non-graph baselines; Sup-GINE and SSL-GINE are used to study graph-based approaches. The results vary by dataset and model: CPJUMP1 shows retrieval above the null in both directions, RxRx3-Core is stronger from drugs to genes, MOTIVE-CRISPR shows positive excess AUROC for the MLP, and the angiogenesis representations show strong early retrieval but little excess AUROC over the null. The interventions also show that features and graph connections can help or hurt retrieval depending on the setting. Overall, the experiments support the authors’ caution that a high retrieval score alone does not establish that cellular profiles explain gene–drug alignment. However, the results do not yet provide a clear account of when cellular profiles reliably help recover these relationships.
Critical strengths
(1)The paper asks an important question: whether retrieval of known gene–drug relationships is supported by cellular information or by the structure of the recorded relationship graph. The degree-preserving null and the separate feature and graph interventions provide a more informative evaluation than retrieval scores alone. (2)The results are heterogeneous rather than uniformly favoring a particular model or modality. This supports the paper’s caution that high retrieval scores should not automatically be interpreted as evidence of phenotype alignment.
Critical weaknesses
(1)The paper’s second research question asks how cellular features and relationship topology contribute to retrieval, with the goal of clarifying when cellular profiles help recover gene–drug relationships. The ablations show that features and graph connections can help, hurt, or have little effect depending on the dataset and model. However, the paper does not synthesize these observations into a clear explanation of when or why cellular profiles provide useful support. As a result, the experiments characterize context-dependent effects but do not fully deliver the stated explanatory contribution. (2)In Table 4, shuffling features lowers Sup-GINE AUROC on the published datasets but raises it for both angiogenesis representations. Since these representations come from the same private dataset, they are not independent replications. The result raises a specific concern about the angiogenesis data or its processing, including feature-to-entity correspondence, preprocessing, or assay-related confounds. The paper should provide evidence that these possibilities have been checked.
Questions for authors
- The current description is insufficient to verify that the evaluation is leakage-free. For each fold, whether held-out query nodes retain any incident edges, and whether degree-preserving nulls are generated from the full graph or training edges only. Also state whether each null is evaluated with a retrained model or fixed embeddings.
- In Table 4, shuffling features lowers Sup-GINE AUROC on the published datasets but raises it for both angiogenesis representations. The two angiogenesis representations come from the same private dataset, so this is one dataset-specific discrepancy, not an independent replication. Report quality-control checks of the entity–profile–label mapping, profile aggregation and normalization, and assay or batch effects, and explain the improvement under feature shuffling.
- Across Sections 4.2–4.4, the paper reports retrieval performance relative to the degree-preserving null (Table 2), topology interventions (Table 3), and feature interventions (Table 4), with additional qualitative observations in Figures 3–4. However, these findings are discussed mainly as dataset- and model-specific outcomes; the paper does not establish when retrieval above the null is supported by the correct cellular features rather than by graph structure. The authors should synthesize the results across these experiments and state what evidence-based conclusion they support about when cellular profiles help recover gene–drug relationships. If the experiments only demonstrate that the effects vary across settings, the paper should limit its claim accordingly.
- In panels A and B, the gene branch shows Gene Phenotype → encoder → Gene Vectors/Embeddings, whereas the drug-branch arrows point toward Drug Phenotype and do not show the corresponding output path consistently. State the intended drug-branch computation and correct the figure to show Drug Phenotype → encoder → Drug Vectors/Embeddings, if that is the implemented architecture.