- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a graph neural network to identify perturbed gene based on expression profiles, then uses it to nominate perturbations in a cancer expression profile that's held out. Gradient attribution is used to analyse nominations.
Critical strengths
-
The focus on hypothesis generation is interesting. Also I quite like the careful experimental and computational setup. the model also explicitly separate biological causal effect from message passing.
-
The benchmarking is robust.
Critical weaknesses
-
I am not too sure about why reverse prediction is required here. If we want to match perturbation responses, why not predict each candidate's response with a forward perturbation model then rank? I think comparison to these kind of the models would make the empirical results a lot stronger.
-
The model depends on selecting valid endpoints. However the correctness of endpoints are not validated on held out data
Questions for authors
- Can you isolate the effect of classifier trivially predicting which gene are requested to decrease rather than predicting genes that are responsible for the perturbed profile?
We thank the reviewer for the interest in hypothesis generation and the comments on our setup and benchmarking.
1. Why inverse rather than forward-then-rank? (W1) We start from a desired change in the cell and ask what supports a nomination for it. The value of this analysis is to move from a candidate name to a specific account of which parts of the desired program support it and which biological relationships provide supporting or opposing contributions. Because the desired program is the inverse model's input, attribution traces each nomination to individual requested changes and the relationships that combine them. For TTK in A549, decreases in checkpoint genes support the nomination, while E2F4, a regulator of TTK transcription, opposes it (Figure 2). The same analysis could serve requests for treatment sensitivity, lineage maturation or beta-cell function (Introduction).
Forward-then-rank is a valid route, and a matched comparison would be the right test of ranking performance. Our claims concern what supports a nomination rather than ranking, so we did not train that benchmark.
2. Does the model just pick the gene requested to decrease? (Q1) We tested this directly. For the original examples it largely does: removing the own decrease moves PYGL and CCDC86 from the top 10 to beyond rank 2,000 (Table 1). We then applied this control to every candidate, requiring its score to rise under the remaining request and fall under reversal: 333 nominations pass in nine held-out lines. TTK/A549 is not among the requested changes and passes in both models that held A549 out (Section 6.1).
3. Is the endpoint supported on held-out data? (W2) The A549 markers come from HPA's validated tier, whose prognostic associations replicate across cohorts (Appendices C.1 and K list every context's tier). In PRECOG, a separate patient-cohort compendium not used to select markers, directions agree for 85.8% of 2,051 matched entries. Measured TTK knockouts reach the 98.2nd percentile of requested-direction agreement among 5,115 A549 knockouts (10 wells). Two SD is our chosen amplitude; at a smaller size anchored to measured knockout displacement, TTK keeps its request and reversal signs, while CKS2 weakens (Appendix M).
Does this address the forward-model and endpoint questions, or is there a remaining point we can clarify?