- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CrispyDrug, which jointly models genetic and chemical perturbations and learns shared perturbation representations through their effects on post-treatment expression. CrispyDrug includes two stages: Stage 1 maps both genetic and chemical perturbations to a common representation space with perturbation-type-specific encoders, optimizing post-treatment expression prediction and perturbation-type balancing concurrently. Stage 2 then fixes the learned representations and uses perturbation-type-specific decoders to reconstruct genetic or chemical perturbagen embeddings, such that it can support cross-perturbation retrieval. The paper uses the potential outcomes framework to formalize causal effects from genetic and chemical perturbations, and discusses the multiple-versions-of-treatment problem due to many-to-one treatment representations. The paper further uses natural soft intervention as its causal interpretation for the learned representations by CrispyDrug. Experimental settings include K562 Replogle genetic perturbation data and sci-Plex3 chemical perturbation data. Stage 1 is compared against applicable baselines on held-out perturbations. It also analyzes joint representation and genetic-chemical cross-retrieval for Stage 2.
Critical strengths
-
The problem is novel and practically meaningful, since CrispyDrug learns joint representations through common cellular outcomes---this is beyond just predicting genetic and chemical perturbations separately. The paper also analyses the multiple-versions-of-treatment problem, and it provides a representation-level causal interpretation via natural soft intervention.
-
The experimental results for Stage 1 are strong. Under the same held-out perturbation splits, CrispyDrug outperforms chemCPA, PRnet, and BioLORD on the chemical tasks, as well as outperforms CPA and GEARS on the genetic task. The reported figures include 5 random seeds, which is appreciated.
Critical weaknesses
- The reported experiments have not demonstrated whether the benefits are from joint training itself.
The paper regards "knowledge sharing between genetic and chemical perturbations" in a joint representation as its main motivation, but Table 2 only compares CrispyDrug with perturbation-type-specific baselines. I believe it lacks at least one important controlled comparison: train the model with the same CrispyDrug architecture with genetic data only or chemical data only, and then compare it with the jointly trained model. Currently, there are many confounders, such as architecture, ZINB objective, perturbation embeddings, or other training choices, that may cause the improvement.
- The causal interpretation and empirical validation for Stage 2 are not sufficiently convincing.
Stage 2 is claimed to recover the natural soft intervention mechanism , and the paper models it in Eq. (18) using perturbation-type-specific decoders with a fixed-variance Gaussian conditional distribution. Yet, Stage 2 is trained only through perturbagen embedding reconstruction and dose reconstruction with L2 losses, and the paper does not directly evaluate whether these decoders accurately approximate the conditional distribution defined in its theoretical analysis. Also, due to the lack of cross-perturbation retrieval baselines, the same-pathway retrieval reported in Figure 5 only shows meaningful biological evidence, but it is insufficient to establish that CrispyDrug captures stronger cross-perturbation structure than simple alternative retrieval methods.
Questions for authors
-
Can you provide a direct joint-vs-separate training ablation? For example, train the models using genetic data only and chemical data only, respectively, while keeping the architecture and losses unchanged, and compare them with the jointly trained model on the same held-out perturbations.
-
Can you explain which evidence shows that the decoders trained with can be interpreted as an approximation of natural soft intervention? Can you use, for example, random retrieval or nearest-neighbor in the original embedding space as cross-retrieval baselines?
-
The fused dataset contains genetic and chemical observations drawn from different experimental datasets. Can you clarify why the perturbation type in the fused dataset should be interpreted as randomized treatment assignment, as stated in causal assumption (iii)?
2 comments
Rebuttal [2/2]
Question 1: Can you provide a direct joint-vs-separate training ablation? For example, train the models using genetic data only and chemical data only, respectively, while keeping the architecture and losses unchanged, and compare them with the jointly trained model on the same held-out perturbations.
Response:
Sure, great idea! .
Action., We have adjusted the benchmark table such that it now includes: Crispydrug trained on only chemical perturbations → evaluated on heldout only chemical perturbations Crispydrug trained on only genetic perturbations → evaluated on heldout only genetic perturbations Crispydrug trained on joint (chemical and genetic) perturbations → evaluated on heldout only chemical perturbations Crispydrug trained on joint (chemical and genetic) perturbations → evaluated on only heldout genetic perturbations Crispydrug trained on joint (chemical and genetic) perturbations → evaluated on heldout joint (chemical and genetic) perturbations These results have helped show the importance of
Question 2: Can you explain which evidence shows that the decoders trained with can be interpreted as an approximation of natural soft intervention? Can you use, for example, random retrieval or nearest-neighbor in the original embedding space as cross-retrieval baselines?
Response:
We thank the reviewer for this suggestion!
Action. To assess whether decoder-based retrieval captures meaningful cross-modal relationships rather than chance or similarity already present in the input embedding space, we added two baselines: (1) random retrieval, which selects a candidate perturbation at random, and (2) nearest-neighbor retrieval in the original perturbation-embedding space, which ranks candidates using their embedding similarity without the learned decoder. We compare these baselines with CrispyDrug using the number of significant pathways overlapping with the drug target on all the data. We report results separately for chemical-to-genetic retrieval. CrispyDrug’s improved retrieval performance over these baselines would indicate that the learned decoder captures cross-modal structure beyond random selection and direct nearest-neighbor matching in the embedding space. We interpret this as evidence consistent with the decoder approximating a natural soft intervention—not as a direct demonstration that the decoder recovers a causal intervention. We have clarified this distinction and added the baseline results to Section 5.
Question 3: The fused dataset contains genetic and chemical observations drawn from different experimental datasets. Can you clarify why the perturbation type in the fused dataset should be interpreted as randomized treatment assignment, as stated in causal assumption (iii)?
Response:
We thank the reviewer for raising this point! Indeed, in general, combining data from separate randomized experiments can only preserve a causal interpretation when the experimental populations are comparable and potential outcomes are not systematically affected by study-specific factors [1], namely, under a cross-study exchangeability assumption. Notably, randomization within each experiment alone does not guarantee this property. Still, our work operates under this cross-study exchangeability assumption: It can be shown mathematically that our assumption (iii) is equivalent to both (a) the within-study randomization and (b) the cross-study exchangeability. Hence, all the fused datasets in our work are treated as good as randomized.
In our experiments specifically, both genetic and chemical perturbation datasets originate from controlled experiments on the same K562 cell line. This shared biological environment supports the plausibility of cross-study exchangeability and, consequently, our randomization assumption (iii) (= we implicitly assume that differences in experimental protocols and batch effects do not systematically affect the potential outcomes).
Action. We have clarified this requirement and acknowledge it as a limitation in the revised manuscript (see a new footnote in Sec. 3 and a paragraph in a new Sec. 6 in the updated PDF).
References:
- [1] Brantner, Carly Lupton, et al. "Comparison of methods that combine multiple randomized trials to estimate heterogeneous treatment effects." Statistics in medicine 43.7 (2024): 1291-1314.
Rebuttal [1/2]
We thank reviewer WKdT for their constructive feedback.
Weakness 1: The reported experiments have not demonstrated whether the benefits are from joint training itself. The paper regards "knowledge sharing between genetic and chemical perturbations" in a joint representation as its main motivation, but Table 2 only compares CrispyDrug with perturbation-type-specific baselines. I believe it lacks at least one important controlled comparison: train the model with the same CrispyDrug architecture with genetic data only or chemical data only, and then compare it with the jointly trained model. Currently, there are many confounders, such as architecture, ZINB objective, perturbation embeddings, or other training choices, that may cause the improvement.
Response:
Great suggestion, we thank the reviewer for identifying these important comparisons.
Action To address this concern, we conducted controlled experiments using the same CrispyDrug architecture and the hyperparameters optimized for training on the full joint dataset. We trained CrispyDrug separately on chemical perturbations only and genetic perturbations only, and evaluated each model held-out perturbations of the corresponding type. We also trained CrispyDrug jointly on chemical and genetic perturbations and evaluated it on (i) held-out chemical perturbations, (ii) held-out genetic perturbations, and (iii) held-out perturbations from both types. Furthermore, we also ran an experiment to compare the ZINB vs. the normal objective.
We have added these experiments and their results to Appendix E.2. The results show that despite the clear outperformance of baseline methods in the separate perturbation type tasks - in the joint task CrispyDrug leads in r2 performance. Additionally, our experiment on the ZINB objective does demonstrate that changing to a normal object improves model r2 performance in comparison to ZINB (see Figure 10).
Weakness 2: The causal interpretation and empirical validation for Stage 2 are not sufficiently convincing. Stage 2 is claimed to recover the natural soft intervention mechanism , and the paper models it in Eq. (18) using perturbation-type-specific decoders with a fixed-variance Gaussian conditional distribution. Yet, Stage 2 is trained only through perturbagen embedding reconstruction and dose reconstruction with L2 losses, and the paper does not directly evaluate whether these decoders accurately approximate the conditional distribution defined in its theoretical analysis. Also, due to the lack of cross-perturbation retrieval baselines, the same-pathway retrieval reported in Figure 5 only shows meaningful biological evidence, but it is insufficient to establish that CrispyDrug captures stronger cross-perturbation structure than simple alternative retrieval methods.
Response:
Great point! We thank the reviewer for noting that.
Action We evaluated both the conditional Gaussian decoder and the conditional affine-flow decoder on the same test set of 825 examples (containing both perturbation types). The conditional affine flow achieved a higher mean summed log-likelihood compared with the conditional Gaussian decoder (see table inline). Thus, the flow assigns higher density to the held-out targets under this evaluation. However, we found that the conditional flow decoder performed poorly on the biological task of Stage 2 (see Figure 10)
Importantly, Stage 2 of our CrispyDrug is generally decoder-agnostic: it conditions a decoder on the Stage 1 representation and trains it to reconstruct the perturbagen representation (and dose). The decoder can therefore be replaced by another model that supports the required conditional prediction, without changing the Stage 1 representation-learning stage. The Gaussian and affine-flow decoders are two examples of this flexibility.