- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RISCAL learns a directed gene graph from paired control and CRISPR perturbation profiles and uses the same model to predict perturbation responses. Each gene has its own nonlinear mechanism determined by its inferred parents. A perturbation is modeled as only partially removing the target gene's effective activity, while the target's measured expression is handled by a separate readout. Evaluation covers synthetic data with known ground-truth graphs, ten LINCS cell lines, and an experiment involving an unseen regulator.
Critical strengths
Separating functional activity from measured RNA abundance is a useful modeling idea, since incomplete loss of function does not necessarily correspond to the measured target transcript going to zero. The synthetic generator is also unusually detailed and introduces biological complications incrementally. The real-data splitting, control pairing, and batch handling appear more careful than in many perturbation-prediction evaluations.
Critical weaknesses
My main concern with the synthetic evaluation is that the later generator stages closely mirror RISCAL's modeling assumptions, including residual activity, nonlinear child-specific mechanisms, sparse directed structure, and reused controls. RISCAL therefore has a substantial specification advantage on these benchmarks. The results still show that the implementation can recover the structure it is designed for, but they tell me less about performance under misspecification. SDCD being essentially at the Random-DAG level on the 3,710-gene benchmark also makes me wonder whether this is a meaningful comparison at that scale. Several closely related joint structure-and-prediction methods discussed in the paper, including DCD-FG, GIM, and Bicycle, are not included in the empirical comparison.
On real data, I am not yet convinced what predictive benefit comes specifically from RISCAL's learned causal structure. RISCAL, GEARS, and the axial transformer operate in almost the same predictive regime, even though GEARS does not receive each test profile's paired control as an input. That makes it difficult to isolate the value of the paired-control information and learned DAG. A very simple reference such as a mean-response predictor would help. The linear-regression and MLP baselines also have macro intervention MSE more than an order of magnitude worse than RISCAL, GEARS, and the axial transformer, which is surprising enough that I would like a sanity check or explanation.
The learned residual-activity distributions on LINCS are also extremely concentrated: the global means are very similar across cell lines and the across-gene standard deviations are small. This makes it unclear how much the gene-specific part of the residual-activity model contributes on real data. Finally, the curated-interaction enrichment is statistically significant against the degree-preserving nulls in many cell lines, but the absolute overlap is still only a small number of curated edges among thousands of learned edges. I would find the evidence more convincing with the same enrichment analysis applied to alternative learned graphs.
The paper is also difficult to read. Much of the information needed to interpret the main claims is in a very long appendix, and I had to move back and forth repeatedly to understand the evaluation.
Questions for authors
-
What does a very simple mean-response baseline achieve on the LINCS prediction task?
-
On LINCS, what happens if the gene-specific residual-activity parameters are replaced by a shared scalar residual activity? Does the gene-specific parameterization materially improve prediction or graph stability?
-
What does the same CollecTRI/SIGNOR enrichment analysis give for alternative inferred graphs such as GENIE3 or a correlation-network baseline?
-
For the unseen-regulator experiment, how well does simply averaging two measured replicates predict the third? I would like a simple experimental-variability reference before interpreting agreement with the external experiment as evidence for generalization.
5 comments
Weaknesses — paragraph 2: graph utility, controls, and predictive baselines
Weaknesses — paragraph 2: graph utility, controls, and predictive baselines
Thank you for these observations. To assess the contributions of the learned graph and individual controls, we conducted additional ablations and evaluated simple predictive baselines. We also verified the implementations and results of linear regression and the MLP.
Contribution of the graph to prediction and interpretive value: we evaluated the RISCAL trained without the SCM term on PC3. The means across five fits are:
| PC3 | Full RISCAL | Without SCM |
|---|---|---|
| Edges per fit | 10,357.6 | 0 |
| Response R² | 0.425 | 0.427 |
| Macro MSE × 10³ | 7.237 | 7.216 |
| Response Pearson | 0.646 | 0.649 |
Without the SCM term, the model predicts each non-target gene's learned baseline μ, while retaining the perturbed gene's specific readout. Both variants achieve comparable predictive performance, while the full RISCAL additionally learns interpretable regulatory relationships. With the same edge budget, RISCAL recovers more curated undirected pairs than coexpression:
| Resource | Pairs per cell line, RISCAL | Pairs per cell line, coexpression |
|---|---|---|
| CollecTRI | 26.3 | 11.7 |
| SIGNOR | 16.2 | 11.3 |
Furthermore, we identified 54 candidate relationships present in all five fits across all ten cell lines and absent from both resources. Our literature review found published biological support for 33 of them.
These results illustrate the graph's interpretive utility in recovering documented relationships and prioritizing candidates for further investigation. Additional biological evidence is presented in our response to Reviewer 2, Weaknesses 3 and Question 4.
See Appendix G.1, Table 38, and Section 7.3, Table 2.
Contribution of the individual control: prompted by the reviewer's comment, we replaced the individual controls with the same mean training control profile and retrained the model. Aggregate performance remains practically unchanged: on PC3, response R² is 0.4252 with the individual control and 0.4253 with the mean control. The recovered graphs are exactly identical with individual and mean controls across all five fits, both on real PC3 data and on the synthetic benchmark at PC3 scale.
While aggregate performance remains comparable, individual controls allow sample-specific predictions. Using the same mean control produces identical predictions for samples of a given intervention, whereas individual controls introduce within-intervention variation in the full predicted profile, increasing from 0 to 0.001324 ± 0.000052.
Thus, individual controls enable sample-specific variation, although this does not translate into improved aggregate performance in the evaluated setting.
See Appendix G.7, Table 43.
Absolute performance against simple baselines: we evaluated three simple baselines, computed using training data and evaluated on the same splits. RISCAL outperforms all three on average across the ten cell lines; the results by cell line are included in the paper.
| Method, mean across ten cell lines | Response R² | Macro MSE × 10³ |
|---|---|---|
| RISCAL | 0.428 | 7.473 |
| Mean treated profile | 0.272 | 9.362 |
| Control + mean response | −0.332 | 17.145 |
| Copy control | −0.005 | 13.129 |
See Section 7.2 and Appendix E.2, Tables 34–35.
Checks of linear regression and the MLP: we reviewed the code, configurations, and saved predictions for PC3. The models use the same splits; recalculating macro MSE reproduces the reported values, and loading the MLP checkpoint reproduces its predictions. We found no implementation or evaluation errors.
The MLP systematically underpredicts across all five fits. In fold 0, the mean bias per gene accounts for 97.5% of its MSE. The regression model, fitted without regularization, has large coefficients and considerably higher validation errors than training errors.
The numbers are therefore reproducible for the training recipes used, but we acknowledge that these baselines could be improved. We do not interpret their results as a general limitation of linear regression or MLPs, and we base the predictive comparison on the strong baselines and simple controls described above.
Weaknesses — paragraph 3: residual activity and biological enrichment
Weaknesses — paragraph 3: residual activity and biological enrichment
We thank the reviewer for these observations and separately address the contribution of gene-specific residual activity and the comparison of biological enrichment with other methods.
Contribution and interpretation of gene-specific residual activity: to explore this limitation, we generated a new synthetic dataset with more heterogeneous ρ: [0.05, 0.95] versus [0.30, 0.50] in the PC3-scale benchmark. We also increased the learning rate from 1e-5 to 3e-4. The Spearman correlation with true ρ is 0.425 for genes with children and 0.557 for genes with at least three children. These results support the parameter's ability to recover differences between genes.
To examine its consistency with real data, we froze the RISCAL models for all ten cell lines and re-estimated only residual activity. Its gene-specific component η has an adjusted Spearman correlation of 0.359 with the change in RNA expression of the perturbed gene, with positive correlations and p<0.05 in all ten cell lines. Although we distinguish RNA expression from functional activity in the paper, this agreement provides experimental support for the estimated differences.
See Appendix G.6 and Appendix D.6, Table 32.
We also evaluated a variant with a learned global ρ in response to the reviewer's comment. The means across five fits are:
| Evaluation | Gene-specific ρ | Global ρ |
|---|---|---|
| Real PC3: response R² | 0.425 | 0.425 |
| Real PC3: macro MSE | 0.007 | 0.007 |
| PC3-scale synthetic: response R² | 0.495 | 0.495 |
| PC3-scale synthetic: macro MSE | 0.007 | 0.007 |
| PC3-scale synthetic: graph F1 | 0.649 | 0.653 |
| PC3-scale synthetic: graph AUPRC | 0.633 | 0.637 |
Both variants achieve comparable predictive and structural performance. The contribution of gene-specific ρ is its ability to represent and investigate differences between perturbations, supported by the recovery and biological association analyses above.
One example is the comparison with gene essentiality in DepMap. This dataset measures how much cell survival depends on each gene. We compared this measure with η from the RISCAL models and obtained a Pearson correlation of 0.241, with positive correlations and p<0.05 in all ten cell lines. This association is consistent with essential-gene perturbations producing stronger downstream transcriptional responses, represented through lower residual activity. Although essentiality and activity reduction are distinct concepts, this relationship further supports the parameter's interpretive potential.
See Appendix G.5, Table 42, and Appendix D.6, Table 31.
Biological enrichment compared with other graphs: we agree that enrichment should be interpreted alongside absolute counts and comparisons with other methods. We extended the same analysis to InterSort, SDCD, CASCADE, and KO→response. For directed relationships with native support selection, the means across the ten cell lines are:
| Method | Edges | CollecTRI / SIGNOR pairs | CollecTRI / SIGNOR enrichment |
|---|---|---|---|
| RISCAL | 8,592.4 | 9.6 / 10.1 | 3.376× / 3.587× |
| InterSort | 4,682,003.8 | 1,292.0 / 1,513.6 | 1.008× / 1.012× |
| CASCADE | 120,786.7 | 573.9 / 346.5 | 1.054× / 1.057× |
| KO→response | 8,592.4 | 7.5 / 9.0 | 2.290× / 1.832× |
| SDCD | 17.1 | 0 / 0 | Undefined |
RISCAL achieves the highest mean enrichment in both resources, while selecting considerably smaller graphs than InterSort and CASCADE. It also achieves higher enrichment than KO→response with the same edge budget. These results support the biological relevance of RISCAL's learned relationships compared with alternative inferred graphs.
See Section 7.3, Table 2, and Appendix D.3.2, Table 17.
Weaknesses — paragraph 4: readability
We thank the reviewer for this feedback. Following this comment and Reviewer 1's suggestion, we moved the compact training and prediction algorithm to the main text (Section 4.6, Algorithm 1) and shortened the synthetic-generator presentation (Appendix B). The algorithm summarizes the full procedure and points directly to the defining equations and parameter and hyperparameter tables (Tables 4 and 5). These changes aim to reduce back-and-forth between the main text and the appendix.
Weaknesses — paragraph 1: synthetic evaluation and related causal methods
We sincerely thank the reviewer for the careful assessment and constructive feedback. Their comments have prompted additional experiments and important clarifications, substantially improving the scientific rigor of our evaluation and the support for our claims. Below, we address each concern and question in detail. All changes are highlighted in blue in the revised manuscript.
Weaknesses — paragraph 1: synthetic evaluation and related causal methods
Thank you for highlighting these concerns. To assess RISCAL under alternative data generating mechanisms and against closely related causal learning methods, we expanded the synthetic evaluation and clarified the comparisons
Recovery with different generating mechanisms: we generated two new families of synthetic data at PC3 scale, retaining the same DAG for each pair of seeds. RISCAL retains its architecture, tanh mechanisms, and official training recipe. We did not adapt the model to the new families or use the ground-truth DAG to select checkpoints.
The first family uses additive softplus mechanisms. The second introduces multiplicative interactions between parents, so that the influence of a gene depends on the state of other parents:
RISCAL maintains strong structural recovery under additive softplus mechanisms and recovers a more conservative subset of relationships under multiplicative interactions. In the latter setting, it achieves precision of 0.829 ± 0.042 and recall of 0.455 ± 0.039, selecting approximately 7,566 edges out of 13,732 true edges. These results extend the evaluation beyond the original generator, demonstrating recovery under different mechanisms while characterizing its reduced recall with multiplicative interactions.
See Section 7.1 and Appendix B.4, Tables 11–12.
Synthetic benchmark at PC3 scale: to provide more informative comparisons at this scale, we expanded the evaluation with InterSort and CASCADE, reporting native F1, oracle F1, and threshold-independent AUPRC:
InterSort and CASCADE recover structural information clearly above chance. Oracle threshold selection narrows the F1 gap, but RISCAL retains the highest F1 and AUPRC.
See Section 7.1, Table 1.
Comparison with closely related methods: given the limited rebuttal period, we prioritized CASCADE as a particularly close reference: it jointly learns a gene-level DAG, nonlinear mechanisms, and intervention coefficients to model perturbation responses. We added its evaluation on the two synthetic benchmarks:
RISCAL achieves higher F1 and AUPRC at both scales. We also added its enrichment evaluation across all ten real cell lines, where RISCAL continues to show higher enrichment against CollecTRI and SIGNOR. We have also revised the related work to give DCD-FG, GIM, and Bicycle appropriate recognition and explain their contributions.
See Sections 2 and 7.1, and Tables 1–2.