- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes EviDR, a framework for predicting drug responses that generalizes from cancer cell lines to individual patients. The method leverages a shared Reactome pathway space, cross-fitted drug–pathway evidence, and drug-conditional interaction representations to enable patient-specific drug response prediction. In experiments using five-fold cross-validation with patients completely held out, EviDR achieved a Macro Spearman coefficient of 0.329, outperforming existing comparison methods.
Critical strengths
-
The research question has practical significance. This paper focuses on cross-domain drug response prediction from cell lines to patients and emphasizes the ranking of responses among different patients to the same drug, which has clearer practical value compared to merely optimizing global correlation metrics.
-
The evaluation protocol is rigorous. The authors employed patient-level partitioning to prevent cross-contamination of patient information between training and testing, and used the drug-specific Macro Spearman correlation as the primary metric. Additionally, they demonstrated the importance of the evaluation protocol through randomized paired experiments.
-
The experimental analysis is thorough. The paper provides multiple baseline comparisons, quintile results, Bootstrap uncertainty analysis, source-domain supervised experiments, and Evidence ablation; overall, the experimental design is comprehensive.
Critical weaknesses
W1. The actual contribution of the core Evidence mechanism is relatively limited. The main innovation of this paper is the use of reliability-weighted, signed drug–pathway Evidence to guide patient-specific interactions. However, as shown in Appendix Table 10, after removing reliability, correlation, or sign information, the Macro Spearman coefficient remains at 0.327–0.328, which is very close to the 0.329 of the full model. In particular, even after randomly shuffling the pathway mappings of the Evidence, the performance still reaches 0.328. This leads me to question whether the biological associations contained in Evidence truly contribute to a meaningful predictive gain.
W2. The performance advantage over existing methods is not very pronounced; could this be due to variations in random seed fluctuations?
EviDR’s Macro Spearman coefficient is 0.329, while TransDRP and MAC-CDR are 0.314 and 0.313, respectively, representing a relatively limited improvement. EviDR outperformed MAC-CDR in only three out of five test setups, and the 95% confidence interval for the performance difference obtained via patient clustering bootstrap is [−0.012, 0.039], which includes zero. Therefore, the current evidence is insufficient to demonstrate that this method has a consistent performance advantage across different patient populations.
W3. The contribution of cross-domain transfer to the performance gains has not yet been fully distinguished from that of the Evidence mechanism.
Appendix Table 10 shows that training using only the target domain alone achieved 0.319; adding PRISM Evidence without joint training on the source domain only improved this to 0.321, while the full model reached 0.329. This suggests that the performance improvement may stem more from the additional source-domain supervision than from the proposed Evidence interaction mechanism. Furthermore, the source-domain data scale experiment in Figure 3 was conducted on only one test fold, which is insufficient to demonstrate the stability of this pattern across patient subdivisions.
Questions for authors
1.Why does the model’s performance show almost no decline after randomly shuffling the mappings between Evidence and pathways? Can the independent contributions of its biological structure and symbolic information to the predictions be further demonstrated?
2.Could you include additional experiments with multiple random seeds and paired statistical analyses against the strongest baseline to further validate the stability of the performance improvement at the patient level?
3.Could you conduct large-scale experiments on source-domain supervision across all five test folds and include a supervised matching baseline that uses the same model capacity but does not incorporate Evidence?
If the author addresses the concerns I raised, I will be willing to update my score.
6 comments
Response to Reviewer xG7y (Part 2/3): 3-Seed x 5-Fold Stability & Paired Statistical Analyses (W2, Q2)
Comment 2 (W2 & Q2): Could you include additional experiments with multiple random seeds and paired statistical analyses against the strongest baseline to further validate the stability of the performance improvement at the patient level?
Response: We thank the reviewer for this suggestion. We performed paired statistical comparisons across the held-out test folds/drugs and evaluated 3 predetermined random seeds (2026, 2027, and 2028) across all 5 outer folds (15 runs per method; Appendix A.4.4, Table 12) against both the overall runner-up TransDRP (0.3139) and MAC-CDR (0.3126):
- Paired Statistical Comparisons Against the Strongest Baselines: In Table 2 and Appendix Table 8, the marginal fold-to-fold standard deviation (~0.04) largely reflects shared cohort variation across the 5 patient partitions (for instance, Fold 5 yields the highest Macro Spearman across all 8 models in Table 8). Evaluating paired differences on the exact same held-out test folds accounts for this shared inter-fold variation:
- EviDR vs. TransDRP (2nd best overall): EviDR outperforms TransDRP in all 5 out of 5 outer folds (+0.013, +0.003, +0.023, +0.014, +0.023), achieving a paired mean gain of +0.0152 (two-sided paired t-test ). EviDR likewise wins 5/5 folds against CaDRReS, CODE-AE, WISER, Drug2TME, DeepSADR, LinClass-7, Elastic Net, and all BeatAML-only per-drug baselines, and 4/5 folds against AITL.
- EviDR vs. MAC-CDR (3rd best overall): MAC-CDR scores higher in Folds 4–5 (0.345 and 0.404) but drops sharply in Folds 1–3 (e.g., 0.241 vs. EviDR's 0.293 in Fold 3, yielding the largest cross-fold SD of 0.0631). Consequently, while the pooled patient-clustered bootstrap 95% CI (+0.013, [-0.012, +0.039]) includes zero due to this cross-fold variance, EviDR achieves a +0.0165 mean fold gain, outperforms MAC-CDR on 60.6% of the 94 evaluable drugs (paired drug-level bootstrap 95% CI: [+0.002, +0.023]), and ranks 1st across mean NDCG@5/10/20%.
- Multi-Seed Robustness Across 15 Runs (3 Seeds x 5 Folds): As shown in the table below, EviDR achieves both the highest overall Macro Spearman (0.3214) and the lowest cross-seed standard deviation (0.0073), outperforming TransDRP on every seed and leading MAC-CDR by +0.0125 overall.
| Method | Seed 2026 (5-Fold) | Seed 2027 (5-Fold) | Seed 2028 (5-Fold) | 3-Seed Mean +/- Seed SD (15 Runs) | Fold Wins (Seed 2026) |
|---|---|---|---|---|---|
| TransDRP [AAAI 2025] | 0.3139 +/- 0.0373 | 0.3051 +/- 0.0515 | 0.2959 +/- 0.0483 | 0.3050 +/- 0.0090 | 0 / 5 vs. EviDR () |
| MAC-CDR [PR 2026] | 0.3126 +/- 0.0631 | 0.3164 +/- 0.0479 | 0.2976 +/- 0.0571 | 0.3089 +/- 0.0099 | 2 / 5 vs. EviDR (60.6% drugs win) |
| EviDR (Ours) | 0.3291 +/- 0.0400 | 0.3146 +/- 0.0411 | 0.3205 +/- 0.0458 | 0.3214 +/- 0.0073 | 5/5 vs. TransDRP; 3/5 vs. MAC-CDR |
Response to Reviewer xG7y (Part 3/3): 5-Fold Source Scaling, Capacity-Matched Baseline & Clarity (W3, Q3)
Comment 3 (W3 & Q3): Could you conduct large-scale experiments on source-domain supervision across all five test folds and include a supervised matching baseline that uses the same model capacity but does not incorporate Evidence?
Response: We have completed both experiments across all five outer test folds (Appendix A.7.1, Table 15 and Figure 6):
-
Five-Fold Source-Domain Supervision Scaling: Holding target labels, Evidence, architecture, and validation criteria fixed, we varied the fraction of PRISM observations in the direct source loss across all 5 folds. Full (100%) PRISM supervision achieves the best Macro Spearman (0.3291), best Macro Pearson (0.3200), and lowest five-fold RMSE (46.422, reduced from 46.878 at 0% direct PRISM loss; Figure 6b). Intermediate fractions (50%–75%) exhibit a mild non-monotonic transition due to source–target gradient competition under partial source coverage: at 25%, optimization remains anchored by BeatAML target supervision, whereas at 50%–75% the source loss weight is large enough to steer shared representations while subsampling still fragments cross-lineage diversity and valid within-batch drug pairs (, Eq. 14), a partial-coverage effect that is resolved at 100% source supervision.
-
Capacity-Matched Supervised Baseline Without Pathway Gate & Evidence (766,988 vs. 766,989 Parameters): We constructed a Dual-Tower Concatenation MLP () that removes the pathway gate and Evidence interaction branch while widening the Fusion MLP to match EviDR within 1 parameter (766,988 vs. 766,989), trained under identical 100% PRISM + BeatAML objectives. Together, Table 15 supports three distinct controlled comparisons:
- Direct Source-Supervision Gain (Fixed EviDR + Evidence): Increasing direct PRISM supervision from 0% to 100% lifts Macro Spearman from 0.3214 to 0.3291 (+0.0077) and lowers five-fold RMSE from 46.878 to 46.422.
- Pathway-Gating & Evidence Interaction Gain (Matched Capacity & 100% Supervision): Compared against the 766,988-parameter Concatenation MLP (which still retains EviDR's Reactome pathway space, biological regularizers, and drug-balanced pairwise ranking loss, already beating all 13 external baselines), EviDR's joint pathway-gating and Evidence interaction design improves Macro Spearman from 0.3239 to 0.3291 (+0.0052), winning in 4 out of 5 outer folds (Fold 1: +0.0296, Fold 3: +0.0085, Fold 4: +0.0058, Fold 5: +0.0033) and improving Macro Pearson (0.3200 vs. 0.3160).
- Overall Transfer Gain: Relative to Target-Only EviDR (0.3194), Full EviDR achieves a +0.0097 Macro Spearman gain, confirming that joint pathway gating and Evidence modulation consistently improve within-drug patient stratification alongside pathway-level biological traceability.
| Model / Supervision Setting (All 5 Folds) | Parameters | Macro Spearman (Up) | Macro Pearson (Up) | Outer-Fold Wins |
|---|---|---|---|---|
| Target-Only EviDR (BeatAML only, no PRISM) | 766,987 | 0.3194 +/- 0.0370 | 0.3118 +/- 0.0432 | — |
| EviDR Source-Scaling: 0% Direct PRISM Loss (+Evidence) | 766,989 | 0.3214 +/- 0.0411 | 0.3125 +/- 0.0445 | — |
| EviDR Source-Scaling: 25% Direct PRISM Supervision | 766,989 | 0.3231 +/- 0.0337 | 0.3156 +/- 0.0437 | — |
| EviDR Source-Scaling: 50% Direct PRISM Supervision | 766,989 | 0.3186 +/- 0.0394 | 0.3090 +/- 0.0426 | — |
| EviDR Source-Scaling: 75% Direct PRISM Supervision | 766,989 | 0.3151 +/- 0.0399 | 0.3067 +/- 0.0497 | — |
| Same-Capacity Concatenation MLP (100% PRISM, w/o Gate/Evidence) | 766,988 | 0.3239 +/- 0.0361 | 0.3160 +/- 0.0371 | 1 / 5 vs. EviDR |
| Full EviDR (100% PRISM + Pathway Gate & Evidence) | 766,989 | 0.3291 +/- 0.0400 | 0.3200 +/- 0.0446 | 4 / 5 vs. MLP |
Manuscript Readability Revisions: To enhance clarity, we structured Sections 3.1–3.3 into an explicit three-step walkthrough detailing how cross-fitted Evidence () modulates patient pathway states (), added a master notation table in Appendix A.1 (Table 4), clarified the fixed-Evidence protocol in Figure 3, and integrated the structured permutation nulls (Figure 4) as well as the 5-fold scaling curves and capacity-matched comparisons (Appendix A.7.1, Table 15 and Figure 6). We respectfully hope these findings merit an upward revision of your score.
Response to Reviewer xG7y (Part 1/3): Evidence Permutation Nulls & Constrained Readout Probes (W1, Q1)
Thank you very much for recognizing the clinical value of our within-drug patient ranking setup, our strict patient-held-out evaluation protocol, and our experimental design, as well as for your willingness to update your score once your questions are addressed. We present our responses in three parts covering: (1) Evidence structure and sign isolation, (2) 3-seed stability and paired statistical comparisons, and (3) 5-fold source-supervision scaling and capacity-matched comparisons (marked in blue in the revised PDF).
Comment 1 (W1 & Q1): Why does the model's performance show almost no decline after randomly shuffling the mappings between Evidence and pathways [in Appendix Table 14b, previously Table 10b]? Can the independent contributions of its biological structure and symbolic information to the predictions be further demonstrated?
Response: We thank the reviewer for this insightful question. We clarify why retraining from scratch compensates for degraded Evidence in Appendix Table 14b (revised Appendix A.7), and provide three experiments isolating the contributions of Evidence's pathway alignment and directional sign:
Why Retraining from Scratch Absorbs Perturbed Evidence: Note that shuffling drug-specific Evidence pathways while keeping patient pathway states x fixed does not commute with elementwise modulation (x⊙(Pdsd)=Pd(x⊙sd)), so dense layers cannot simply invert the permutation. Instead, when retrained end-to-end from scratch over PRISM and BeatAML, the full architecture adapts around degraded Evidence via complementary prior and bypass branches: in Eq. (9), the gate gd=σ(Wgzd+bg+pd+ηrrd) combines curated Reactome drug-target priors (pd) and learned molecular gate projections (Wgzd) alongside empirical Evidence (rd), while Eq. (12) retains the unmodulated sample and drug representations (zx∥zd), allowing trainable interaction and fusion layers to rely on alternative signals and down-weight noisy Evidence channels.
Isolating Biological Pathway Alignment and Sign Without Retraining Compensation: To test Evidence information directly without bypass fallback or end-to-end retraining adaptation, we conducted three complementary evaluations (Appendix A.6, Table 13 & Figure 4):