- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This manuscript studies the stability of model rankings under molecular extrapolation. Specifically, the authors introduce a chemical-support-resolved evaluation architecture that measures test-drug similarity to the training set and examines both predictive performance and model ranking stability across multiple molecular association tasks. The findings show that models favored by conventional benchmarks may not remain the best choices for chemically distant compounds.
Critical strengths
-
The work examines the reliability of model selection under molecular extrapolation, rather than focusing solely on predictive performance.
-
The paper shows that conventional cold-start splits do not adequately characterize the chemical support of test compounds, and distinguishes predictive performance from model-selection stability.
Critical weaknesses
-
The main ranking comparison involves Random Pair and extreme low-support evaluation, which differ in both training data and splitting protocols. Therefore, the observed ranking changes cannot be attributed solely to decreasing chemical support.
-
In MDAD, the extreme low-support cohort contains 19.2 drugs on average, but only 8.3 are used in the final ranking analysis. The reasons for excluding these drugs and the potential impact on model rankings are not sufficiently explained.
-
Replacing Morgan fingerprints with MACCS substantially increases ranking correlations on MDAD and ChEMBL37. This suggests that the main findings may depend strongly on the molecular representation used to define chemical support.
Questions for authors
-
The paper primarily demonstrates ranking instability by comparing Random Pair with the extreme low-support regime. However, these settings use different training data and splitting protocols. Although Section 4.3 includes within-Drug-Cold comparisons, why is the cross-protocol comparison still presented as the main result? To what extent can the observed ranking changes be attributed to chemical support itself?
-
Table 14 reports an average of 19.2 drugs in the extreme low-support cohort of MDAD, whereas Table 25 uses only 8.3 drugs for the final eight-model ranking analysis. Which models fail to produce valid predictions for the excluded drugs?
-
Ranking instability becomes substantially weaker when evaluating 6 representative DTI architectures. For example, on Yamanishi 08, Spearman correlation increases from -0.302 to 0.543, and the rankings of all 6 models based on mean AP remain unchanged in Table 8. Does this suggest that the observed ranking instability depends heavily on the candidate model set, limiting the generality of the main conclusions?
-
The positive prevalence and number of drug associations differ across chemical-support groups. The authors perform covariate adjustment for predictive performance and matching analysis for MDAD, but why is a similar analysis not conducted for ranking stability on Yamanishi 08?
-
The abstract is too long, and the motivation is not clear. The authors should clearly state the limitations of existing work, what they propose, what challenges they address, and what they contribute to the community. The introduction also needs better logical organization and a clearer flow.
9 comments
Response to Reviewer Nk2E (Part 2 / 3)
Response to W3: Representation Dependence
A3: We thank the reviewer for this observation. We acknowledge that chemical support is representation dependent (Section 5.2; Appendix A.15). Morgan-r2 Tanimoto provides a reproducible operational definition, rather than a representation-invariant measure of molecular novelty.
Cross-representation evidence. Existing analyses show strong agreement between Morgan-r2 and Morgan-r3 (ρ = 0.978–0.987), whereas MACCS-166 attenuates ranking transitions on MDAD and ChEMBL37 (ρ = 0.667/0.556; reversals = 11.9%). We further evaluated frozen ChemBERTa-77M and MoLFormer-XL embeddings using cosine similarity, matching low-support cohort sizes while keeping partitions, models, predictions, and metrics fixed.
Table R3.3. Ranking changes across chemical-support representations.
| Dataset | Morgan-r2 | ChemBERTa | MoLFormer-XL |
|---|---|---|---|
| Yamanishi 08 | 48.8% / 2/3 | 33.3% / 2/3 | 41.7% / 3/3 |
| MDAD | 34.5% / 3/3 | 19.0% / 2/3 | 20.2% / 2/3 |
| ChEMBL37 | 34.5% / 2/3 | 15.5% / 3/3 | 21.4% / 2/3 |
Meaningful pairwise reversals / partitions with Top-1 changes; reversals require |ΔAP| > 0.005 in both regimes.
Interpretation and scope. Although pretrained representations attenuate ranking changes, Top-1 changes persist in at least 2/3 partitions across all datasets. Their low-support cohorts also show limited overlap with Morgan-r2 (25.4%, 12.0%, and 23.6%, respectively). These findings confirm that both cohort composition and model-selection outcomes depend on the support representation. Our framework therefore evaluates ranking reliability under explicitly specified support definitions, without assuming cross-representation invariance.
Response to Q3: Candidate-Model-Set Dependence
A4: We thank the reviewer for raising this point. We clarify that ranking instability is conditional on the candidate model set and chemical-support regime, rather than a universal phenomenon (Introduction and 4.5).
Evidence across model families. As shown in Figure 4 and Appendix A.5 (Tables 8–10), six representative DTI architectures exhibit weaker but non-zero ranking changes:
Table R3.4. Ranking stability across candidate model sets.
| Dataset | Candidate models | Mean Spearman ρ | Meaningful reversals | Top-1 agreement |
|---|---|---|---|---|
| Yamanishi 08 | Six DTI architectures | 0.543 | 13.3% | 1/3 |
| ChEMBL37 | Six DTI architectures | 0.467 | 13.3% | 2/3 |
| MDAD | Three task-specific models | 0.667 | 11.1% | 1/3 |
Results are averaged across three independent partitions.
Importantly, aggregate ranking agreement does not imply run-level stability. On Yamanishi 08, the run-averaged model ordering remains unchanged, yet the Top-1 model changes in 2/3 individual runs, with 13.3% meaningful pairwise reversals.
These results demonstrate that ranking stability depends on both the candidate model set and chemical-support regime. Our framework quantifies this dependence and identifies when conventional benchmark rankings remain informative or warrant reassessment.
Response to Reviewer Nk2E (Part 3 / 3)
Response to Q4: Covariate-Matched Ranking Stability on Yamanishi 08
A5: We thank the reviewer for this suggestion. To assess whether ranking changes persist after accounting for measured covariates, we conducted a covariate-matched analysis on Yamanishi 08, using four variables: positive count, positive prevalence, molecular weight, and heavy-atom count.
Matching protocol. Within each of three fixed Drug-Cold runs, we performed one-to-one matching between low-support and moderate-support drugs (), using the original predictions of all eight models without retraining. All low-support drugs were successfully matched (100% coverage).
Table R3.5. Ranking stability after covariate matching.
| Low-support cohort | Mean drugs/run | Coverage | Spearman ρ (Mean ± SD) | Meaningful reversals | Top-1 agreement |
|---|---|---|---|---|---|
| 5.0 | 100% | −0.468 ± 0.213 | 45.2% | 0/3 | |
| Bottom 10% | 16.0 | 100% | 0.000 ± 0.536 | 42.9% | 0/3 |
Rankings compare matched moderate- and low-support cohorts within each run. Meaningful reversals require in both regimes. Top-1 agreement denotes the number of runs selecting the same best-performing model in both cohorts.
Results and implications. Both low-support definitions retain substantial ranking changes after matching, with 45.2% and 42.9% meaningful pairwise reversals, respectively, and no Top-1 agreement across the three runs. These findings indicate that matching on the four measured covariates does not eliminate ranking transitions. The matching protocol and results are provided in Appendix A.13 (Table 27).
Response to Q5: Abstract and Introduction
A6: We thank the reviewer for this suggestion. We will streamline the Abstract to highlight the motivation, our chemical-support-resolved evaluation framework, and the key finding that ranking instability is conditional rather than universal.
Introduction. We will reorganize it around motivation → limitations → research question → evaluation framework → contributions, clarifying how conventional Drug-Cold and Scaffold-Cold splits may conceal heterogeneous chemical support and distinguishing our focus on model-ranking reliability from prior evaluations of individual-model performance.
We will incorporate these additional analyses and clarifications into the revised manuscript while preserving the original results and conclusions. These changes will be included in the camera-ready version.
We are happy to see from your comments that you have an overall positive view of our work, and we respectfully ask if you'd be comfortable to adjust your ratings, based on all the new experiments and justification to your questions. Thank you once again for your time and effort. We look forward to incorporating these insightful discussions into the final camera-ready version.
Response to Reviewer Nk2E (Part 1 / 3)
We thank the reviewer for their detailed, constructive feedback and their recognition of the importance of model-selection reliability under molecular extrapolation, as well as the value of distinguishing predictive performance from ranking stability beyond conventional cold-start evaluations.
We have carefully addressed your concerns through additional analyses. We hope these efforts help strengthen your confidence in our findings and respectfully invite you to reconsider your rating. Below, we provide detailed responses to each comment.
Response to W1 and Q1: Protocol Attribution
A1: We thank the reviewer for this observation. The Random Pair-to-extreme comparison is intended as a benchmark-to-deployment transfer diagnostic, assessing whether conventional rankings remain informative for chemically under-supported candidates, rather than estimating the causal effect of chemical support.
Within-protocol validation. To control for split-protocol differences, we compare moderate (Smax∈[0.4,0.6)) and extreme (Smax<0.2) support cohorts within identical Drug-Cold partitions, using the same trained models and evaluation protocol (Section 4.3; Appendix A.9, Table 21).
Table R3.1. Within-Drug-Cold ranking stability.
Cohort-size control. We further compare extreme-support cohorts against 1,000 size-matched moderate-support samples per run. Extreme-support ranking agreement with Random Pair is lower on Yamanishi 08 (−0.302 vs. 0.604), MDAD (0.183 vs. 0.540), and ChEMBL37 (0.087 vs. 0.732). Both rank-correlation and meaningful-reversal differences remain significant after Holm correction (p ≤ 0.011 and p ≤ 0.013, respectively; Figure 3b; Appendix A.12, Table 26).
Together, these analyses show that ranking changes persist within fixed evaluation protocols and cannot be explained by cohort size alone, supporting the practical need to reassess benchmark rankings for chemically under-supported candidates.
Response to W2 and Q2: MDAD Cohort Accounting and Prediction Coverage
A2: We thank the reviewer for identifying this discrepancy. We audited the split manifests, chemical-support records, and model predictions to identify its source.
Cohort accounting. The MDAD extreme-support Drug-Cold cohorts contain 8, 9, and 8 drugs across three runs (mean 8.3). All eight models (M1–M8) produce valid predictions for all 169 Drug-Cold test drugs per run, with no model-specific exclusions. The discrepancy instead arises from Appendix A.7 (Table 14), where drug counts were inadvertently averaged across Drug-Cold and Scaffold-Cold protocols.
Table R3.2. MDAD support-bin drug counts.
Mean counts across three runs. Originally reported values equal the averages of the two protocols (up to rounding).
Impact and correction. This is a reporting inconsistency, not missing predictions. The primary analyses already use the correct Drug-Cold cohorts (8/9/8 drugs); therefore, all reported rankings, correlations, and reversal rates remain unchanged. Our broader audit found no similar inconsistencies. We will correct Table 14, clarify the aggregation error, and include the prediction-coverage audit in the camera-ready version.