acceptodds
Under review as a conference paper at ICLR 2027

Chemical-Support-Resolved Evaluation of Model-Selection Stability under Molecular Extrapolation

Abstract

Molecular benchmarks are commonly used not only to estimate predictive performance, but also to select which model should be deployed. This practice implicitly assumes that the model ordering observed on a benchmark remains informative when target compounds move beyond the chemical support of the training data. We examine this assumption through chemical-support-resolved evaluation, which characterizes each held-out drug by its maximum similarity to the training collection and evaluates both predictive performance and model-order stability as support decreases. Across RNA–drug, drug–protein, and drug–microbe association tasks, together with a separate ChEMBL37 direct-human drug–protein evaluation, we find that conventional cold-start partitions do not define a uniform extrapolation regime: even under Scaffold-Cold, 8.2–15.3% of test drugs retain relatively close training analogues. Predictive degradation and model-order stability are distinct. In the extreme-support regime (), the controlled model set exhibits substantial ranking transitions in Yamanishi_08, MDAD, and ChEMBL37; in Yamanishi_08, Random Pair and low-support rankings yield Spearman and 48.8% near-tie-adjusted pairwise reversals. Equal-size moderate-support controls show significantly greater ranking agreement, indicating that the transition is not explained by sparse cohort size alone. These results show that a benchmark leaderboard should be interpreted relative to the chemical-support regime in which model selection will be applied, rather than assumed to remain informative across molecular extrapolation.

Primary area:
datasets and benchmarks
Keywords:
out-of-distribution generalization, model selection, model evaluation, distribution shift, molecular machine learning, chemical similarity, extrapolation, molecular association prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Nk2EOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript studies the stability of model rankings under molecular extrapolation. Specifically, the authors introduce a chemical-support-resolved evaluation architecture that measures test-drug similarity to the training set and examines both predictive performance and model ranking stability across multiple molecular association tasks. The findings show that models favored by conventional benchmarks may not remain the best choices for chemically distant compounds.

Critical strengths
  1. The work examines the reliability of model selection under molecular extrapolation, rather than focusing solely on predictive performance.

  2. The paper shows that conventional cold-start splits do not adequately characterize the chemical support of test compounds, and distinguishes predictive performance from model-selection stability.

Critical weaknesses
  1. The main ranking comparison involves Random Pair and extreme low-support evaluation, which differ in both training data and splitting protocols. Therefore, the observed ranking changes cannot be attributed solely to decreasing chemical support.

  2. In MDAD, the extreme low-support cohort contains 19.2 drugs on average, but only 8.3 are used in the final ranking analysis. The reasons for excluding these drugs and the potential impact on model rankings are not sufficiently explained.

  3. Replacing Morgan fingerprints with MACCS substantially increases ranking correlations on MDAD and ChEMBL37. This suggests that the main findings may depend strongly on the molecular representation used to define chemical support.

Questions for authors
  1. The paper primarily demonstrates ranking instability by comparing Random Pair with the extreme low-support regime. However, these settings use different training data and splitting protocols. Although Section 4.3 includes within-Drug-Cold comparisons, why is the cross-protocol comparison still presented as the main result? To what extent can the observed ranking changes be attributed to chemical support itself?

  2. Table 14 reports an average of 19.2 drugs in the extreme low-support cohort of MDAD, whereas Table 25 uses only 8.3 drugs for the final eight-model ranking analysis. Which models fail to produce valid predictions for the excluded drugs?

  3. Ranking instability becomes substantially weaker when evaluating 6 representative DTI architectures. For example, on Yamanishi 08, Spearman correlation increases from -0.302 to 0.543, and the rankings of all 6 models based on mean AP remain unchanged in Table 8. Does this suggest that the observed ranking instability depends heavily on the candidate model set, limiting the generality of the main conclusions?

  4. The positive prevalence and number of drug associations differ across chemical-support groups. The authors perform covariate adjustment for predictive performance and matching analysis for MDAD, but why is a similar analysis not conducted for ranking stability on Yamanishi 08?

  5. The abstract is too long, and the motivation is not clear. The authors should clearly state the limitations of existing work, what they propose, what challenges they address, and what they contribute to the community. The introduction also needs better logical organization and a clearer flow.

9 comments
AuthorsOpenReview ↗

Response to Reviewer Nk2E (Part 1 / 3)

We thank the reviewer for their detailed, constructive feedback and their recognition of the importance of model-selection reliability under molecular extrapolation, as well as the value of distinguishing predictive performance from ranking stability beyond conventional cold-start evaluations.

We have carefully addressed your concerns through additional analyses. We hope these efforts help strengthen your confidence in our findings and respectfully invite you to reconsider your rating. Below, we provide detailed responses to each comment.


Response to W1 and Q1: Protocol Attribution

A1: We thank the reviewer for this observation. The Random Pair-to-extreme comparison is intended as a benchmark-to-deployment transfer diagnostic, assessing whether conventional rankings remain informative for chemically under-supported candidates, rather than estimating the causal effect of chemical support.

Within-protocol validation. To control for split-protocol differences, we compare moderate (Smax⁡∈[0.4,0.6)S_{\max}\in[0.4,0.6)) and extreme (Smax⁡<0.2S_{\max}<0.2) support cohorts within identical Drug-Cold partitions, using the same trained models and evaluation protocol (Section 4.3; Appendix A.9, Table 21).

Table R3.1. Within-Drug-Cold ranking stability.

DatasetSpearman ρMeaningful reversals
Yamanishi 08−0.37348.8%
MDAD−0.00844.0%
ChEMBL370.08739.3%

Cohort-size control. We further compare extreme-support cohorts against 1,000 size-matched moderate-support samples per run. Extreme-support ranking agreement with Random Pair is lower on Yamanishi 08 (−0.302 vs. 0.604), MDAD (0.183 vs. 0.540), and ChEMBL37 (0.087 vs. 0.732). Both rank-correlation and meaningful-reversal differences remain significant after Holm correction (p ≤ 0.011 and p ≤ 0.013, respectively; Figure 3b; Appendix A.12, Table 26).

Together, these analyses show that ranking changes persist within fixed evaluation protocols and cannot be explained by cohort size alone, supporting the practical need to reassess benchmark rankings for chemically under-supported candidates.


Response to W2 and Q2: MDAD Cohort Accounting and Prediction Coverage

A2: We thank the reviewer for identifying this discrepancy. We audited the split manifests, chemical-support records, and model predictions to identify its source.

Cohort accounting. The MDAD extreme-support Drug-Cold cohorts contain 8, 9, and 8 drugs across three runs (mean 8.3). All eight models (M1–M8) produce valid predictions for all 169 Drug-Cold test drugs per run, with no model-specific exclusions. The discrepancy instead arises from Appendix A.7 (Table 14), where drug counts were inadvertently averaged across Drug-Cold and Scaffold-Cold protocols.

Table R3.2. MDAD support-bin drug counts.

Support binDrug-ColdScaffold-ColdOriginally reported
0.8–1.022.71.712.2
0.6–0.860.320.740.5
0.4–0.643.045.744.3
0.2–0.434.771.052.8
0.0–0.28.330.019.2

Mean counts across three runs. Originally reported values equal the averages of the two protocols (up to rounding).

Impact and correction. This is a reporting inconsistency, not missing predictions. The primary analyses already use the correct Drug-Cold cohorts (8/9/8 drugs); therefore, all reported rankings, correlations, and reversal rates remain unchanged. Our broader audit found no similar inconsistencies. We will correct Table 14, clarify the aggregation error, and include the prediction-coverage audit in the camera-ready version.

AuthorsOpenReview ↗

Response to Reviewer Nk2E (Part 2 / 3)

Response to W3: Representation Dependence

A3: We thank the reviewer for this observation. We acknowledge that chemical support is representation dependent (Section 5.2; Appendix A.15). Morgan-r2 Tanimoto provides a reproducible operational definition, rather than a representation-invariant measure of molecular novelty.

Cross-representation evidence. Existing analyses show strong agreement between Morgan-r2 and Morgan-r3 (ρ = 0.978–0.987), whereas MACCS-166 attenuates ranking transitions on MDAD and ChEMBL37 (ρ = 0.667/0.556; reversals = 11.9%). We further evaluated frozen ChemBERTa-77M and MoLFormer-XL embeddings using cosine similarity, matching low-support cohort sizes while keeping partitions, models, predictions, and metrics fixed.

Table R3.3. Ranking changes across chemical-support representations.

DatasetMorgan-r2ChemBERTaMoLFormer-XL
Yamanishi 0848.8% / 2/333.3% / 2/341.7% / 3/3
MDAD34.5% / 3/319.0% / 2/320.2% / 2/3
ChEMBL3734.5% / 2/315.5% / 3/321.4% / 2/3

Meaningful pairwise reversals / partitions with Top-1 changes; reversals require |ΔAP| > 0.005 in both regimes.

Interpretation and scope. Although pretrained representations attenuate ranking changes, Top-1 changes persist in at least 2/3 partitions across all datasets. Their low-support cohorts also show limited overlap with Morgan-r2 (25.4%, 12.0%, and 23.6%, respectively). These findings confirm that both cohort composition and model-selection outcomes depend on the support representation. Our framework therefore evaluates ranking reliability under explicitly specified support definitions, without assuming cross-representation invariance.


Response to Q3: Candidate-Model-Set Dependence

A4: We thank the reviewer for raising this point. We clarify that ranking instability is conditional on the candidate model set and chemical-support regime, rather than a universal phenomenon (Introduction and 4.5).

Evidence across model families. As shown in Figure 4 and Appendix A.5 (Tables 8–10), six representative DTI architectures exhibit weaker but non-zero ranking changes:

Table R3.4. Ranking stability across candidate model sets.

DatasetCandidate modelsMean Spearman ρMeaningful reversalsTop-1 agreement
Yamanishi 08Six DTI architectures0.54313.3%1/3
ChEMBL37Six DTI architectures0.46713.3%2/3
MDADThree task-specific models0.66711.1%1/3

Results are averaged across three independent partitions.

Importantly, aggregate ranking agreement does not imply run-level stability. On Yamanishi 08, the run-averaged model ordering remains unchanged, yet the Top-1 model changes in 2/3 individual runs, with 13.3% meaningful pairwise reversals.

These results demonstrate that ranking stability depends on both the candidate model set and chemical-support regime. Our framework quantifies this dependence and identifies when conventional benchmark rankings remain informative or warrant reassessment.

AuthorsOpenReview ↗

Response to Reviewer Nk2E (Part 3 / 3)

Response to Q4: Covariate-Matched Ranking Stability on Yamanishi 08

A5: We thank the reviewer for this suggestion. To assess whether ranking changes persist after accounting for measured covariates, we conducted a covariate-matched analysis on Yamanishi 08, using four variables: positive count, positive prevalence, molecular weight, and heavy-atom count.

Matching protocol. Within each of three fixed Drug-Cold runs, we performed one-to-one matching between low-support and moderate-support drugs (Smax⁡∈[0.4,0.6)S_{\max}\in[0.4,0.6)), using the original predictions of all eight models without retraining. All low-support drugs were successfully matched (100% coverage).

Table R3.5. Ranking stability after covariate matching.

Low-support cohortMean drugs/runCoverageSpearman ρ (Mean ± SD)Meaningful reversalsTop-1 agreement
Smax⁡<0.2S_{\max}<0.25.0100%−0.468 ± 0.21345.2%0/3
Bottom 10%16.0100%0.000 ± 0.53642.9%0/3

Rankings compare matched moderate- and low-support cohorts within each run. Meaningful reversals require ∣ΔAP∣>0.005|\Delta\mathrm{AP}|>0.005 in both regimes. Top-1 agreement denotes the number of runs selecting the same best-performing model in both cohorts.

Results and implications. Both low-support definitions retain substantial ranking changes after matching, with 45.2% and 42.9% meaningful pairwise reversals, respectively, and no Top-1 agreement across the three runs. These findings indicate that matching on the four measured covariates does not eliminate ranking transitions. The matching protocol and results are provided in Appendix A.13 (Table 27).


Response to Q5: Abstract and Introduction

A6: We thank the reviewer for this suggestion. We will streamline the Abstract to highlight the motivation, our chemical-support-resolved evaluation framework, and the key finding that ranking instability is conditional rather than universal.

Introduction. We will reorganize it around motivation → limitations → research question → evaluation framework → contributions, clarifying how conventional Drug-Cold and Scaffold-Cold splits may conceal heterogeneous chemical support and distinguishing our focus on model-ranking reliability from prior evaluations of individual-model performance.

We will incorporate these additional analyses and clarifications into the revised manuscript while preserving the original results and conclusions. These changes will be included in the camera-ready version.


We are happy to see from your comments that you have an overall positive view of our work, and we respectfully ask if you'd be comfortable to adjust your ratings, based on all the new experiments and justification to your questions. Thank you once again for your time and effort. We look forward to incorporating these insightful discussions into the final camera-ready version.

Reviewer_D8hzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the inconsistence between models’ performance in benchmark and deployment due to the distributional shift. Specifically, when testing compounds in deployment are far from the training distribution, the model performance ordering observed in a benchmark would be different from model ordering in actual deployment. The authors propose to use a drug-specific chemical-support, S_{max}, the maximum Tanimoto similarity between Morgan fingerprints to the training data, to measures how far a testing drug is from the training distribution. The authors report the model performance and model rankings for different S_{max} under different training-testing splits, including random, drug-cold, and scaffold-cold. The paper concludes that 1. Even under scaffold-cold, there are ~10% test drugs are similar to the training distribution; 2. The model ranking changed largely in out-of-distribution regime (small support), which motivates that future benchmark leaderboard should be conditional on distributional shifts level to access models’ extrapolation ability.

Critical strengths

This paper studies the potential mismatch of model performance on benchmarks and on real-world applications where distributional shifts are common, which is an important bridge to connect model development and deployment in the drug property prediction field. This paper provides a framework to test if a model remains competitive performance across different extrapolation levels and it also potentially motivates more research in developing models that have robust performance under distributional shifts.

Critical weaknesses

My biggest concern of this paper is its significance. Most of the results are not surprising under Occam's razor principle: if a model generalizes well on severe OOD regimes, it is likely to only have a modest performance on ID regimes. Therefore, the model ranking will likely to change under different OOD level. Moreover, the paper defines OOD level by the maximum Tanimoto similarity between Morgan fingerprints, which is not properly justified. There are other ways of defining OOD level, such as the distance between embeddings from a molecular foundation model or the probability of a test datapoint in the training distribution (represented by a deep generative model).

Questions for authors
  1. Why do the authors choose to use maximum Tanimoto similarity between Morgan fingerprints, rather than alternative approaches, such as the minimum distance between molecular embeddings from some pretrained FMs or the testing data likelihood on the training distribution represented by a molecular diffusion model?
  2. What is mean macro AP in the results section? Is it the AP averaged across different methods? If that’s the case, won’t it be biased by several very strong or very weak methods?
  3. It is unclear why there are three bars for each category in Figure 2a. The meaning of different lines in Figure 2c is also unclear.
  4. Did the authors repeat experiments using different random seeds of training-testing splits? If so, it would be important to show the standard deviation of each result to access the statistical significance of the conclusion.
5 comments
AuthorsOpenReview ↗

Response to Reviewer D8hz (Part 1 / 2)

We sincerely thank the reviewer for the thoughtful and constructive feedback and for recognizing the importance of bridging benchmark evaluation and real-world deployment, as well as the value of assessing model reliability across different extrapolation levels.

We have carefully addressed your concerns through additional analyses. We hope these efforts strengthen your confidence in our findings and respectfully invite you to reconsider your rating. Below, we provide detailed responses to each comment.


Response to Q1 and W1: Chemical-Support Definition and Representation Sensitivity

A1: We thank the reviewer for this question. We adopt maximum Morgan–Tanimoto similarity as an interpretable, deterministic, and predictor-independent measure of nearest-neighbor chemical support (Section 3.2; Appendix A.3). Unlike learned embedding distances or generative likelihoods, it directly measures structural proximity without requiring a fitted representation or density model.

Representation sensitivity. We additionally evaluated frozen ChemBERTa-77M and MoLFormer-XL embeddings using cosine similarity, keeping data splits, predictions, evaluation candidates, and cohort sizes fixed.

Table R2.1. Ranking changes across chemical representations.

DatasetMorgan-r2ChemBERTa-77MMoLFormer-XL
Yamanishi 0848.8% / 2/333.3% / 2/341.7% / 3/3
MDAD34.5% / 3/319.0% / 2/320.2% / 2/3
ChEMBL3734.5% / 2/315.5% / 3/321.4% / 2/3

Meaningful pairwise reversal rate / partitions with Top-1 changes.

Top-1 changes persist in at least 2/3 partitions across all representations, although reversal rates vary, particularly on MDAD and ChEMBL37. This supports representation-dependent rather than universal ranking instability (Section 4.5; Appendix A.14). We therefore use Morgan–Tanimoto as a transparent reference measure, without claiming its superiority over alternative representations.

Top-1 changes persist in at least 2/3 partitions across all representations, although reversal rates vary, particularly on MDAD and ChEMBL37. This supports representation-dependent rather than universal ranking instability (Section 4.5; Appendix A.14). We therefore use Morgan–Tanimoto as a transparent reference measure, without claiming its superiority over alternative representations.

Top-1 changes persist in at least 2/3 partitions across all representations, although reversal rates vary, particularly on MDAD and ChEMBL37. This supports representation-dependent rather than universal ranking instability (Section 4.5; Appendix A.15). We therefore use Morgan–Tanimoto as a transparent reference measure, without claiming its superiority over alternative representations.


Response to Q2: Clarification of Mean Macro AP

A2: We thank the reviewer for this question. Macro drug-level AP is computed independently for each model, without pooling predictions or labels across models (Section 3.3; Appendix A.2):

MacroAPm,r(R)=(1/∣DR∣)sumdinDRAPm,r(d)MacroAP_{m,r}(R) = (1/|D_R|) sum_{d in D_R} AP_{m,r}(d)

Each drug is equally weighted, with AP = 0 assigned when no positive candidates are observed.

Mean macro AP (Figure 2b; Table 15) is a descriptive average of model-specific scores across models and runs. Model rankings instead use each model's own macro AP within the same run and support regime, never the cross-model average. Model-specific results are provided in Appendix A.7 (Tables 16–18). We have clarified this distinction in the revised manuscript (Appendix A.7, Table 15).


Response to Q3: Clarification of Figure 2

A3: We clarify the visual encodings in Figure 2:

Figure 2(a). The three adjacent elements represent independent runs, not methods. Thin intervals show the 10th–90th percentiles, thick segments the interquartile range, and markers the median. Colors distinguish Drug-Cold and Scaffold-Cold.

Figure 2(c). Each trajectory represents one of eight models across Random Pair and decreasing Drug-Cold support. Grey lines denote other models; highlighted solid and dashed lines identify the Random Pair and extreme-support winners, respectively. Rank 1 indicates the best model.

We will clarify these conventions in the Figure 2 caption in the camera-ready version.

AuthorsOpenReview ↗

Response to Reviewer D8hz (Part 2 / 2)

Response to Q4: Independent Partitions and Run-to-Run Variability

A4: We thank the reviewer for this suggestion. All primary experiments were conducted across three independent train–test partitions, with identical partitions shared across the compared models within each run, as detailed in Appendix A.2. The run-level model-ranking results are already provided in Appendix A.9 (Table 20).

In response to the reviewer's suggestion, we have additionally computed the sample standard deviations across the three runs for the main ranking-stability metrics (Table 3) and support-resolved macro AP results (Appendix A.7, Table 15). We will report these results as mean ± standard deviation in the camera-ready version.


Response to W1: Significance and the Role of the Evaluation Protocol

A5: We thank the reviewer for this comment. We agree that model-specific performance trade-offs under distribution shift are plausible. However, our contribution is not simply to observe that ID and OOD rankings can differ. Rather, we investigate when conventional benchmark leaderboards become unreliable for model selection, whether observed changes persist under controlled comparisons, and whether they have practical selection consequences.

Controlled evidence. We distinguish performance degradation from ranking stability while accounting for split differences, near ties, and cohort size. Compared with size-matched moderate-support cohorts, extreme-support ranking agreement is lower on Yamanishi 08 (ρ = −0.302 vs. 0.604), MDAD (0.183 vs. 0.540), and ChEMBL37 (0.087 vs. 0.732; Figure 3b; Appendix A.12; Holm-adjusted p ≤ 0.013). Thus, cohort size alone does not explain the observed differences.

Selection consequences. Label-disjoint selection improves held-out macro AP on MDAD (+0.059) but reduces it on ChEMBL37 (−0.016; Appendix A.14), demonstrating task-dependent rather than universal benefits.

Our contribution is a reproducible chemical-support-resolved protocol linking benchmark rankings to the chemical support of intended deployment. Practitioners can characterize target libraries using unlabeled structures, evaluate models on labeled compounds across support regimes, and use label-disjoint selection to assess transfer to disjoint low-support compounds. When rankings vary, support-matched validation is warranted before deployment (Table 4).


We hope the response has addressed your concerns. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work. We are more than happy to include all these discussions in the camera-ready version of this work.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Reviewer_SNAAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A concise summary of the work is that it focuses on the stability of molecular model selection under chemical extrapolation, especially when test compounds have low similarity to the training data, which is a very challenge faced in the real world. Thus, their main contribution is an evaluation framework for analyzing performance degradation and changes in model rankings. Experiments on several molecular datasets and models show that rankings can change considerably across support regimes, although the practical value for improving model selection is less clear and can be strengthened.

Critical strengths

S1. They study an interesting and practical problem of whether model rankings remain reliable when test molecules become less similar to training data (which is generally the case for these massive virtual libraries that exist today going well become even diverse/representative data using to obtaining the labeled training data).

S2. The proposed evaluation provides a more fine-grained view compared to conventional cold-start splits, which could be useful for molecular benchmarks. Also, the experiments cover multiple datasets/architectures, and the authors include a good number of additional analyses.

S3. The distinction between performance degradation and ranking stability is interesting, since lower performance does not necessarily mean a different model should be used instead.

Critical weaknesses

W1) The technical novelty is somewhat limited (although finding the results interesting). The framework generally uses existing chemical similarity measures and standard ranking metrics, while being observational. While the evaluation perspective is useful, the contribution seems largely empirical rather than introducing a substantially new methodology or way to observe/analyze these trends.

W2) Some of the main findings rely on very small cohorts, e.g., only a few drugs per run. This makes it difficult to tell how much of the ranking instability is due to genuine differences between models versus sampling variation...

W3) The choice of similarity measure seems to have a considerable impact on the results. Different fingerprint representations can produce different support groups and ranking behavior, which raises concerns about how general the conclusions are and would welcome further discussion on this.

W4) It is still unclear how much the proposed evaluation helps in practice (see question below).

Questions for authors

Q1)What would be the main advantage of this framework over simply evaluating model rankings across different similarity based test subsets? Is there an additional methodological contribution beyond this analysis? (This is seen as the most fundamental question I had for this work.)

Q2) How reliable are the ranking changes in the (extreme) low-support groups given their small sample sizes? With the follow-up: Could some of these changes be explained by evaluation variance?

Q3) Based on these findings, then how should practitioners choose the chemical similarity measure, and do the authors expect the resulting model selections to generalize across different representations?

4 comments
AuthorsOpenReview ↗

Response to Reviewer SNAA (Part 1 / 2)

We sincerely thank the reviewer for the thoughtful and constructive feedback and for recognizing the practical importance of model-selection reliability under molecular extrapolation, as well as the value of our finer-grained evaluation framework in distinguishing performance degradation from ranking stability.

We have carefully addressed your concerns through additional analyses. We hope these efforts strengthen your confidence in our findings and respectfully invite you to reconsider your rating. Below, we provide detailed responses to each comment.


Response to W1 and Q1: Contribution Beyond Similarity-Based Evaluation

A1: We thank the reviewer for this question. While chemical similarity analysis and ranking metrics are established, our contribution is to assess the reliability of benchmark-based model rankings and selection under chemical extrapolation, rather than merely measuring performance degradation.

Methodological distinction. Unlike SAE [1], which examines similarity-dependent performance, and DomainBed [2], which studies selection under distribution shift, our framework evaluates ranking stability across chemical-support regimes and its consequences for model selection. It combines within-Drug-Cold comparisons, meaningful ranking reversals, size-matched controls, drug-level uncertainty, and label-disjoint selection (Sections 3.3–4.4; Appendices A.9, A.12, and A.14).

Key evidence. On Yamanishi 08, extreme-support ranking agreement is substantially lower than the size-matched reference (ρ = −0.302 vs. 0.604; Holm-adjusted p = 0.003; Table 26), ruling out cohort size as the sole explanation. Label-disjoint selection changes held-out macro AP by +0.059 on MDAD and −0.016 on ChEMBL37 (Table 29), revealing dataset-dependent selection consequences.

Our contribution is therefore a reproducible framework for evaluating when benchmark rankings can be trusted for model selection. We will sharpen this distinction in the camera-ready version.

References

[1] Zhang et al. Rethinking the Generalization of Drug Target Affinity Prediction Algorithms via Similarity Aware Evaluation. ICLR 2025.

[2]Gulrajani and Lopez-Paz. In Search of Lost Domain Generalization. ICLR 2021.


Response to W2 and Q2: Ranking Reliability in Small Extreme-Support Cohorts

A2: We thank the reviewer for this concern. We assess whether ranking changes arise from small cohort sizes through size-matched controls, drug-level uncertainty, and influence analyses (Section 4.3; Appendix A.12).

Size-matched controls. Using 1,000 moderate-support subsets per run matched to extreme-cohort sizes, with models and predictions fixed, ranking correlations were −0.302 vs. 0.604 (Yamanishi 08), 0.183 vs. 0.540 (MDAD), and 0.087 vs. 0.732 (ChEMBL37). All differences remained significant (Holm-adjusted p ≤ 0.011; Table 26), as did meaningful pairwise reversals (p ≤ 0.013), indicating that cohort size alone cannot explain the instability.

Uncertainty and robustness. Whole-drug bootstraps reveal substantial uncertainty (Yamanishi 08: 95% CI [−0.437, 0.207]; Table 25). However, leave-one-drug-out analyses never restored the Random Pair winner in runs with differing winners, and all eight models predicted every drug, excluding single-drug dominance and missing-prediction filtering as explanations (Appendix A.12).

These results support systematic ranking differences across chemical-support regimes beyond cohort-size effects.

AuthorsOpenReview ↗

Response to Reviewer SNAA (Part 2 / 2)

Response to W3 and Q3: Chemical Representation and Generalization

A3: We thank the reviewer for this important observation. We clarify that chemical support is representation dependent, and Morgan–Tanimoto is not intended as a representation-invariant measure of molecular novelty.

Similarity choice. We use radius-2 Morgan fingerprints with Tanimoto similarity [3, 4] as a pre-specified, interpretable measure of local structural support. It provides a deterministic and predictor-independent measure of nearest-neighbor chemical proximity (Section 3.2; Appendix A.3).

Cross-representation evidence. Under identical Drug-Cold partitions and predictions, Morgan-r3 closely agrees with Morgan-r2 (Smax⁡S_{\max}) correlations = 0.978–0.987) and preserves the principal ranking transitions: Yamanishi 08 retains negative rank agreement (ρ=-0.246), while MDAD and ChEMBL37 retain 34.5% meaningful pairwise reversals. In contrast, MACCS-166 produces different low-support cohorts (25.0%–49.0% overlap) and attenuates ranking transitions on MDAD and ChEMBL37 ((ρ=0.667/0.556), with 11.9% meaningful reversals in both; Appendix A.15, Tables 30–31).

Practical implications. These findings support robustness across closely related Morgan representations, but not representation-invariant model selection. We recommend pre-specifying the similarity measure according to the target chemical context, reporting cohort overlap across reasonable alternatives, and checking ranking and Top-1 consistency. When different representations yield different model selections, target-like labeled validation is necessary.

Overall contribution. Our contribution is therefore not to claim cross-representation invariance, but to provide an evaluation protocol that makes support definitions explicit, quantifies ranking stability across support regimes, and assesses whether the resulting model-selection decisions transfer to target-like compounds. We will incorporate these recommendations in the camera-ready version.

[3] Rogers and Hahn. Extended-Connectivity Fingerprints. JCIM 2010.

[4]Bajusz et al. Why Is Tanimoto Index an Appropriate Choice for Fingerprint-Based Similarity Calculations? Journal of Cheminformatics 2015.


Response to W4: Practical Utility for Model Selection

A4: We thank the reviewer for this question. Our framework helps practitioners assess whether benchmark-selected models remain reliable for a target chemical regime.

Practical guidance. Using unlabeled molecular structures, practitioners can characterize target-library chemical support, examine support-conditioned rankings and uncertainty on labeled validation data, and identify regimes requiring targeted validation (Section 5.2; Table 4).

Decision-level evidence. The retrospective oracle gap reaches 0.130 macro AP on MDAD (Table 28). More importantly, label-disjoint selection changes held-out macro AP by +0.059 on MDAD, +0.029 on MIFNDRA, and −0.016 on ChEMBL37 (Table 29), demonstrating both potential benefits and limitations.

Thus, our practical contribution is a reproducible procedure for auditing model-selection reliability and identifying when support-matched validation is warranted, rather than assuming universal transferability of benchmark rankings.


We hope the response has addressed your concerns. If you have any additional concerns or comments that we may have missed in our responses, we would be most grateful for any further feedback from you to help us further enhance our work. We are more than happy to include all these discussions in the camera-ready version of this work.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer SNAA,

Thank you again for your thoughtful review and constructive questions. As the discussion period approaches its close, we would like to follow up on our responses above.

Since posting our responses, we have uploaded a revised manuscript that further clarifies our methodological contribution beyond similarity-based performance evaluation, strengthens the presentation of size-matched controls and uncertainty analyses (Section 4.3; Appendix A.12), and elaborates on representation sensitivity and practical model-selection guidance (Sections 4.5 and 5.2; Table 4). The relevant changes are highlighted in blue.

We would greatly appreciate it if you could take another look and let us know whether our responses and revisions address your concerns. We would be happy to provide any further clarification before the discussion closes.

Best regards,

The Authors

Comments on the paper

1 comment

We sincerely thank all three reviewers for their thoughtful and constructive feedback and for recognizing the importance of model-selection reliability under molecular extrapolation. We have carefully addressed their concerns through targeted analyses, statistical clarifications, and focused improvements to the presentation. Below, we summarize the core contributions of our work and the updates made in response to the reviews.

Core Contributions of Our Work

  1. Chemical-Support-Resolved Evaluation. We introduce a finer-grained evaluation framework that characterizes each held-out drug by its continuous training-set chemical support. This complements conventional Drug-Cold and Scaffold-Cold splits by revealing chemical-support heterogeneity that coarse split labels cannot capture.

  2. Systematic Assessment of Leaderboard Reliability. Beyond measuring predictive degradation, our framework evaluates whether benchmark-derived model rankings remain informative under chemical extrapolation. It combines support-conditioned ranking analysis with controlled comparisons, near-tie handling, and finite-cohort uncertainty assessment to distinguish ranking changes from evaluation artifacts.

  3. Practical Guidance for Model Selection. Our framework connects benchmark evaluation to target screening libraries by using unlabeled molecular structures to characterize chemical support and labeled validation compounds to assess model rankings. Label-disjoint selection further evaluates whether model choices transfer to independent low-support compounds, informing support-matched validation before deployment.

Targeted Experimental and Statistical Updates

  • Ranking reliability and confounding (R1, R3): Clarified the roles of cross-protocol and within-Drug-Cold comparisons and supplemented the existing robustness evidence with covariate-matched ranking analysis on Yamanishi 08 (Section 4.3; Appendix A.13).
  • Representation sensitivity (R1, R2, R3): Clarified the Morgan–Tanimoto support definition and examined frozen ChemBERTa-77M and MoLFormer-XL representations while holding evaluation partitions, predictions, and candidate models fixed (Section 4.5; Appendix A.15).
  • Statistical reporting and cohort accounting (R2, R3): Clarified macro drug-level AP aggregation, reported three-run standard deviations (Tables 3 and 15), and corrected the MDAD support-bin accounting in Table 14. This correction does not affect the original model-ranking results.

Clarifications to Presentation and Practical Guidance

  • Evaluation framework and significance (R1, R2): Clarified the distinction between similarity-based performance assessment and evaluating the reliability of benchmark-based model selection.
  • Interpretation and practical use (R1, R3): Refined Section 5.2 and Table 4 to clarify representation dependence, support-aware validation, and the practical interpretation of ranking changes.
  • Presentation and clarity (R2, R3): Clarified the visual conventions in Figure 2 and refined the Abstract and Introduction to better articulate the original motivation, research question, and contributions.

These focused updates address the reviewers' concerns and strengthen the clarity and reliability of our findings while preserving the scope and principal conclusions of the original submission. For ease of reference, the accompanying annotated manuscript highlights the relevant changes in blue.

We sincerely appreciate the reviewers' time and consideration and hope these clarifications strengthen confidence in the reliability and practical relevance of our findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.