- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether cross-validation results can reliably reflect model performance in the target application domain for small molecule property prediction. The authors propose an MCES-based data splitting method with cluster-size constraints and use MCES distance distributions to compare different CV splits with the target application domain. They evaluate 5 baseline models and 9 foundation models on 10 experimental datasets, finding that MCES splits generally produce larger structural separation, better match the distance distributions of the application domain, and lead to lower predictive performance. They also analyze the structural overlap between foundation model pretraining data and evaluation datasets.
Critical strengths
The paper investigates an important generalization problem in molecular machine learning. The experiments cover multiple datasets and a wide range of baseline and foundation models. The analysis of structural overlap between pretraining and evaluation datasets is also useful.
Critical weaknesses
Weakness 1: Insufficient justification of the methodological novelty. The use of "for the first time" is inappropriate. MCES has already been used for molecular structural similarity analysis and for data splitting and evaluation in specific tasks. In addition, comparing distance distributions between cross-validation and the intended application domain is not unique to MCES. The authors should clarify the novelty of their approach.
Weakness 2: Lack of key comparisons with existing methods. The authors compare structural separation using MCES-based Butina splitting. However, two important comparisons are missing:
- MCES versus Tanimoto similarity in measuring how closely cross-validation reflects the intended application domain.
- Whether cross-validation performance under MCES-based splitting or conventional Tanimoto-based Butina splitting better reflects performance on the real test set. In addition, the Tanimoto similarity threshold used for Butina splitting could not be found in the code. This threshold can affect the resulting data splits. These missing comparisons make it difficult to determine whether MCES offers advantages over existing Tanimoto-based methods for assessing domain differences and model generalization.
The writing could also be more concise, and a workflow figure would help clarify the overall splitting and evaluation framework.
Questions for authors
- What are the specific advantages of MCES over Tanimoto distance in measuring the gap between CV and the target application domain?
- What evidence, beyond matching the real test set's MCES distance distribution, shows that MCES-based splitting better reflects the intended application domain rather than simply producing a more challenging split?
4 comments
Reviewer:
What evidence, beyond matching the real test set's MCES distance distribution, shows that MCES-based splitting better reflects the intended application domain rather than simply producing a more challenging split?
Response: It is impossible to prove that a split is realistic. What our test can do is falsify a hypothesis: if a split's distance distribution to its training data differs from that of the application domain, its estimate cannot be assumed to transfer. This is the case for uniform and scaffold splits, but not for MCES splits. While we cannot prove splits to be realistic, we have now added a case study on a new external dataset, demonstrating that uniform and scaffold estimates are too optimistic for every machine learning model we looked at, while the performance estimates from MCES splits are not. See our reply to Reviewer 4UF9 for more details.
Thank you for adding the Tanimoto analysis and ESOL–AqSolDB external validation. The new experiments substantially address my concern about real-world generalization, and I acknowledge these results. However, I still have two reservations:
-
Methodological novelty: My intention in raising Tanimoto was not to compare the merits of the two distance metrics, but to point out that the authors' emphasized claim that "the closeness of a cross-validation result to an explicitly stated application domain can be measured" is not unique to MCES. By the paper's definition, other structural distance metrics can also perform this comparison. Regarding the claimed contribution of "performing the comparison at all," please see my next point.
-
Evaluation criteria and actual contribution: The original paper uses MCES to construct data splits and also to calculate the reference distance distribution of the application domain, then concludes that MCES splits are closer to that domain. This evaluation is inherently biased. I acknowledge that the additional Tanimoto analysis and external validation partly address this concern. However, if the authors merely demonstrate that MCES better matches the application domain, or that performance estimates from MCES splits are closer to external test results, these findings primarily demonstrate the effectiveness of the existing MCES method and are insufficient to support the central contribution of this benchmark paper.
Regarding writing: I suggest shortening the last two sentences of the first paragraph, the second paragraph, and the final paragraph of the Introduction. These sections contain considerable basic ML background and experimental setup details, whereas the core motivation and contributions are mainly presented in the penultimate paragraph. I recommend emphasizing the central research question and contributions while reducing unnecessary explanations of basic ML concepts.
We thank the reviewer for engaging with the new experiments.
Reviewer:
Methodological novelty: My intention in raising Tanimoto was not to compare the merits of the two distance metrics, but to point out that the authors' emphasized claim that "the closeness of a cross-validation result to an explicitly stated application domain can be measured" is not unique to MCES. By the paper's definition, other structural distance metrics can also perform this comparison. Regarding the claimed contribution of "performing the comparison at all," please see my next point.
and
Evaluation criteria and actual contribution: The original paper uses MCES to construct data splits and also to calculate the reference distance distribution of the application domain, then concludes that MCES splits are closer to that domain. This evaluation is inherently biased. I acknowledge that the additional Tanimoto analysis and external validation partly address this concern. However, if the authors merely demonstrate that MCES better matches the application domain, or that performance estimates from MCES splits are closer to external test results, these findings primarily demonstrate the effectiveness of the existing MCES method and are insufficient to support the central contribution of this benchmark paper.
Response: We take these two points together. We agree that MCES distances and also MCES-based splitting are already established. Our contribution consists of two things.
First, we fix a known flaw in plain MCES splitting. A single-linkage cut at a meaningful threshold produces one cluster far larger than a fold may be, and prior work either avoids the threshold or accepts the consequence: Gupta et al. 2026 lower it to τ = 3 because higher values give oversized clusters, while Kretschmer et al. 2026 keep τ = 10 and accept the resulting unbalanced folds. We replace both by a size bound, which yields k balanced folds automatically and relaxes the threshold only for the clusters that exceed it, not for the whole dataset. The new d<10 column reports what this costs for every dataset and α.
Second, we compare a given split against the chosen application domain, which is what distinguishes a realistic split from a merely harder one. Raising the threshold alone only makes a split harder: at α = 0.5 the external deviation already turns positive for four of five models. The intended use is to adjust α and τ until the distributions match given a chosen application domain. The ten MoleculeNet datasets demonstrate the comparison rather than perform that calibration: τ = 10 is taken from prior work and α = 0.3 to 0.5 is the range keeping folds balanced, so neither was tuned against a domain here.
Reviewer:
Regarding writing: I suggest shortening the last two sentences of the first paragraph, the second paragraph, and the final paragraph of the Introduction. These sections contain considerable basic ML background and experimental setup details, whereas the core motivation and contributions are mainly presented in the penultimate paragraph. I recommend emphasizing the central research question and contributions while reducing unnecessary explanations of basic ML concepts.
Response: Thank you for the concrete pointers; we adjusted the text accordingly.
We thank the reviewer for the detailed and concrete criticism.
Reviewer:
Response: Regarding novelty, we have to respectfully disagree. We do not claim novelty for MCES, nor for MCES-based splitting: prior uses are cited in Sec. 2 (Kretschmer et al. 2024; Bushuiev et al. 2024; Gupta et al. 2026). However, we propose a method that is able to compare given splits to an explicitly stated application domain. The comparison is indeed not restricted to MCES distances, and we do not claim that in the manuscript. The sentence in question is already restricted to a different claim — that the closeness of a cross-validation result to an explicitly stated application domain can be measured, and it explicitly sets aside the "our folds are harder" reading. We agree that this comparison is not tied to MCES, which is why we repeated it under Tanimoto with the same outcome. The contribution is performing the comparison at all, not the distance used for it. We are not aware of earlier work that quantifies the gap between cross-validation folds and a stated application domain in this way, and we would gladly cite it if the reviewer can point us to it.
Reviewer:
Response: The paper is not about MCES versus Tanimoto as splitting methods; it is about comparing a split against a stated application domain, and that comparison works with any distance. We repeated it under maximum ECFP4 Tanimoto similarity for all ten datasets, and the outcome is unchanged: uniform, scaffold, and Butina fail the test in Tanimoto space just as they do in MCES space, while MCES passes it. This plot will be added to the manuscript.
Reviewer:
Response: Thanks for pointing out the missing threshold for Butina splits, sorry for leaving it out. We use a cutoff of 0.65 for ECFP4 Tanimoto coefficients, with the clusters greedily packed into folds, as now stated in the paper. The threshold indeed influences the split, and we tested out multiple values (0.50 and 0.35) on six datasets. While lower cutoffs lead to a greater separation of folds, the trends do not change.
Reviewer:
Response: We would be happy to make the manuscript more concise where possible, but we would need to know the exact parts of the manuscript, since the reviewers differ in their opinion on whether the text needs to be more concise or more details need to be added. We have added a workflow figure; good point.
Reviewer:
Response: As stated above, this paper is not about MCES versus Tanimoto, but we still prefer MCES for constructing splits, because Tanimoto is a normalised set overlap: the same chemical modification shifts it by an amount that depends on molecular size, moving a small molecule a long way and a large one almost not at all. MCES counts edit operations, so distance 1 means one edit regardless of the molecule, which is what makes a threshold such as τ = 10 interpretable.