acceptodds
Under review as a conference paper at ICLR 2027

Reliable evaluation for small molecule machine learning

Abstract

Cross-validation (CV) allows us to assess the generalizability of machine learning models for small molecule property prediction. Yet, checking whether evaluation results carry over to the target application domain is non-trivial. We introduce a method for comparing any CV split of small molecules against any chosen domain of application. We find that existing splitting methods produce test sets far from the application domain and are prone to data leakage. Next, we introduce a new splitting method based on Maximum Common Edge Subgraph (MCES) distances, and show that resulting CV splits align much better with the application domain of biomolecules. We evaluate five baseline and nine foundation models on different splits, and find that the strongest baseline consistently outperforms all but four of the foundation models on commonly used experimental datasets. All models show a drop of about 0.06 and 0.1 AUROC for MCES splits, indicating that previous splits substantially overstate real-world performance. Notably, pretraining data of foundation models overlaps substantially with downstream evaluation datasets. In the future, our methods may help to reach reliable evaluations for other graph-based machine learning tasks.

Primary area:
datasets and benchmarks
Keywords:
evaluation, splits, small molecules, datasets, mces, foundation models, application domain, cross-validation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_57h4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether cross-validation results can reliably reflect model performance in the target application domain for small molecule property prediction. The authors propose an MCES-based data splitting method with cluster-size constraints and use MCES distance distributions to compare different CV splits with the target application domain. They evaluate 5 baseline models and 9 foundation models on 10 experimental datasets, finding that MCES splits generally produce larger structural separation, better match the distance distributions of the application domain, and lead to lower predictive performance. They also analyze the structural overlap between foundation model pretraining data and evaluation datasets.

Critical strengths

The paper investigates an important generalization problem in molecular machine learning. The experiments cover multiple datasets and a wide range of baseline and foundation models. The analysis of structural overlap between pretraining and evaluation datasets is also useful.

Critical weaknesses

Weakness 1: Insufficient justification of the methodological novelty. The use of "for the first time" is inappropriate. MCES has already been used for molecular structural similarity analysis and for data splitting and evaluation in specific tasks. In addition, comparing distance distributions between cross-validation and the intended application domain is not unique to MCES. The authors should clarify the novelty of their approach.

Weakness 2: Lack of key comparisons with existing methods. The authors compare structural separation using MCES-based Butina splitting. However, two important comparisons are missing:

  1. MCES versus Tanimoto similarity in measuring how closely cross-validation reflects the intended application domain.
  2. Whether cross-validation performance under MCES-based splitting or conventional Tanimoto-based Butina splitting better reflects performance on the real test set. In addition, the Tanimoto similarity threshold used for Butina splitting could not be found in the code. This threshold can affect the resulting data splits. These missing comparisons make it difficult to determine whether MCES offers advantages over existing Tanimoto-based methods for assessing domain differences and model generalization.

The writing could also be more concise, and a workflow figure would help clarify the overall splitting and evaluation framework.

Questions for authors
  1. What are the specific advantages of MCES over Tanimoto distance in measuring the gap between CV and the target application domain?
  2. What evidence, beyond matching the real test set's MCES distance distribution, shows that MCES-based splitting better reflects the intended application domain rather than simply producing a more challenging split?
4 comments

We thank the reviewer for the detailed and concrete criticism.

Reviewer:

Insufficient justification of the methodological novelty. The use of "for the first time" is inappropriate. MCES has already been used for molecular structural similarity analysis and for data splitting and evaluation in specific tasks. In addition, comparing distance distributions between cross-validation and the intended application domain is not unique to MCES. The authors should clarify the novelty of their approach.

Response: Regarding novelty, we have to respectfully disagree. We do not claim novelty for MCES, nor for MCES-based splitting: prior uses are cited in Sec. 2 (Kretschmer et al. 2024; Bushuiev et al. 2024; Gupta et al. 2026). However, we propose a method that is able to compare given splits to an explicitly stated application domain. The comparison is indeed not restricted to MCES distances, and we do not claim that in the manuscript. The sentence in question is already restricted to a different claim — that the closeness of a cross-validation result to an explicitly stated application domain can be measured, and it explicitly sets aside the "our folds are harder" reading. We agree that this comparison is not tied to MCES, which is why we repeated it under Tanimoto with the same outcome. The contribution is performing the comparison at all, not the distance used for it. We are not aware of earlier work that quantifies the gap between cross-validation folds and a stated application domain in this way, and we would gladly cite it if the reviewer can point us to it.

Reviewer:

Weakness 2: Lack of key comparisons with existing methods. The authors compare structural separation using MCES-based Butina splitting. However, two important comparisons are missing:

  1. MCES versus Tanimoto similarity in measuring how closely cross-validation reflects the intended application domain.

Response: The paper is not about MCES versus Tanimoto as splitting methods; it is about comparing a split against a stated application domain, and that comparison works with any distance. We repeated it under maximum ECFP4 Tanimoto similarity for all ten datasets, and the outcome is unchanged: uniform, scaffold, and Butina fail the test in Tanimoto space just as they do in MCES space, while MCES passes it. This plot will be added to the manuscript.

Reviewer:

2. Whether cross-validation performance under MCES-based splitting or conventional Tanimoto-based Butina splitting better reflects performance on the real test set. In addition, the Tanimoto similarity threshold used for Butina splitting could not be found in the code. This threshold can affect the resulting data splits. These missing comparisons make it difficult to determine whether MCES offers advantages over existing Tanimoto-based methods for assessing domain differences and model generalization.

Response: Thanks for pointing out the missing threshold for Butina splits, sorry for leaving it out. We use a cutoff of 0.65 for ECFP4 Tanimoto coefficients, with the clusters greedily packed into folds, as now stated in the paper. The threshold indeed influences the split, and we tested out multiple values (0.50 and 0.35) on six datasets. While lower cutoffs lead to a greater separation of folds, the trends do not change.

Reviewer:

The writing could also be more concise, and a workflow figure would help clarify the overall splitting and evaluation framework.

Response: We would be happy to make the manuscript more concise where possible, but we would need to know the exact parts of the manuscript, since the reviewers differ in their opinion on whether the text needs to be more concise or more details need to be added. We have added a workflow figure; good point.

Reviewer:

What are the specific advantages of MCES over Tanimoto distance in measuring the gap between CV and the target application domain?

Response: As stated above, this paper is not about MCES versus Tanimoto, but we still prefer MCES for constructing splits, because Tanimoto is a normalised set overlap: the same chemical modification shifts it by an amount that depends on molecular size, moving a small molecule a long way and a large one almost not at all. MCES counts edit operations, so distance 1 means one edit regardless of the molecule, which is what makes a threshold such as τ = 10 interpretable.

Reviewer:

What evidence, beyond matching the real test set's MCES distance distribution, shows that MCES-based splitting better reflects the intended application domain rather than simply producing a more challenging split?

Response: It is impossible to prove that a split is realistic. What our test can do is falsify a hypothesis: if a split's distance distribution to its training data differs from that of the application domain, its estimate cannot be assumed to transfer. This is the case for uniform and scaffold splits, but not for MCES splits. While we cannot prove splits to be realistic, we have now added a case study on a new external dataset, demonstrating that uniform and scaffold estimates are too optimistic for every machine learning model we looked at, while the performance estimates from MCES splits are not. See our reply to Reviewer 4UF9 for more details.

Thank you for adding the Tanimoto analysis and ESOL–AqSolDB external validation. The new experiments substantially address my concern about real-world generalization, and I acknowledge these results. However, I still have two reservations:

  1. Methodological novelty: My intention in raising Tanimoto was not to compare the merits of the two distance metrics, but to point out that the authors' emphasized claim that "the closeness of a cross-validation result to an explicitly stated application domain can be measured" is not unique to MCES. By the paper's definition, other structural distance metrics can also perform this comparison. Regarding the claimed contribution of "performing the comparison at all," please see my next point.

  2. Evaluation criteria and actual contribution: The original paper uses MCES to construct data splits and also to calculate the reference distance distribution of the application domain, then concludes that MCES splits are closer to that domain. This evaluation is inherently biased. I acknowledge that the additional Tanimoto analysis and external validation partly address this concern. However, if the authors merely demonstrate that MCES better matches the application domain, or that performance estimates from MCES splits are closer to external test results, these findings primarily demonstrate the effectiveness of the existing MCES method and are insufficient to support the central contribution of this benchmark paper.

Regarding writing: I suggest shortening the last two sentences of the first paragraph, the second paragraph, and the final paragraph of the Introduction. These sections contain considerable basic ML background and experimental setup details, whereas the core motivation and contributions are mainly presented in the penultimate paragraph. I recommend emphasizing the central research question and contributions while reducing unnecessary explanations of basic ML concepts.

We thank the reviewer for engaging with the new experiments.

Reviewer:

Methodological novelty: My intention in raising Tanimoto was not to compare the merits of the two distance metrics, but to point out that the authors' emphasized claim that "the closeness of a cross-validation result to an explicitly stated application domain can be measured" is not unique to MCES. By the paper's definition, other structural distance metrics can also perform this comparison. Regarding the claimed contribution of "performing the comparison at all," please see my next point.

and

Evaluation criteria and actual contribution: The original paper uses MCES to construct data splits and also to calculate the reference distance distribution of the application domain, then concludes that MCES splits are closer to that domain. This evaluation is inherently biased. I acknowledge that the additional Tanimoto analysis and external validation partly address this concern. However, if the authors merely demonstrate that MCES better matches the application domain, or that performance estimates from MCES splits are closer to external test results, these findings primarily demonstrate the effectiveness of the existing MCES method and are insufficient to support the central contribution of this benchmark paper.

Response: We take these two points together. We agree that MCES distances and also MCES-based splitting are already established. Our contribution consists of two things.

First, we fix a known flaw in plain MCES splitting. A single-linkage cut at a meaningful threshold produces one cluster far larger than a fold may be, and prior work either avoids the threshold or accepts the consequence: Gupta et al. 2026 lower it to τ = 3 because higher values give oversized clusters, while Kretschmer et al. 2026 keep τ = 10 and accept the resulting unbalanced folds. We replace both by a size bound, which yields k balanced folds automatically and relaxes the threshold only for the clusters that exceed it, not for the whole dataset. The new d<10 column reports what this costs for every dataset and α.

Second, we compare a given split against the chosen application domain, which is what distinguishes a realistic split from a merely harder one. Raising the threshold alone only makes a split harder: at α = 0.5 the external deviation already turns positive for four of five models. The intended use is to adjust α and τ until the distributions match given a chosen application domain. The ten MoleculeNet datasets demonstrate the comparison rather than perform that calibration: τ = 10 is taken from prior work and α = 0.3 to 0.5 is the range keeping folds balanced, so neither was tuned against a domain here.

Reviewer:

Regarding writing: I suggest shortening the last two sentences of the first paragraph, the second paragraph, and the final paragraph of the Introduction. These sections contain considerable basic ML background and experimental setup details, whereas the core motivation and contributions are mainly presented in the penultimate paragraph. I recommend emphasizing the central research question and contributions while reducing unnecessary explanations of basic ML concepts.

Response: Thank you for the concrete pointers; we adjusted the text accordingly.

Reviewer_42ekOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this work, the authors propose a data splitting algorithm for molecular graph datasets. In particular, they propose using the MCES distance to split datasets, and propose a number of algorithm choices to ensure that using MCES will result in data splits that are relatively balanced in sizes. The authors argue that their method will result in splits that are more applicable for scientific domains and how datasets will be split and used in real scientific applications. The authors illustrate that using their splits results in worse performance for most modern machine learning models and by extension the authors claim that this is a more realistic estimate of downstream performance.

Critical strengths
  1. Dataset splitting is an important discipline in molecular datasets due to easy overestimation of model generalisation with poor splitting strategies.
Critical weaknesses
  1. The presentation and writing is at times hard to follow, and significant details of the methodology is missing, making the work difficult to understand.
  2. The proposed splitting method is proposed as if it will better match application settings, but this is not clear at all from the work, making it difficult/impossible to say if this work will make a difference in the claimed domains. Given that this is the most major contribution of such a method, I find it very difficult to recommend accepting the work in its current state.
Questions for authors
  1. Could the authors describe and include some more details on the myopic MCES distance and how exactly it works? Currently, it is (in my opinion) impossible to understand the actually distance computation algorithm due to lack of explanations in Section 3. Additionally, I would strongly suggest making Section 3 more concise but with more details - I currently find it very difficult to read, primarily due to lack of details and the lengthiness with which many things are written. Overall I am left with a lack of understanding of how exactly the proposed method ends up actually splitting the data. An algorithm box for example could be very helpful.
  2. Could the authors provide evidence of why they think their methodology for splitting would result in more "correct" splits? My general issue with this claim is that it may be extremely application dependent what kind of folds are correct, so generally claiming this to be the case for a splitting method seems over the top.

To summarise my opinions on this work: Firstly, work is in my opinion very lengthy and difficult to follow at times, making it difficult to gauge exactly what the splitting method does (and by extension, how this affects the splits) - I think the overall presentation could be significantly improved. Secondly, I think there are a lot of strong claims around what the correct splitting method might be for different applications, that is not at all supported currently - e.g. this splitting method works on molecular graph structure alone (and this being number of deletable edges, and actually on bonds, not on neighbourhoods), but for some applications of molecules this may not be correct - e.g. this will (as far as I am aware) not capture functional similarities of molecules. My recommendation to the authors would be to change the narrative to the work, such that the method is a general graph splitting algorithm, and then perhaps illustrating applications (with concrete evidence) where such a graph splitting algorithm will lead to better generalisation estimates.

5 comments

We thank the reviewer for the feedback on presentation.

Reviewer:

Critical Weakness 1: The presentation and writing is at times hard to follow, and significant details of the methodology is missing, making the work difficult to understand. and Question 1: Could the authors describe and include some more details on the myopic MCES distance and how exactly it works? Currently, it is (in my opinion) impossible to understand the actual distance computation algorithm due to lack of explanations in Section 3.

Response: We want to make clear that this paper is not about the myopic MCES distance. For this, see Kretschmer et al., 2025, where this distance is introduced. Our splitting method requires a distance measure for molecular structures, and we chose MCES distances as a biologically intuitive measure. Our splitting method is however agnostic to the distance measure chosen. A detailed description of the myopic MCES distance is therefore out of scope of this paper.

Reviewer:

Additionally, I would strongly suggest making Section 3 more concise but with more details - I currently find it very difficult to read, primarily due to lack of details and the lengthiness with which many things are written. Overall I am left with a lack of understanding of how exactly the proposed method ends up actually splitting the data. An algorithm box for example could be very helpful.

Response: We fail to see how we can simultaneously be more concise and add more details, sorry. We would welcome specific pointers to the passages that were hardest to follow, so that a revision addresses the right ones. Following also suggestions from other reviews, we now added a figure illustrating our whole method.

Reviewer:

Critical Weakness 2: The proposed splitting method is proposed as if it will better match application settings, but this is not clear at all from the work, making it difficult/impossible to say if this work will make a difference in the claimed domains. Given that this is the most major contribution of such a method, I find it very difficult to recommend accepting the work in its current state. and Question 2: Could the authors provide evidence of why they think their methodology for splitting would result in more "correct" splits? My general issue with this claim is that it may be extremely application dependent what kind of folds are correct, so generally claiming this to be the case for a splitting method seems over the top.

Response: We agree that which folds are "correct" is application-dependent, and this is the paper's position rather than a claim against it: Sec. 4 states that the analysis is agnostic to the application domain and that the domain must be explicitly stated rather than assumed. We do not claim that one fold structure is correct in general. We claim that, once a domain is stated, the gap between a cross-validation split and that domain can be measured — and that established splits fail that test while MCES does not. We additionally added a new case study that makes this claim more concrete, see details in the reply to 4UF9

Reviewer:

To summarise my opinions on this work: Firstly, work is in my opinion very lengthy and difficult to follow at times, making it difficult to gauge exactly what the splitting method does (and by extension, how this affects the splits) - I think the overall presentation could be significantly improved.

Response: We would kindly ask the reviewer to give concrete examples that we should adjust. But to make it better to understand we added a figure illustrating our whole method.

Reviewer:

Secondly, I think there are a lot of strong claims around what the correct splitting method might be for different applications, that is not at all supported currently - e.g. this splitting method works on molecular graph structure alone (and this being number of deletable edges, and actually on bonds, not on neighbourhoods), but for some applications of molecules this may not be correct - e.g. this will (as far as I am aware) not capture functional similarities of molecules.

Response: This is correct, and we do not claim otherwise. MCES measures structural difference. Where a property is not determined by structure, no structural distance and no split can recover it, and this limits every structure-based model rather than our split in particular. We now added a sentence to make this more clear.

Reviewer:

My recommendation to the authors would be to change the narrative to the work, such that the method is a general graph splitting algorithm, and then perhaps illustrating applications (with concrete evidence) where such a graph splitting algorithm will lead to better generalisation estimates.

Response: We disagree. Our main contribution is the comparison between a given cross-validation split and the intended application domain; we use MCES for comparing molecule structures as this distance agrees more with chemical intuition than, for instance, Tanimoto coefficients (See for example Fig. 9 in Kretschmer et al., 2025 showcasing many instances of counterintuitive similarity values). The comparison is in no way tied to MCES: Tanimoto coefficients can be used alternatively, and we have now added a new figure showcasing this. The outcome is in fact very similar but remains better interpretable with MCES. We go into more details on this in our answer to 57h4. The method is not about graphs, and while the distance measure from the MCES-based splits is indeed a general graph-based measure, we are only confident that it is a very reasonable measure for molecular structures; we do not intend our method to be used outside this domain. We assess multiple other splits in our paper – scaffold, Butina, and UMAP – which are not graph-based.

Reviewer_42ekOpenReview ↗

Reply to Authors

Dear Authors,

Thank you for replying to my initial review. Below I elaborate on my concerns and respond to the rebuttal.

I accept the authors response the main contribution is not a general graph splitting algorithm. Instead, I understand the main claim (as I read the authors to also state) to be that MCES splitting provides better estimates of generalisation to the intended application domain.

However, I still remain unconvinced that this claim is sufficiently supported. As I previously argued, generalisation is application dependent, and closer alignment with the chosen biomolecule reference (in the original Figure 1) does not provide evidence for more accurate performance estimates for other applications. To really support such a statment, I would suggest evaluations on datasets with practitioner-defined test splits, such as the OpenADMET challenges. The authors could apply MCES and competing splitting methods within the training data, then assess which estimates best match performance on the unchanged test set.

I acknowledge the authors adding the external ESOL–AqSolDB experiment during rebuttal, which provides useful evidence. However, I am concerned as AqSolDB combines measurements from different sources, and the comparability of experimental conditions and label quality is not entirely clear. The external generelisation gap could reflect measurement differences as well as structural novelty. This does not invalidate the result, but in my opinion limits the conclusions that can be drawn from it.

I think there is a potentially strong contribution in this work, but in my opinion, the evidence currently remains insufficient to support it. This is my main concern and alone warrants maintaining my reject score. Separately, I believe the presentation needs substantial work, as discussed below.

With regards to writing, presentation and precision, I provide some examples of where I find the work needs significant improvement because the authors requested it. However, to be clear, I find this to be all throughout the work and these are just some illustrative example:

  1. "Importantly, our take is not that our folds are ‘harder’ than previous splitting methods — which turns out to be true — but rather, that we can, for the first time, measure how close our cross-validation results are in comparison to our desired domain of application […]”. This sentence is particularly long, and it combines motivation, an empirical result and a novelty claim all at the same time. Morever, the authors here (to me) blur the distinction between structural-distance distributions and comparing cross-validation results with application specific performance.
  2. "To that end, all weights of the foundation model have to be kept frozen (static embeddings).” Using “have to” in this context conflates something that authors have chosen to do (e.g. keeping weight fixed), with something that is a condition required for generalization. There are many examples of fine-tuning that have shown to improve generalisation for specific applications, again I refer to the OpenADMET challenges.
  3. "[…] given that we use �0��0� in our computations, this is irrelevant for all practical considerations.” I do not really see how the authors qualify the statement that this is irrelevant for all practical considerations? The authors report few inexact distances below the clustering threshold, but do not demonstrate whether these affect the resulting folds (currently we just have to take their word for it). This is of high importance to practitioners if they have to do these splits themselves.
  4. “Our evaluation of CLAMP suffers from data contamination […] consequently, our results for CLAMP should be taken with a grain of salt.” This is quite conversational and imprecise. Specifically, I am left wondering: What are the consequences of this contamination, and which conclusions in this work remain supported?
  5. “It is undisputed that out-of-distribution predictions are hard, and it is also undisputed that MCES distances are a good distance measure for chemical dissimilarity.” This is rhetorical rather than analytical, and it does not establish anything about how the proposed MCES-based splitting makes generalisation estimates better than other existing methods. In my opinion, for this sentence to substantiate the conversation around the proposed method, I believe it should state the relevant assumptions about the splitting strategy (e.g. why is it undisputed that MCES distances are good for chemical similarity besides that of similarities in bonds? Can I expect it to be a good measure for functional similarity?) and why does this make a good parallel for how we should expect out-of-distribution predictions to generalise?
Reviewer_4UF9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the gap between cross-validation results and intended application settings in small-molecule property prediction. The authors compare evaluation splits with explicitly defined application domains using distributions of minimum maximum common edge subgraph (MCES) distances from query molecules to training data. They further introduce an MCES-based splitting method combining single-linkage hierarchical clustering, cluster-size constraints, and greedy assignment to folds. Experiments on ten experimental MoleculeNet datasets evaluate five baseline models and nine foundation models using frozen representations with downstream prediction heads. Compared with uniform and scaffold splits, MCES splits generally provide greater structural separation and distance distributions closer to those of the chosen application domains, while yielding lower predictive performance. The results also show that many foundation models do not outperform strong fingerprint-based baselines under this protocol. Additionally, an audit of pretraining corpora reveals substantial structural overlap with downstream evaluation datasets for several models.

Critical strengths

The paper addresses a practically relevant question: to what extent do cross-validation results in small-molecule property prediction reflect generalization to intended applications? For experimental datasets with limited size and substantial structural clustering, splitting strategies can considerably affect reported performance and conclusions about model comparisons.

Critical weaknesses
  1. The validity of minimum MCES distance as a proxy for prediction difficulty remains insufficiently established. MCES distance primarily measures the number of structural edits without adequately reflecting their task-specific importance. Modifying an activity-determining functional group and changing a less consequential side chain may produce similar distances but substantially different property changes. Activity cliffs further illustrate that structural neighbors are not necessarily easy to predict. Consequently, matching nearest-distance distributions between cross-validation and an application domain does not necessarily imply comparable prediction difficulty, limiting the support this structural proxy provides for the paper’s central evaluation claims.
  2. The claimed improvement in evaluation realism lacks direct validation on labeled application-domain data. The paper uses biomolecules and ChEMBL molecules as structural references but does not report predictive performance on independent, labeled target-domain datasets corresponding to the downstream tasks. The experiments therefore demonstrate greater structural separation without directly establishing that MCES-based performance estimates are closer to actual application performance than estimates from alternative splits.
  3. Subdividing oversized clusters removes the guarantee of cross-fold distances being at least τ. Achieved separation consequently varies across datasets and α values, limiting a consistent interpretation of the threshold.
  4. It is hard to understand the method proposed, since there is no figure to help understand.
Questions for authors
  1. What task-specific evidence supports minimum MCES distance as a proxy for prediction difficulty, particularly for activity-cliff molecules? How do the authors assess whether similar distance distributions correspond to comparable prediction difficulty?
  2. Have the authors compared cross-validation estimates with measured performance on independent, labeled target-domain data? If not, what evidence distinguishes improved evaluation realism from simply increased test difficulty?
  3. After subdividing oversized clusters, what fraction of test molecules has a training neighbor at a distance below τ for each α? How sensitive are the conclusions to the chosen τ and α values?
2 comments

We thank the reviewer for the precise and constructive criticism.

Reviewer:

Critical Weakness 1: The validity of minimum MCES distance as a proxy for prediction difficulty remains insufficiently established. MCES distance primarily measures the number of structural edits without adequately reflecting their task-specific importance. Modifying an activity-determining functional group and changing a less consequential side chain may produce similar distances but substantially different property changes. Activity cliffs further illustrate that structural neighbors are not necessarily easy to predict. Consequently, matching nearest-distance distributions between cross-validation and an application domain does not necessarily imply comparable prediction difficulty, limiting the support this structural proxy provides for the paper’s central evaluation claims. and Question 1: What task-specific evidence supports minimum MCES distance as a proxy for prediction difficulty, particularly for activity-cliff molecules? How do the authors assess whether similar distance distributions correspond to comparable prediction difficulty?

Response: We agree that structural distance does not alone capture task-specific importance, but we would argue this limits every structure-based model rather than our split in particular: where structure does not determine the property, no distance and no split can recover it.

For activity cliffs specifically, the consequence runs the other way. A cliff pair A, A′ is at small MCES distance, so our method places both in the same fold — the model can learn the difference rather than being asked to extrapolate to it. Whether that is the better choice we cannot say with certainty, and we are not aware of evidence that settles it either way.

Reviewer:

Critical Weakness 2: The claimed improvement in evaluation realism lacks direct validation on labeled application-domain data. The paper uses biomolecules and ChEMBL molecules as structural references but does not report predictive performance on independent, labeled target-domain datasets corresponding to the downstream tasks. The experiments therefore demonstrate greater structural separation without directly establishing that MCES-based performance estimates are closer to actual application performance than estimates from alternative splits. and Question 2: Have the authors compared cross-validation estimates with measured performance on independent, labeled target-domain data? If not, what evidence distinguishes improved evaluation realism from simply increased test difficulty?

Response: We have now added a case study for this. We trained each baseline on the ESOL dataset and evaluated it on AqSolDB, an independent dataset for the same property, solubility. In detail, AqSolDB is an aggregation of nine solubility sources with ESOL being one of them. We removed all ESOL compounds by standardized SMILES, leaving 8856 measurements. Entries below are the cross-validation estimate minus the measured external RMSE, so a negative value means the estimate was too optimistic. We can see, that the conclusion still holds.

ModelExternalUniformScaffoldMCES α=0.3/0.4MCES α=0.5
XGB1.848−0.70−0.33−0.06+0.07
RF1.934−0.74−0.28−0.00+0.08
GCN2.009−0.56−0.38−0.39+0.06
kNN2.030−0.76−0.25+0.18+0.25
D-MPNN2.415−1.50−1.43−1.30−0.84

We additionally compared a D-MPNN with a hyperparameter search and early stopping. Here the hyperparameters are selected separately under each split, so the external error differs per split as well:

D-MPNN (tuned)UniformScaffoldMCES α=0.3/0.4MCES α=0.5
Cross-validation0.6270.7090.8671.016
External2.3842.4232.1472.200
Deviation−1.76−1.71−1.28−1.18

The ordering is unchanged, and the hyperparameters chosen under an MCES split transfer well, lowering the external error from 2.415 to 2.147, while selection under the uniform or scaffold split leaves it unchanged. This shows that hyperparameters chosen based on a split where the distributions match closer can improve predictions.

A split that was merely harder would overshoot: α=0.5 separates more than α=0.3 and already crosses to positive deviations for four of the five models. Increased difficulty and improved realism are therefore distinguishable. This remains a proxy, however, and we cannot simulate realism beyond it.

We would also be precise about the claim. We do not argue that MCES folds are realistic. We argue that the distribution test can falsify: a split whose distance distribution to its training data differs from that of the application domain and cannot be assumed to transfer. Uniform and scaffold fail that test; MCES does not. If further properties of the application domain become known, they can be added as further tests.

Reviewer:

Critical Weakness 3: Subdividing oversized clusters removes the guarantee of cross-fold distances being at least τ. Achieved separation consequently varies across datasets and α values, limiting a consistent interpretation of the threshold. and Question 3: After subdividing oversized clusters, what fraction of test molecules has a training neighbor at a distance below τ for each α? How sensitive are the conclusions to the chosen τ and α values?

Response: We agree that subdivision can break the guarantee, and we state this in Sec. 3.3. To make it explicit, we added a d<10 column to the fold-statistics tables, giving the fraction of test molecules below τ for every dataset and α. We report three values of α throughout. τ = 10 was fixed in advance from prior work (Bushuiev et al. 2024; Kretschmer et al. 2025) and not tuned. Changing τ would change how the single linkage tree would be built but not in a way that would change any conclusions. Changing alpha would follow the trend observable in Fig. 4 but would also make the splits less fair, e.g. an alpha of 0.8 would allow one fold to hold 80% of all data making a cross validation unfeasible hence we decided for 0.3-0.5.

Changing τ would change how the single linkage tree would be built but not in a way that would change any conclusions. Changing α would follow the trend observable in Fig. 4 but would also make the splits less fair e.g. an α of 0.8 would allow one fold to hold 80% of all data making a cross validation unfeasible hence we decided for 0.3-0.5. We added a small discussion on this to the text.

Reviewer:

Critical Weakness 4: It is hard to understand the method proposed, since there is no figure to help understand.

Response: Thank you for the suggestion, we now added a figure illustrating our splitting method to the paper.

Reviewer_LRP2OpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This article points out that the current test indicators are seriously inflated. It is believed that the random partition and backbone partition commonly used in current molecular machine learning suffer from severe data leakage, causing the models to perform well on the test set but having a reduced generalization ability in real drug development scenarios. Based on the maximum common edge subgraph chemical graph edit distance, the author proposed an evaluation method and clustering partition algorithm that can quantitatively measure the distribution of the distance between cross-validation partitions and the real application domain. Under the MCES partition that is closer to the real distribution, the performance of all models declines. Simple classical baselines outperform the performance of most complex pre-trained molecular large models. The author believes that the high performance of many existing large models actually stems from data contamination.

Critical strengths

The author's choice of the MCES distance is indeed very insightful. This is something I consider quite reasonable. Then, by introducing the maximum cluster ratio constraint, traversing the clustering tree diagram, and locally lowering the threshold to split large clusters that exceed the limit, a partitioning algorithm called MCESFolds was constructed.

Critical weaknesses

The author did not perform any hyperparameter tuning or early stopping, and also prohibited end-to-end fine-tuning. I believe this is unfair to deep learning algorithms and large models.

Questions for authors

In the process of drug discovery in reality, the basic model is almost always fine-tuned through downstream tasks. When fine-tuning is allowed, will the author's conclusion still hold? Can the author provide a small-scale fine-tuning experiment based on the MCES division and adopt an appropriate early stopping mechanism?

2 comments

We thank the reviewer for the careful reading and the positive assessment.

Reviewer:

The author did not perform any hyperparameter tuning or early stopping, and also prohibited end-to-end fine-tuning. I believe this is unfair to deep learning algorithms and large models.

Response: We wanted to answer one main question regarding current foundation models for molecular structures, which is whether these models truly provide foundational embeddings, i.e., embeddings that contain information about the underlying chemistry immediately useful for downstream tasks, such as prediction of toxicity or binding affinity. End-to-end training brings the risk of substantially overfitting to the specific downstream tasks, especially with the large number of trainable parameters in foundation models – compared to the often small downstream datasets. Fine-tuning yields better evaluation statistics; however, the resulting representations are very specific to the downstream task in question, and in all likelihood not foundational, showing overfitting but not generalization. Following Praski et al. 2025, we therefore chose to freeze the embeddings, allowing fine-tuning just for the three different “heads” (linear layer, MLP, Random Forest). We decided to prevent hyperparameter optimization and early stopping for the same reason. Furthermore, without hyperparameter search, which has to be performed for each model individually, results become directly comparable between the different splitting methods, with the split being the only variable.

Reviewer:

In the process of drug discovery in reality, the basic model is almost always fine-tuned through downstream tasks. When fine-tuning is allowed, will the author's conclusion still hold? Can the author provide a small-scale fine-tuning experiment based on the MCES division and adopt an appropriate early stopping mechanism?

Response: We realize that preventing fine-tuning can impede performance substantially. Especially for D-MPNN, we observe decreased performance compared to what has been reported in the literature. We therefore followed the reviewer’s suggestion to perform an additional experiment in order to verify that our conclusions still hold, performing a hyperparameter search and enabling early stopping for D-MPNN on the ESOL dataset, across all splits. This resulted in 125 trials per split. We have added the resulting table to the manuscript (shown below for convenience). Tuning lowers every estimate, but the ordering of the splits is unchanged. Additionally, a new external validation shows that the untuned D-MPNN is not disadvantaged by this protocol: it has the best cross-validation RMSE of all five baselines under every split, but the worst external RMSE. Details are in our reply to Reviewer 4UF9.

SplitTuned RMSEUntuned RMSE
uniform0.627 +- 0.0330.919
scaffold0.709 +- 0.0610.985
MCES α=0.3/0.40.867 +- 0.0721.115
MCES α=0.51.016 +- 0.1701.576

Table: Error values (RMSE) for D-MPNN evaluated on the ESOL dataset, with and without hyperparameter optimization and early stopping.

Thank you for your reply. You have solved all my problems. I will maintain a positive evaluation and increase my confidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.