acceptodds
Withdrawn from ICLR 2027

MoG-Bench: Robust Evaluations for Molecular Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Molecular foundation models provide general purpose molecular representations that can be used to predict many different chemical properties. Because these properties are often slow and costly to measure, researchers are most interested in models that make accurate predictions for new chemically distinct compounds and can perform well on limited data. However, existing molecular property prediction benchmarks were not designed to evaluate models under these constraints. We introduce the Molecular Generalization Benchmark (MoG-Bench), containing 20 diverse molecular property prediction tasks organized into 4 categories: binding affinity prediction; functional activity prediction; absorption, distribution, metabolism, and excretion (ADME); and hit screening. We make the following contributions: 1) for every dataset in MoG-Bench, we ensure rigorous quality control and a thorough, predefined train and test split that estimates models' ability to accurately predict properties of chemically distinct compounds; 2) two versions of MoG-Bench, MoG-Bench Medium Data (MD) and MoG-Bench Low Data (LD), evaluating model performance on standard supervised learning and few-shot learning; 3) benchmarking eight state-of-the-art foundation models, providing practical guidance on how to improve the accuracy of these models, and takeaways to guide future model development.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
molecular foundation models, computational drug discovery, molecular property prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Q4QROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a new benchmark for chemical foundation models (CFMs), by combining twenty datasets spanning four categories. The data for this benchmark was created by collecting existing datasets. Data was selected when it fulfilled all requirements such as 'was data from a single experimental data source'. Additionally selected data was canonicalized, deduplicated and molecules below 150 Dalton(Da) or grater than 1000 Da where removed. The benchmark consists of two different variants being MoG-Bench-Medium Data(MD) and MoG-Bench-Low Data(LD). Those modes differ in split size, while MD considered most or all molecules, LD consists of a very small margin to simulate task with very little data.

Critical strengths

The issue that current evaluations lack good curation and standardization is valid - so the main question this paper tries to answer is important.

Critical weaknesses

Despite the relevant problem the manuscript tries to solve, we recommend rejection for the following reasons:

  1. The methods are difficult to understand since a lot of details are missing for example the data selection only mentions the requirement of being one single data source, even though the text indicates more.The only thing described is the next step being the standardization and removal of duplicates. Also their splitting method is not described well enough, in the appendices is a short explanation on how clusters are created but not how those clusters end up in a split. Also there is "our MILP" mentioned that is never explained. Additionally the introduction and methods section mentioned molecules get either selected for a split or end up in holdout but how this mechanism works is not described.

  2. Results are not conclusive enough. When comparing their provided splits to uniform or scaffold is it not enough to just show some example molecules. How does your split handle those examples? I would expect statistics such as how many molecules are very similar per split (e.g., how many molecules are above Tanimoto 0.8 when comparing test vs train splits). Additionally on those statistics I would like to see how the tested models compare those splits. Most readers would be familiar with performance on uniform or scaffold splits so directly comparing CFMs on all three would allow readers to compare those. Furthermore the paper compares the different models on both variants and reports one metric to show what model wins. But its not clear how this is chosen since all datasets are differently sized and have different metrics to be scored on how are those combined and how does a model win?

  3. A lot of inconsistency. The main text is vague on explaining details and mostly refers to the appendix for more detail, but those seem to be incomplete or miss form the appendices entirely. Examples are:

  • Line 207: "Those additional steps are listed in the Appendix" --- Appendix does list the steps of the computational quality control but still leaves the question open what the manual curation did. The paragraph itself mentioned that datasets or tasks that failed manual curation standards where discarded. But nowhere are statistics on how many datasets this includes and also not what does standards are other than the already mentioned "must come from one source" rule. I think it would be a real contribution to show how many public datasets where not considered due to any errors since other readers might use them without knowing that there might be something wrong.
  • Line 261: "More details on MBDSplitter can be found in the Appendix." -- Appendix give some additional information but not for the splitter itself just how clusters are build. The appendix itself claims to have further information in the appendix (see line 833).
  • Line 347-350: They evaluated Mol-JEPA and CLAMP, but excluded them due to overlap of training data and evaluation and that results can be found in the appendix. This section seems to be missing completely and I can't find it in the supplementary zip either.
  • Line 358: "Further details on benchmarking setup can be found in the Appendix" -- This also seems to be missing completely.
  • Line 464: "We observe a drop in performance from Medium Data to Low Data (see Appendix);.." -- There are no performance stats in the appendices.
  • I would suggest to proofread the manuscript (for example section 2.4) again.
  • Also the AI statement is missing
Questions for authors
  • What does "carefully curated and reviewed for real world relevance" mean? (E.g., what standards, what abnormalities, failing conditions)
  • How do contribution 2 and 3 differ?
  • What is holdout and how does a molecule end up there? (contribution 3 says MoG-Bench-MD aims to minimizing holdout but holdout is never mentioned again)
  • Is Table 1 only showing the MD or LD split numbers?
  • Why are the Train/Test numbers median over three folds?
  • Are all task weighted the same or is a task with 50 samples the same importance as one with 6000 samples?
  • What is the avg or median similarity between test/train for each split?
  • How often can the issue of figure 2 be observed?
  • What constraints does MBDSplitter place within the test?
  • How does MBDSplitter prevent one single large cluster?
  • Why Tanimoto 0.4?
  • What is the label diversity of the datasets and how does it change given your splits?
  • How is diversity measured?
  • Why not use your split for the other benchmark datasets that fail do achieve diverse tasks?
  • What head was used in Figure 3?
  • What was the std between the 5 seeds?
  • What are the hyperparams for the heads?
  • How did you measure overfitting?
  • How was the "finetuning" done?
  • Figure 4, what is the dashed line?
  • Figure 5 how was the improvement measured?
  • How many CFMs can I stack until the performance reaches a plateau?
Reviewer_FD4ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MoG-Bench, a collection of 20 molecular property prediction tasks covering ADMET, binding affinity, functional activity, and screening. The benchmark emphasizes single-assay curation and a diversity-aware splitting procedure, MBDSplitter, intended to limit train–test chemical similarity while maintaining diversity within test sets. It includes medium-data (MD) and low-data (LD) settings and evaluates six chemical foundation models using frozen representations with different prediction heads, as well as finetuning. The authors report advantages for linear probes and model ensembling, and use a simple molecular-descriptor baseline to investigate low-data limitations. The intended contribution is a more informative evaluation resource for molecular generalization, together with practical guidance for using pretrained models.

Critical strengths
  1. A useful and practically motivated evaluation resource.

I appreciate the effort to curate experimental tasks across four relevant categories (Table 1, Section 2.1). The emphasis on single-assay sources is well motivated and may reduce heterogeneity arising from different experimental protocols. A reliable benchmark with this scope could help researchers choose or develop molecular representations for experimental property prediction.

  1. A thoughtful focus on test-set diversity.

Considering both train–test separation and diversity within the test set addresses a meaningful evaluation concern. Performance on one closely related chemical series may provide a limited picture of generalization, even when train–test similarity is controlled (Section 2.3). The supplied task data and detailed result files are also helpful, since they allow the reported findings to be examined beyond aggregate figures.

Critical weaknesses
  1. The main evaluation protocol is not yet sufficiently reproducible from the supplied materials.

Table 1's reported split sizes are consistent with the mbd2 results, whereas the supplied molecular assignment files are documented as cluster_kfold_diverse (task_data/README.md, line 3). The difference is substantial for some tasks. The median LogD training size is 6,007 under mbd2 but 67 under the supplied assignments; for Efflux, the corresponding sizes are 3,002 and 100. It is therefore unclear how the released assignments relate to the main reported results.

The supplied mbds.py also explicitly states that it does not implement the canonical separator search, component holdout, or cluster-diversity objective, and cannot regenerate the published assignments (lines 4–7). Appendix 5.4 defines the diversity measures, including LCP and normalized cluster entropy, but does not fully specify the optimization procedure. These may be versioning or packaging issues that the authors can resolve. They do not by themselves establish that the reported results are incorrect. However, the benchmark protocol is a central contribution, so I would need the actual splits and a complete description or implementation to assess it confidently.

  1. The practical recommendations need clearer supporting evidence and a more carefully delimited interpretation.

Figure 4 includes finetuning, but I could not find corresponding per-task finetuning results in the supplied result CSVs. The checkpoint choices, tuning settings, and aggregation of win rates and error bars are also insufficiently documented. This makes it difficult to assess the recommendation favoring linear probes (Section 3.2). The missing artifacts concern verification of the comparison, rather than evidence that finetuning was not performed.

MD and LD also use different test sets, so their performance difference does not isolate the effect of training-set size (Sections 2.4, 3.4). Some LD result records use 14–23 training molecules rather than the stated 24, which needs explanation when interpreting the few-shot setting. Finally, Figure 6's legend labels the comparator as the best foundation model, while the caption and text describe an average model. These points do not necessarily invalidate the observed results, but they limit the conclusions about preferred adaptation methods and low-data behavior.

I lean toward weak rejection because the protocol and supporting evidence need to be made more readily verifiable. Clarification and the corresponding artifacts could change my recommendation.

Questions for authors

The following questions concern the submitted evaluation and could materially affect my assessment. Clarification and existing artifacts would be especially helpful.

  1. Could you explain the relationship between cluster_kfold_diverse, mbd2, and ld_v2, and provide the molecular assignments actually used for the main MD and LD results, including inner validation folds and LD support sets? A complete description or implementation of the canonical MBDSplitter, together with per-task train–test similarity, test-set LCP, and data-retention statistics, would help establish that the intended constraints are satisfied.

  2. Could you provide the per-task finetuning results and the checkpoint, hyperparameter-selection, and early-stopping settings used in the comparison? Please also define the comparison unit for the win rates and the source of variation represented by the error bars in Figure 4. These details are needed to evaluate the recommendation favoring linear probes.

  3. Could you clarify how LD support sets are selected, including class balance for screening tasks and cases with fewer than 24 training molecules? For interpreting the MD–LD difference, is a comparison on a common test set available? Otherwise, the conclusions should be limited to the two evaluated settings. Please also reconcile the average-versus-best-model description in Figure 6.

Reviewer_QJiROpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MoG-Bench, a benchmark for evaluating the generalization capabilities of Chemical Foundation Models (CFMs). It consists of 20 curated experimental datasets across four categories: ADMET, binding affinity, functional activity, and screening.  The authors propose MBDSplitter to reduce train-test structural similarity while promoting chemical diversity within the test set. They also introduce Medium-Data (MD) and Low-Data (LD) settings, with the latter using only 24 labeled training compounds.  Experiments on six foundation models suggest that linear probes generally outperform more complex prediction heads and fine-tuning. A simple descriptor-based Ridge baseline also shows strong performance on the four regression tasks evaluated in the LD setting.

Critical strengths

S1. Practical relevance and comprehensive evaluation  The benchmark covers 20 curated experimental datasets across diverse molecular property prediction tasks. The inclusion of both MD and LD settings provides a practically relevant evaluation of foundation models under different levels of supervision.  S2. Chemically diverse generalization evaluation  MBDSplitter considers both train-test structural separation and test-set chemical diversity, providing a more challenging and potentially informative setting for evaluating generalization to chemically distinct compounds.

Critical weaknesses

W1. Insufficient robustness analysis of LD splits  The LD setting uses only 24 training compounds, but the precise selection procedure is not sufficiently described. Performance may be sensitive to sample composition and label distribution. Although the paper reports fold-based evaluation with five random seeds per fold, it is unclear whether these seeds correspond to different data splits, model training, or both. Further analysis of performance variability across independently generated LD splits is needed.  W2. Insufficient fine-tuning details  Although the authors state that all six foundation models are fine-tuned, the experimental configurations are not described in sufficient detail. For Uni-Mol2 in particular, key information regarding its checkpoint, model size, fine-tuning hyperparameters, and task-level results is not clearly provided. This limits reproducibility and makes it difficult to assess the conclusion that linear probing generally outperforms fine-tuning.  W3. Limited conventional baselines  The descriptor-based Ridge baseline in Figure 6 covers only four regression tasks. Although Random Forest and XGBoost prediction heads are evaluated on frozen foundation-model embeddings, these do not replace conventional baselines trained directly on molecular features. Stronger conventional baselines, such as Morgan fingerprints with Random Forest or XGBoost, should be evaluated across all 20 tasks, especially in the LD setting.  W4. Statistical uncertainty and reporting inconsistencies  Some test sets are very small (e.g., median test-set sizes of 33 for LogD and 32 for liver intrinsic clearance), raising concerns about the reliability of task-level performance estimates. Although several figures include error bars, their interpretation and the statistical uncertainty of individual tasks are not sufficiently clear. Reporting task-level confidence intervals would help. Additionally, Section 3.3 mentions eight-model ensembles, whereas Figure 5 shows ensembles of up to six models. Figure 6 also contains inconsistent descriptions of the "best" versus "average" foundation model between its legend, caption, and main text.

Questions for authors

Q1. How exactly are the 24 LD training compounds selected, and how is label distribution handled? Do the five random seeds correspond to data splitting, model training, or both?  Q2. How sensitive are LD results to independently generated data splits? Could the authors report performance variability across independent split seeds, beyond the currently reported fold-based evaluations?  Q3. Which Uni-Mol2 checkpoint and model size were used? What fine-tuning hyperparameters were adopted? Could the authors provide task-level results comparing frozen linear probing and fine-tuning?  Q4. How do the foundation models compare with strong conventional molecular prediction baselines, such as Morgan fingerprints with Random Forest or XGBoost, across all 20 tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.