- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces a new benchmark for chemical foundation models (CFMs), by combining twenty datasets spanning four categories. The data for this benchmark was created by collecting existing datasets. Data was selected when it fulfilled all requirements such as 'was data from a single experimental data source'. Additionally selected data was canonicalized, deduplicated and molecules below 150 Dalton(Da) or grater than 1000 Da where removed. The benchmark consists of two different variants being MoG-Bench-Medium Data(MD) and MoG-Bench-Low Data(LD). Those modes differ in split size, while MD considered most or all molecules, LD consists of a very small margin to simulate task with very little data.
Critical strengths
The issue that current evaluations lack good curation and standardization is valid - so the main question this paper tries to answer is important.
Critical weaknesses
Despite the relevant problem the manuscript tries to solve, we recommend rejection for the following reasons:
-
The methods are difficult to understand since a lot of details are missing for example the data selection only mentions the requirement of being one single data source, even though the text indicates more.The only thing described is the next step being the standardization and removal of duplicates. Also their splitting method is not described well enough, in the appendices is a short explanation on how clusters are created but not how those clusters end up in a split. Also there is "our MILP" mentioned that is never explained. Additionally the introduction and methods section mentioned molecules get either selected for a split or end up in holdout but how this mechanism works is not described.
-
Results are not conclusive enough. When comparing their provided splits to uniform or scaffold is it not enough to just show some example molecules. How does your split handle those examples? I would expect statistics such as how many molecules are very similar per split (e.g., how many molecules are above Tanimoto 0.8 when comparing test vs train splits). Additionally on those statistics I would like to see how the tested models compare those splits. Most readers would be familiar with performance on uniform or scaffold splits so directly comparing CFMs on all three would allow readers to compare those. Furthermore the paper compares the different models on both variants and reports one metric to show what model wins. But its not clear how this is chosen since all datasets are differently sized and have different metrics to be scored on how are those combined and how does a model win?
-
A lot of inconsistency. The main text is vague on explaining details and mostly refers to the appendix for more detail, but those seem to be incomplete or miss form the appendices entirely. Examples are:
- Line 207: "Those additional steps are listed in the Appendix" --- Appendix does list the steps of the computational quality control but still leaves the question open what the manual curation did. The paragraph itself mentioned that datasets or tasks that failed manual curation standards where discarded. But nowhere are statistics on how many datasets this includes and also not what does standards are other than the already mentioned "must come from one source" rule. I think it would be a real contribution to show how many public datasets where not considered due to any errors since other readers might use them without knowing that there might be something wrong.
- Line 261: "More details on MBDSplitter can be found in the Appendix." -- Appendix give some additional information but not for the splitter itself just how clusters are build. The appendix itself claims to have further information in the appendix (see line 833).
- Line 347-350: They evaluated Mol-JEPA and CLAMP, but excluded them due to overlap of training data and evaluation and that results can be found in the appendix. This section seems to be missing completely and I can't find it in the supplementary zip either.
- Line 358: "Further details on benchmarking setup can be found in the Appendix" -- This also seems to be missing completely.
- Line 464: "We observe a drop in performance from Medium Data to Low Data (see Appendix);.." -- There are no performance stats in the appendices.
- I would suggest to proofread the manuscript (for example section 2.4) again.
- Also the AI statement is missing
Questions for authors
- What does "carefully curated and reviewed for real world relevance" mean? (E.g., what standards, what abnormalities, failing conditions)
- How do contribution 2 and 3 differ?
- What is holdout and how does a molecule end up there? (contribution 3 says MoG-Bench-MD aims to minimizing holdout but holdout is never mentioned again)
- Is Table 1 only showing the MD or LD split numbers?
- Why are the Train/Test numbers median over three folds?
- Are all task weighted the same or is a task with 50 samples the same importance as one with 6000 samples?
- What is the avg or median similarity between test/train for each split?
- How often can the issue of figure 2 be observed?
- What constraints does MBDSplitter place within the test?
- How does MBDSplitter prevent one single large cluster?
- Why Tanimoto 0.4?
- What is the label diversity of the datasets and how does it change given your splits?
- How is diversity measured?
- Why not use your split for the other benchmark datasets that fail do achieve diverse tasks?
- What head was used in Figure 3?
- What was the std between the 5 seeds?
- What are the hyperparams for the heads?
- How did you measure overfitting?
- How was the "finetuning" done?
- Figure 4, what is the dashed line?
- Figure 5 how was the improvement measured?
- How many CFMs can I stack until the performance reaches a plateau?