MoG-Bench: Robust Evaluations for Molecular Foundation Models

Molecular foundation models provide general purpose molecular representations that can be used to predict many different chemical properties. Because these properties are often slow and costly to measure, researchers are most interested in models that make accurate predictions for new chemically distinct compounds and can perform well on limited data. However, existing molecular property prediction benchmarks were not designed to evaluate models under these constraints. We introduce the Molecular Generalization Benchmark (MoG-Bench), containing 20 diverse molecular property prediction tasks organized into 4 categories: binding affinity prediction; functional activity prediction; absorption, distribution, metabolism, and excretion (ADME); and hit screening. We make the following contributions: 1) for every dataset in MoG-Bench, we ensure rigorous quality control and a thorough, predefined train and test split that estimates models' ability to accurately predict properties of chemically distinct compounds; 2) two versions of MoG-Bench, MoG-Bench Medium Data (MD) and MoG-Bench Low Data (LD), evaluating model performance on standard supervised learning and few-shot learning; 3) benchmarking eight state-of-the-art foundation models, providing practical guidance on how to improve the accuracy of these models, and takeaways to guide future model development.