Mole-PAIR: Improving Reliability of Molecular Foundation Models with Preference Optimization

Molecular foundation models pre-trained on large chemical corpora have become powerful tools for scientific discovery, yet their behavior can be unpredictable when inputs deviate from the pre-training distribution, which severely hinders deployment in safety-critical settings such as biomedical discovery and materials design. To mitigate this issue, we propose Molecular Preference-Aligned Instance Ranking (Mole-PAIR), a plug-and-play module that can be attached to existing foundation models and improve their reliability w.r.t. out-of-distribution (OOD) data through cost-effective post-hoc preference optimization on frozen representations. The key idea of Mole-PAIR is to formulate OOD detection as a preference optimization problem whose objective aims at optimizing the model's “preferences” on in-distribution (ID) samples over OOD ones. Our analysis shows that this new objective adaptively prioritizes hard pairs throughout the training dynamics, which facilitates fast learning in practice. We demonstrate the practical efficacy of this approach with recently proposed molecular foundation models and challenging OOD benchmarks, achieving up to 9.7% improvement of AUROC and 48.5% reduction of FPR95 over powerful OOD detection approaches under shifts of scaffold, assay and molecular size.