- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Mole-PAIR, a simple framework to solve the OOD test issue and improve the reliability of the molecular foundation models by OOD detection. It proposes using preference optimization, in other words, a pairwise loss, to determine whether the evaluation data is OOD or ID. This method improves AUROC and FPR95 under some OOD-setting datasets such as DrugOOD.
Critical strengths
- The topic of OOD detection for molecular foundation models is critical and useful.
- The proposed method is simple and efficient. The framework only trains a lightweight MLP on frozen molecular representations, making it easy to apply to existing foundation models.
- Mole-PAIR achieves better OOD detection performance than most baselines across DrugOOD and GOOD, including methods using the same auxiliary OOD data.
Critical weaknesses
-
The contribution of this work is trivial. The main technique is a standard pairwise ranking approach applied to OOD detection, using a frozen molecular encoder and a lightweight MLP. The proposed loss is essentially a pairwise logistic ranking loss, and the hard-pair emphasis discussed in the theoretical analysis is also a natural property of this loss. The pairwise loss has already been commonly used in current molecular and drug discovery research. The overall workflow is also simple. Some other experiments in the main paper are parameter tuning. Therefore, it is not very convincing regarding the main methodological novelty and the contribution of this work.
-
The paper claims that the method does not require prior knowledge of the test-time OOD distributions. However, the main experiments use auxiliary OOD samples from the corresponding OOD-validation split for each distribution shift. Although the validation and test domains are different, the auxiliary OOD data are still selected based on the same shift type as the test data. It largely diminishes the usage of this work. Although Table 9 provides some cross-shift evaluations, all transferred models are tested on Size shifts, which are easy to distinguish. It is still unclear whether the method can generalize to more challenging unseen OOD types, such as Scaffold or Assay shifts, without using shift-specific auxiliary OOD samples.
-
Since the method already uses ID and auxiliary OOD samples for training, a simple, straightforward binary classifier would be a natural baseline. Although Appendix D6 compares pairwise ranking with BCE and MSE, it only provides a score distribution visualization on one dataset, without quantitative results. Also, Appendix D6 shows a pairwise hinge loss, different from the pairwise logistic loss in Eq.6. A more comprehensive comparison is needed to demonstrate the advantage of the proposed preference optimization.
-
Some results appear inconsistent. For example, MiniMol on EC50-Assay reports an AUROC of 0.743 in Table 4 but 0.711 in Table 7. Similarly, Uni-Mol on EC50-Scaffold reports 0.976 in Table 4 but 0.965 in Table 12.
Questions for authors
If pairwise loss is the key contribution which is better than the simple binary classification, it is necessary to use comprehensive experiments or analysis to prove it.
6 comments
Response to Reviewer Reviewer z1tf (2)
W3: "Since the method already uses ID and auxiliary OOD samples for training, a simple, straightforward binary classifier would be a natural baseline. Although Appendix D6 compares pairwise ranking with BCE and MSE, it only provides a score distribution visualization on one dataset, without quantitative results. Also, Appendix D6 shows a pairwise hinge loss, different from the pairwise logistic loss in Eq.6. A more comprehensive comparison is needed to demonstrate the advantage of the proposed preference optimization."
Q1: "If pairwise loss is the key contribution which is better than the simple binary classification, it is necessary to use comprehensive experiments or analysis to prove it."
Thanks for suggesting this comparison, which improves our evaluation. We add a binary cross-entropy (BCE) detector that uses the same frozen embeddings and the same scoring head as Mole-PAIR. ID molecules have target 0 and auxiliary OOD molecules have target 1.
Comparison under different auxiliary-data budgets. We evaluate on the challenging DrugOOD EC50-Assay shift. All methods use the same nested auxiliary-molecule lists, the same frozen MiniMol embeddings, the same validation and test sets, and a common full-batch training protocol. Results are AUROC (mean ± SD over five seeds), reported in Appendix D.6:
| Auxiliary OOD molecules | Mole-PAIR | BCE | Best-OE Baseline |
|---|---|---|---|
| 200 | 0.683 ± 0.002 | 0.634 ± 0.005 | 0.596 ± 0.013 |
| 40 | 0.625 ± 0.005 | 0.527 ± 0.012 | 0.553 ± 0.011 |
| 20 | 0.617 ± 0.007 | 0.531 ± 0.015 | 0.562 ± 0.020 |
Mole-PAIR exceeds BCE by 0.049, 0.098, and 0.086 AUROC with 200, 40, and 20 auxiliary molecules.
Robustness to the choice of auxiliary molecules. We repeat the 20-molecule setting with four independently sampled auxiliary sets and five initialization seeds each. Mole-PAIR obtains 0.580 ± 0.046 AUROC, compared with 0.536 ± 0.042 for BCE and 0.552 ± 0.023 for MSP-OE, the base-OE baseline.
W2: "The paper claims that the method does not require prior knowledge of the test-time OOD distributions. However, the main experiments use auxiliary OOD samples from the corresponding OOD-validation split for each distribution shift. It is still unclear whether the method can generalize to more challenging unseen OOD types, such as Scaffold or Assay shifts, without using shift-specific auxiliary OOD samples."
Thank you for the review. We agree that the original Table 9 was incomplete, We supplement two new experiments.
Generic public background without shift-specific selection. We replace the benchmark OOD-validation pool with molecules sampled from PubChem without using the target shift type. We exclude structures appearing in any of the six DrugOOD subsets. This gives 2,000 training and 600 validation molecules. The ID data, frozen MiniMol encoder, architecture, and test sets are unchanged.
| Target shift | Official auxiliary pool | PubChem background | Best ID-only baseline |
|---|---|---|---|
| EC50-Scaffold | 0.970 ± 0.000 | 0.914 ± 0.001 | 0.685 |
| EC50-Assay | 0.711 ± 0.002 | 0.602 ± 0.003 | 0.572 |
We found that a generic public background provides substantial value for the structural shift (0.914 vs. 0.685 for the best ID-only baseline).
Transfer to Scaffold targets. Models are trained and selected using source-domain data only, then evaluated once on the target test set (1,000 ID and 1,000 OOD molecules). The target-trained model is a separate reference with access to target-domain training and validation data. Full results are in Table 10, Appendix D.5.
| Source → target | Transferred Mole-PAIR | Target-trained Mole-PAIR |
|---|---|---|
| EC50-Size → IC50-Scaffold | 0.909 ± 0.001 | 0.982 ± 0.001 |
The direction retains strong detection without target-specific auxiliary data. We thank you again for pointing out this.
Dear Reviewer Z1tf,
Thank you for your review.
We have added all the experiments and discussions that you have asked for in your rebuttal. The discussion phase will end soon and we are hoping to hear your thoughts.
Regards,
Authors
Response to Reviewer Reviewer z1tf (1)
Thank you for recognizing the importance of molecular OOD detection, the efficiency of our framework, and its performance against exposure-based baselines. We have added the requested binary-classification controls, challenging unseen-shift experiments, and a correction of the inconsistent results.
We thank the reviewer for the review. We respectfully disagree with this interpretation of our contributions. While pairwise ranking is an established technique, our contribution is not merely applying an existing loss or tuning its parameters. We address a distinct problem in molecular learning: equipping pretrained molecular foundation models with the ability to identify unfamiliar molecular inputs without downstream property labels or backbone retraining. We highlight three aspects:
A new problem formulation for molecular foundation models. Existing molecular ranking approaches primarily learn preferences over molecular properties, compound activities, or medicinal chemistry choices [1]. In contrast, Mole-PAIR formulates OOD detection as learning the relative preference between ID and auxiliary background molecules. This provides a dedicated OOD affinity score independent of downstream property prediction. (Section 3.1)
A lightweight and task-agnostic detection framework. Rather than deriving OOD scores from a supervised property classifier, Mole-PAIR directly learns a scoring head on frozen molecular representations. This design requires neither molecular property labels nor classifier logits and applies to both 2D and 3D foundation models. Our controlled experiments further demonstrate its practical effectiveness over OE classifier baselines under matched auxiliary-data budgets, particularly on challenging Assay shifts. (Table 1, Requirements columns; Table 4, best AUROC in 11 of 12 cases per backbone; Appendix D.6)
Theoretical characterization and empirical significance. We establish the connection between the pairwise objective and Bayes-optimal AUROC ranking under the stated population assumptions, and characterize its adaptive emphasis on difficult pairs during optimization. While the underlying ranking principle is established, these analyses provide a principled understanding of the objective in our molecular OOD setting. Moreover, our extensive experiments validate its effectiveness across multiple molecular foundation models, datasets, and distribution shifts. (Proposition 5.1, Lemma 5.2, Proposition 5.3, Section 5)
Importantly, methodological significance should not be determined solely by whether an underlying optimization primitive is new. Our contribution lies in identifying and validating a practical, theoretically motivated approach to molecular foundation-model reliability, rather than proposing a new generic ranking loss. The hyperparameter analyses are supplementary evaluations, not the central contribution. Section 4.2 is now renamed "Sensitivity Analysis", with the two scans combined into Figure 2 and the joint grid moved to Appendix D.2 (Figure 5).
References:
[1] Extracting Medicinal Chemistry Intuition via Preference Machine Learning, Nature Communications 2023, https://doi.org/10.1038/s41467-023-42242-1
Thank you for carefully checking our results and identifying these inconsistencies. We have reconciled the affected results, and aligned the related tables and summary averages accordingly.