acceptodds
Under review as a conference paper at ICLR 2027

Simple Models Can Still Compete in Out-of-Distribution Molecular Property Prediction

Abstract

This study began as a replication of BOOM, a recent benchmark for out-of-distribution (OOD) molecular property prediction (Antoniuk et al., 2025), and developed into a broader investigation of how model choice, evaluation design, and metrics affect conclusions about molecular OOD performance. We reproduce and extend BOOM using six models spanning linear, tree-based, message-passing, pretrained language, and 3D-equivariant approaches. A simple Elastic Net model based on 2D molecular descriptors performs competitively with Chemprop on the original property-based OOD tasks, showing that increased model complexity does not necessarily translate into better extrapolation. This conclusion remains qualitatively robust across repeated runs and after validation-based hyperparameter tuning of Chemprop. In contrast, Random Forest and XGBoost perform poorly on these tasks, illustrating that the ability to extrapolate beyond the training target range, rather than model complexity alone, strongly influences performance. We further show that BOOM's original OOD evaluation measures extrapolation to extreme property values rather than structural novelty. When we introduce structure-based OOD splits, all evaluated models perform substantially better and differences between them narrow. Finally, we identify a metric discrepancy in the original benchmark: values reported as correspond to squared Pearson correlation (). Using the coefficient of determination instead reveals many strongly negative OOD scores and a more severe extrapolation problem. Together, these results show that simple models remain important baselines for molecular property prediction and that conclusions about OOD performance depend critically on the type of distribution shift, the models included, and the evaluation metric.

Primary area:
datasets and benchmarks
Keywords:
molecular property prediction, out-of-distribution generalization, QSPR, molecular machine learning, simple baselines, graph neural networks, extrapolation, benchmark evaluation, reproducibility, applicability domain

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LKRWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits the BOOM benchmark for out-of-distribution molecular property prediction, investigating how model selection, OOD split design, and evaluation metrics affect benchmarking conclusions. The authors compare six models across ten molecular property prediction tasks, introduce a structure-based OOD split using UMAP and HDBSCAN, and examine the discrepancy between the coefficient of determination (R²) and squared Pearson correlation (ρ²). The results suggest that Elastic Net can remain competitive with more complex models, that property-based and structure-based OOD splits produce substantially different performance patterns, and that the original BOOM evaluation may underestimate the severity of extrapolation errors due to its metric choice.

Critical strengths
  • Useful benchmark reassessment. The identification of the R²/ρ² discrepancy is valuable and highlights the importance of using appropriate evaluation metrics for molecular OOD prediction.

  • Reasonable robustness checks. The authors include repeated-seed experiments and validation-based hyperparameter tuning, while acknowledging several limitations of their reproduction and experimental setup.

Critical weaknesses
  • Limited novelty and significance of the contributions. The paper primarily revisits an existing benchmark rather than introducing a new methodology, evaluation framework, or substantially new understanding of OOD generalization. Several central observations, including the competitiveness of simple models and the dependence of model performance on the type of distribution shift, are already well established. While the metric discrepancy is a useful finding, it primarily changes the interpretation of absolute performance rather than the main conclusions or model rankings of BOOM. Overall, the incremental insights appear insufficiently significant for ICLR.

  • Insufficient validation of the structural OOD evaluation. The structural OOD analysis relies on a single heuristic UMAP/HDBSCAN split, without quantitatively establishing the degree of structural novelty or systematically comparing against established splitting strategies. The substantially higher performance under this split may reflect its particular construction and difficulty rather than a generalizable distinction between structural and property-based OOD generalization. Moreover, differences in test-set composition and metric aggregation further limit direct comparisons between the two settings.

Questions for authors
  • Could the authors clarify which findings provide fundamentally new insights beyond the original BOOM benchmark and existing studies on classical baselines and molecular OOD evaluation?

  • Can the authors provide quantitative evidence, potentially from existing experiments, that the proposed structural split introduces meaningful molecular novelty? How much of the performance difference could be explained by differences in split difficulty or target distributions?

  • To what extent do the discrepancies in the reproduced GotenNet and MolFormer results affect the reliability of the cross-model conclusions?

Reviewer_E7pTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors present a reproduction and extension of the BOOM benchmark for out-of-distribution molecular property prediction. They investigate three aspects of the benchmark. (1) They identify a discrepancy in the original evaluation metric, (2) they introduce a structure-based OOD split, and (3) they include two additional baselines, Elastic Net and XGBoost. The authors find that a simple Elastic Net model can perform competitively with more complex models such as Chemprop on the original property-based OOD splits, although other simple models, including Random Forest and XGBoost, perform poorly. Under the structure-based split, all evaluated models perform relatively well, with little difference in performance.

Critical strengths

Understanding how molecular property prediction models generalize to unseen distributions is an important problem and it is important to correct and expand existing benchmarks to do a better job of measuring generalizability and reflect more current methods.

Critical weaknesses

My main concern is the limited novelty of the work. One of the central conclusions is that simpler models can perform competitively with substantially more complex models, including under out-of-distribution evaluation. This observation is already well established in molecular machine learning, with several existing studies demonstrating the competitiveness of classical models against more complex approaches. Similarly, structure-aware splitting strategies, including UMAP-based molecular clustering, have already been explored in the literature. Consequently, I do not see a substantially novel scientific finding emerging from these experiments.

More broadly, the paper largely consists of a re-evaluation of an existing benchmark through a correction to the reported evaluation metric, the inclusion of two additional baseline models, and the introduction of another structural split based on existing approaches. While these are useful contributions for maintaining and improving the BOOM benchmark, I do not believe they rise to the level of a sufficiently novel machine learning contribution for ICLR. The metric discrepancy is worth correcting, but the authors themselves acknowledge that this primarily changes the magnitude and interpretation of OOD performance rather than the original benchmark's central conclusions.

These contributions seem more appropriate as an update to the existing BOOM benchmark and its accompanying repository than as a standalone research paper. Overall, I do not see sufficient novelty in either the methodological approach or the empirical findings to justify acceptance, and I recommend rejection.

Questions for authors

No questions

Reviewer_go2QOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes BOOM, a benchmark for OOD accuracy on multiple chemical property predictors. They analyze a number of simpler models for OOD performance, following BOOM. They also introduce a different, structure-based definition of OOD, and find that it results in better OOD performance. They also argue that the BOOM R^2 is incorrect, and show results using what they believe to be the correct definition.

Critical strengths

This paper's critique of R^2 in BOOM seems to be well-founded, though I don't believe that it changes BOOM's central claims.

The introduction of a structure-based definition of OOD is a step in the right direction, though I think more research will be needed to fully understand OOD in chemical space.

Critical weaknesses

The OOD validation is too weak. How do we know that the held-out molecules do not have significant overlap with the training set in chemical space? The use of UMAP also does not seem to be very justified. There are other clustering methods that could work directly in the feature space. Even HDBSCAN could be used on the Tanimoto distance.

The model comparisons are incomplete. The GotenNet and Molformer don’t reproduce the

Questions for authors

What are other ways of defining distance in the chemical space, and how do they compare

Why are results not replicated for GotenNet and Molforer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.