- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper revisits the BOOM benchmark for out-of-distribution molecular property prediction, investigating how model selection, OOD split design, and evaluation metrics affect benchmarking conclusions. The authors compare six models across ten molecular property prediction tasks, introduce a structure-based OOD split using UMAP and HDBSCAN, and examine the discrepancy between the coefficient of determination (R²) and squared Pearson correlation (ρ²). The results suggest that Elastic Net can remain competitive with more complex models, that property-based and structure-based OOD splits produce substantially different performance patterns, and that the original BOOM evaluation may underestimate the severity of extrapolation errors due to its metric choice.
Critical strengths
-
Useful benchmark reassessment. The identification of the R²/ρ² discrepancy is valuable and highlights the importance of using appropriate evaluation metrics for molecular OOD prediction.
-
Reasonable robustness checks. The authors include repeated-seed experiments and validation-based hyperparameter tuning, while acknowledging several limitations of their reproduction and experimental setup.
Critical weaknesses
-
Limited novelty and significance of the contributions. The paper primarily revisits an existing benchmark rather than introducing a new methodology, evaluation framework, or substantially new understanding of OOD generalization. Several central observations, including the competitiveness of simple models and the dependence of model performance on the type of distribution shift, are already well established. While the metric discrepancy is a useful finding, it primarily changes the interpretation of absolute performance rather than the main conclusions or model rankings of BOOM. Overall, the incremental insights appear insufficiently significant for ICLR.
-
Insufficient validation of the structural OOD evaluation. The structural OOD analysis relies on a single heuristic UMAP/HDBSCAN split, without quantitatively establishing the degree of structural novelty or systematically comparing against established splitting strategies. The substantially higher performance under this split may reflect its particular construction and difficulty rather than a generalizable distinction between structural and property-based OOD generalization. Moreover, differences in test-set composition and metric aggregation further limit direct comparisons between the two settings.
Questions for authors
-
Could the authors clarify which findings provide fundamentally new insights beyond the original BOOM benchmark and existing studies on classical baselines and molecular OOD evaluation?
-
Can the authors provide quantitative evidence, potentially from existing experiments, that the proposed structural split introduces meaningful molecular novelty? How much of the performance difference could be explained by differences in split difficulty or target distributions?
-
To what extent do the discrepancies in the reproduced GotenNet and MolFormer results affect the reliability of the cross-model conclusions?