- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the use of molecular orbital information for molecular property prediction. It proposes three MO representations, namely Direct MO, Cuboid MO, and Unified MO, and integrates them with existing molecular learning architectures. Experiments on ChromophoreV3-DCM and QM9 suggest that incorporating MO information can improve predictive accuracy.
Critical strengths
- The proposed approach can leverage molecular orbital information to improve molecular property prediction.
- The ChromophoreV3-DCM dataset may provide a useful resource for research on OLED-related molecular property prediction, thereby contributing to the open-source research community.
Critical weaknesses
- The methodological novelty is not sufficiently clear. Previous studies, such as MOB-ML, have already explored the use of molecular orbital information for property prediction. The authors should more clearly distinguish their contributions from prior work.
- Molecular orbitals have specific symmetries. This concern is particularly relevant to cMO, which achieves better results than the other two representations in the reported experiments but relies on a 3D CNN for feature extraction. Since a standard 3D CNN is not inherently equivariant to arbitrary three-dimensional rotations, the authors rely on data augmentation to mitigate rotational sensitivity. However, data augmentation does not guarantee exact rotational equivariance. Given the substantial progress in equivariant neural networks, the authors should better justify this design choice and evaluate the sensitivity of cMO to rotations and other relevant transformations.
- Obtaining molecular orbitals generally requires electronic-structure calculations, which can be computationally expensive, particularly for large molecular systems. Since the molecules in datasets such as QM9 are relatively small, the authors should discuss how the proposed approach scales with molecular size.
- The experiments primarily compare MO-augmented models with DimeNet++, eSEN, and UMA. These ablation studies help assess the contribution of MO information within the selected architectures, but do not fully establish the competitiveness of the proposed approach relative to existing methods. Relevant approaches such as CEONet and OrbNet should be considered as comparison points. Furthermore, QM9 performance has already been substantially improved by existing MLFF methods using only structural information. The reported results also appear to show a performance gap relative to several orbital-based methods, such as OrbNet-Equi. The authors should provide fair and direct comparisons where feasible, or clarify the differences in experimental settings that prevent such comparisons. In addition, the reported QM9 results for some baselines, such as DimeNet++, appear to be worse than their originally reported results, which may lead to an unfair comparison.
- The related work lacks sufficient coverage of relevant literature on MLFF and equivariant graph neural networks.
Questions for authors
- What are the respective advantages and limitations of Direct MO, cMO, and uMO? The paper introduces three different representations, but a systematic comparison beyond predictive accuracy would be beneficial. Isn particular, in what scenarios is each MO representation most suitable, and what factors should guide the choice between them?
5 comments
Rebuttal (2/3)
[...] the authors should discuss how the proposed approach scales with molecular size.
We discuss this in detail in appendix F.1. In summary,
- ChromophoreV3 molecules are significantly bigger than QM9 (up to 150 atoms) and computation was reasonable.
- Other works use significantly larger molecular systems already (300+ atoms) at higher levels of DFT theory (typically for computing DFT properties).
- There are a number of techniques to scale to much larger (1000+ atoms) systems, like GFN2-xTB.
- Our approach uses 6 frontier MOs, which is 6 * K (K being the number of basis functions). Works such as OrbNet take 4 * K^2 as input, which scales significantly worse.
- The cMO representation grows with the volume of the molecule, while the uMO integration grid increases linearly with the atom count.
- The UMA model scales linearly with the number of atoms. Default OrbNet scales quadratically in the number of basis functions O(K^2), which is usually a decent bit larger than the number of atoms. The frontier MO input scales with O(K), while the eSEN/UMA backbones scale with O(N_atoms).
Relevant approaches such as CEONet and OrbNet should be considered as comparison points.
We agree that such comparisons would make the paper stronger, but we explain in the related works section why these methods are not easily comparable: CEONet targets orbital-level labels, not molecule-level properties, and OrbNet takes O(K^2) matrices as input and does not integrate into existing SotA MLFF models as easily. Also, a key strength of our paper is that we separate the contribution of electronic structure representation from the model that processes that representation, while it is entangled in both CEONet and OrbNet.
Furthermore, QM9 performance has already been substantially improved by existing MLFF methods using only structural information.
Our approach improves UMA, which is a SotA MLFF method, on QM9, but we are not training it in the same full training pipeline that is known to yield maximum performance (see details below) because it would require significantly more computation. The QM9 experiment is intended to show that MOs are helpful to a wider variety of properties, so our intention is not to reach SotA performance on this dataset, but to provide a reasonable, scientifically valid comparison between models that don’t and do use MOs.
Please elaborate on this point if we have misunderstood the intention behind it.
The reported results also appear to show a performance gap relative to several orbital-based methods, such as OrbNet-Equi. The authors should provide fair and direct comparisons where feasible, or clarify the differences in experimental settings that prevent such comparisons. In addition, the reported QM9 results for some baselines, such as DimeNet++, appear to be worse than their originally reported results, which may lead to an unfair comparison.
We explain the discrepancy to published DimeNet++ numbers in Appendix E. We focus on apples-to-apples comparisons by using the exact same training setups between the different models so that the only factor of variation is the change we want to evaluate. We believe this to be more scientifically interesting than exactly reproducing a more costly training setup of a specific model (DimeNet++). This is in line with the main message of our paper, which is “MO information improves various models for molecule property prediction” and not “we want to achieve SotA on QM9”.
As an example from appendix E, an easy way to improve the absolute numbers is to train one model per target property. DimeNet already evaluated this to be a significant gain over multi-property prediction with a single model. We did not judge this to be an interesting change to evaluate considering the 9x increased training costs for each of the 9 targets. Other factors are longer training with early stopping, larger batch size, and different data splits.
These are also the reasons why we believe the published OrbNet-Equi results to be significantly improved over our results on QM9, with the multi-property vs multiple single property prediction aspect being one of the main factors. Again, since OrbNet-Equi uses a different set of data (Fock, overlap, core Hamiltonian, density), it is not an apples-to-apples comparison to our method using MOs, so a fair comparison is not easily possible. For ChromophoreV3-DCM, its default scaling behavior is much worse (quadratic in basis functions, many basis functions per atom) than models like UMA (linear in atoms), so it is unclear whether our much larger ChromophoreV3-DCM molecules (up to 150 atoms) can be reasonably used with the OrbNet approach, which was evaluated on molecules below 50 atoms.
Rebuttal (3/3)
The related work lacks sufficient coverage of relevant literature on MLFF and equivariant graph neural networks.
We will add some more detail on this and cover methods such as NequIP, MACE, EquiformerV2, (in addition to eSEN and UMA that we already have) and point to relevant review articles for a more in-depth background on these topics.
What are the respective advantages and limitations of Direct MO, cMO, and uMO? The paper introduces three different representations, but a systematic comparison beyond predictive accuracy would be beneficial. Isn particular, in what scenarios is each MO representation most suitable, and what factors should guide the choice between them?
This is a good question that we will incorporate into the paper. Some of these are speculative, as we have not explicitly performed experiments verifying them. In short:
dMO:
- Most direct representation from DFT, so least amount of processing steps involved; faster conversion from DFT logs and smallest disk space usage.
- No information sharing between different atoms means less sample efficiency.
- Perhaps useful when precise shapes matter and a lot of data is available. Given the weak experimental results so far, prefer one of the other ones.
- Unsuitable when basis sets change within dataset
cMO:
- Not tied to basis functions centered on atoms, so could be better than uMO when there is significant MO amplitudes relatively far away from atoms
- In the same vein, possibly better at recognizing when global MO structure is important.
- Not structurally rotation equivariant (only approximate through augmentations), so it’s best to use it when there is a reasonable and somewhat consistent notion of canonical orientation of molecules.
- Representation size scales with volume, so can take more disk space than other representations. Suitable when basis sets change within the dataset
uMO:
- Fits most easily into current SotA MLFF models, so a good reasonable default choice.
- Fully rotation equivariant, so consistent canonical pose not necessary; better when molecular structures vary significantly within the dataset
- Basis functions are atom-centered, so scales better for larger molecules by focusing representation capabilities on where non-trivial MO amplitudes are likely to be found.
- Suitable when basis sets change within the dataset
In general, we would recommend uMO as a default choice as it has a useful combination of properties. In specific settings as outlined above, cMO can be worth considering.
Rebuttal (1/3)
The contribution is identifying the symmetry structure of MO inputs and showing how to handle it inside of existing SotA MLFF architectures. In more detail:
While there are some shared considerations with works such as CEONet, the details are fairly different, such as our uMO representation relying on Spherical Channels used by current state-of-the-art geometric deep learning methods, while CEONet uses a Cartesian basis that does not integrate into existing SotA MLFF methods as cleanly. We go into more detail of how our method is different in the related works section.
Our work is markedly different from typical MOB-ML, as they typically seek invariant representations for use with classical machine learning techniques, while we focus on equivariant representations that modern geometric deep learning methods are able to handle.
It was not our primary objective to perform a thorough evaluation of (equivariant) CNNs or 3D transformers, but to show that even a simple CNN without explicit rotation equivariance handling already performs well. In our view, such evaluation is not necessary for the main messages that we want to deliver through the paper. Please note that we still structurally enforce eigenspace sign symmetry for the cMO representation.
We opted to focus our equivariant modeling efforts on the uMO representation, which is rotation equivariant, in order to fit in with SotA MLFF models. It is certainly possible that results for cMO would improve with a stronger model, which we leave to future work.
As requested, we performed sensitivity analysis on a subset of the data for the cMO model (uMO is fully rotation equivariant up to numerical precision). Here, we evaluate the performance when using rotations that are outside of the training distribution set of augmentations.
To be clear, the results we report in the paper and the first row here are the average single-view MAE performance across the 48 views (no prediction ensembling). Therefore, despite the sensitivity to rotations, the cMO approach still performs somewhat similarly to uMO. This shows that structural rotation-equivariance is perhaps less important on this dataset (with greater importance on the eigenspace sign symmetries), but also motivates future work to investigate stronger, rotation-equivariant CNN models to see if such models can further improve.