acceptodds
Under review as a conference paper at ICLR 2027

Molecule property prediction with molecular orbitals

Abstract

Molecular orbitals describe the distribution of electrons in a molecule and are frequently used by chemists to understand properties of molecules, yet deep learning has largely neglected them for property prediction so far. In many setups where atom coordinates are computed, they can be obtained cheaply and quickly – a useful source of complementary information, particularly when data is scarce. We give an introduction to molecular orbitals for a machine learning audience and propose models to process three different representations of them. Experiments on a dataset with experimental properties show that including molecular orbitals significantly improves performance and sample efficiency over a pretrained molecular foundation model in real-world settings.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
molecular orbitals, chemistry, use-inspired, geometric deep learning, density functional theory, molecules, property prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nwY9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the use of molecular orbital information for molecular property prediction. It proposes three MO representations, namely Direct MO, Cuboid MO, and Unified MO, and integrates them with existing molecular learning architectures. Experiments on ChromophoreV3-DCM and QM9 suggest that incorporating MO information can improve predictive accuracy.

Critical strengths
  • The proposed approach can leverage molecular orbital information to improve molecular property prediction.
  • The ChromophoreV3-DCM dataset may provide a useful resource for research on OLED-related molecular property prediction, thereby contributing to the open-source research community.
Critical weaknesses
  • The methodological novelty is not sufficiently clear. Previous studies, such as MOB-ML, have already explored the use of molecular orbital information for property prediction. The authors should more clearly distinguish their contributions from prior work.
  • Molecular orbitals have specific symmetries. This concern is particularly relevant to cMO, which achieves better results than the other two representations in the reported experiments but relies on a 3D CNN for feature extraction. Since a standard 3D CNN is not inherently equivariant to arbitrary three-dimensional rotations, the authors rely on data augmentation to mitigate rotational sensitivity. However, data augmentation does not guarantee exact rotational equivariance. Given the substantial progress in equivariant neural networks, the authors should better justify this design choice and evaluate the sensitivity of cMO to rotations and other relevant transformations.
  • Obtaining molecular orbitals generally requires electronic-structure calculations, which can be computationally expensive, particularly for large molecular systems. Since the molecules in datasets such as QM9 are relatively small, the authors should discuss how the proposed approach scales with molecular size.
  • The experiments primarily compare MO-augmented models with DimeNet++, eSEN, and UMA. These ablation studies help assess the contribution of MO information within the selected architectures, but do not fully establish the competitiveness of the proposed approach relative to existing methods. Relevant approaches such as CEONet and OrbNet should be considered as comparison points. Furthermore, QM9 performance has already been substantially improved by existing MLFF methods using only structural information. The reported results also appear to show a performance gap relative to several orbital-based methods, such as OrbNet-Equi. The authors should provide fair and direct comparisons where feasible, or clarify the differences in experimental settings that prevent such comparisons. In addition, the reported QM9 results for some baselines, such as DimeNet++, appear to be worse than their originally reported results, which may lead to an unfair comparison.
  • The related work lacks sufficient coverage of relevant literature on MLFF and equivariant graph neural networks.
Questions for authors
  • What are the respective advantages and limitations of Direct MO, cMO, and uMO? The paper introduces three different representations, but a systematic comparison beyond predictive accuracy would be beneficial. Isn particular, in what scenarios is each MO representation most suitable, and what factors should guide the choice between them?
5 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

The methodological novelty is not sufficiently clear. [...]. The authors should more clearly distinguish their contributions from prior work.

The contribution is identifying the symmetry structure of MO inputs and showing how to handle it inside of existing SotA MLFF architectures. In more detail:

  • We identify the symmetries relevant for modeling MO inputs: permutation symmetry, rototranslation symmetry, eigenspace symmetry.
  • We derive three different representations that capture different subsets of these symmetries
  • We design models that appropriately handle these representations in different ways, e.g. spherical harmonic node features in spherical channels with a SignNet style architecture for uMO, a voxel CNN with canonicalization and augmentations for cMO. Here, we are using a variety of existing individual components or ideas in novel combinations to solve the problems at hand
  • We explain how to integrate these learned features into existing methods, such as spherical channel based methods like eSEN and UMA.
  • We empirically show the dependence of different aspects of the MO features with different experimental properties. Ablations such as Table 5 show that rotation equivariance is less important for MO representation (since cMO and uMO perform similarly) than aspects such as eigenspace sign invariance (sign flip channels are important for Emi_Peak, improving over absolute value channels).

While there are some shared considerations with works such as CEONet, the details are fairly different, such as our uMO representation relying on Spherical Channels used by current state-of-the-art geometric deep learning methods, while CEONet uses a Cartesian basis that does not integrate into existing SotA MLFF methods as cleanly. We go into more detail of how our method is different in the related works section.

Our work is markedly different from typical MOB-ML, as they typically seek invariant representations for use with classical machine learning techniques, while we focus on equivariant representations that modern geometric deep learning methods are able to handle.

Molecular orbitals have specific symmetries. [...]. Given the substantial progress in equivariant neural networks, the authors should better justify this design choice and evaluate the sensitivity of cMO to rotations and other relevant transformations.

It was not our primary objective to perform a thorough evaluation of (equivariant) CNNs or 3D transformers, but to show that even a simple CNN without explicit rotation equivariance handling already performs well. In our view, such evaluation is not necessary for the main messages that we want to deliver through the paper. Please note that we still structurally enforce eigenspace sign symmetry for the cMO representation.

We opted to focus our equivariant modeling efforts on the uMO representation, which is rotation equivariant, in order to fit in with SotA MLFF models. It is certainly possible that results for cMO would improve with a stronger model, which we leave to future work.

As requested, we performed sensitivity analysis on a subset of the data for the cMO model (uMO is fully rotation equivariant up to numerical precision). Here, we evaluate the performance when using rotations that are outside of the training distribution set of augmentations.

ModelTest transformEmi_PeakEmi_FWHMAbs_PeakAbs_FWHM
cMO (3D CNN + augmentation)identity34.4918.2029.5117.29
SO(3) rotations39.6719.4936.0118.73
uMO (equivariant)identity33.4317.7327.8816.00
SO(3) rotations33.4617.7327.9216.02

To be clear, the results we report in the paper and the first row here are the average single-view MAE performance across the 48 views (no prediction ensembling). Therefore, despite the sensitivity to rotations, the cMO approach still performs somewhat similarly to uMO. This shows that structural rotation-equivariance is perhaps less important on this dataset (with greater importance on the eigenspace sign symmetries), but also motivates future work to investigate stronger, rotation-equivariant CNN models to see if such models can further improve.

AuthorsOpenReview ↗

Rebuttal (2/3)

[...] the authors should discuss how the proposed approach scales with molecular size.

We discuss this in detail in appendix F.1. In summary,

  • ChromophoreV3 molecules are significantly bigger than QM9 (up to 150 atoms) and computation was reasonable.
  • Other works use significantly larger molecular systems already (300+ atoms) at higher levels of DFT theory (typically for computing DFT properties).
  • There are a number of techniques to scale to much larger (1000+ atoms) systems, like GFN2-xTB.
  • Our approach uses 6 frontier MOs, which is 6 * K (K being the number of basis functions). Works such as OrbNet take 4 * K^2 as input, which scales significantly worse.
  • The cMO representation grows with the volume of the molecule, while the uMO integration grid increases linearly with the atom count.
  • The UMA model scales linearly with the number of atoms. Default OrbNet scales quadratically in the number of basis functions O(K^2), which is usually a decent bit larger than the number of atoms. The frontier MO input scales with O(K), while the eSEN/UMA backbones scale with O(N_atoms).

Relevant approaches such as CEONet and OrbNet should be considered as comparison points.

We agree that such comparisons would make the paper stronger, but we explain in the related works section why these methods are not easily comparable: CEONet targets orbital-level labels, not molecule-level properties, and OrbNet takes O(K^2) matrices as input and does not integrate into existing SotA MLFF models as easily. Also, a key strength of our paper is that we separate the contribution of electronic structure representation from the model that processes that representation, while it is entangled in both CEONet and OrbNet.

Furthermore, QM9 performance has already been substantially improved by existing MLFF methods using only structural information.

Our approach improves UMA, which is a SotA MLFF method, on QM9, but we are not training it in the same full training pipeline that is known to yield maximum performance (see details below) because it would require significantly more computation. The QM9 experiment is intended to show that MOs are helpful to a wider variety of properties, so our intention is not to reach SotA performance on this dataset, but to provide a reasonable, scientifically valid comparison between models that don’t and do use MOs.

Please elaborate on this point if we have misunderstood the intention behind it.

The reported results also appear to show a performance gap relative to several orbital-based methods, such as OrbNet-Equi. The authors should provide fair and direct comparisons where feasible, or clarify the differences in experimental settings that prevent such comparisons. In addition, the reported QM9 results for some baselines, such as DimeNet++, appear to be worse than their originally reported results, which may lead to an unfair comparison.

We explain the discrepancy to published DimeNet++ numbers in Appendix E. We focus on apples-to-apples comparisons by using the exact same training setups between the different models so that the only factor of variation is the change we want to evaluate. We believe this to be more scientifically interesting than exactly reproducing a more costly training setup of a specific model (DimeNet++). This is in line with the main message of our paper, which is “MO information improves various models for molecule property prediction” and not “we want to achieve SotA on QM9”.

As an example from appendix E, an easy way to improve the absolute numbers is to train one model per target property. DimeNet already evaluated this to be a significant gain over multi-property prediction with a single model. We did not judge this to be an interesting change to evaluate considering the 9x increased training costs for each of the 9 targets. Other factors are longer training with early stopping, larger batch size, and different data splits.

These are also the reasons why we believe the published OrbNet-Equi results to be significantly improved over our results on QM9, with the multi-property vs multiple single property prediction aspect being one of the main factors. Again, since OrbNet-Equi uses a different set of data (Fock, overlap, core Hamiltonian, density), it is not an apples-to-apples comparison to our method using MOs, so a fair comparison is not easily possible. For ChromophoreV3-DCM, its default scaling behavior is much worse (quadratic in basis functions, many basis functions per atom) than models like UMA (linear in atoms), so it is unclear whether our much larger ChromophoreV3-DCM molecules (up to 150 atoms) can be reasonably used with the OrbNet approach, which was evaluated on molecules below 50 atoms.

AuthorsOpenReview ↗

Rebuttal (3/3)

The related work lacks sufficient coverage of relevant literature on MLFF and equivariant graph neural networks.

We will add some more detail on this and cover methods such as NequIP, MACE, EquiformerV2, (in addition to eSEN and UMA that we already have) and point to relevant review articles for a more in-depth background on these topics.

What are the respective advantages and limitations of Direct MO, cMO, and uMO? The paper introduces three different representations, but a systematic comparison beyond predictive accuracy would be beneficial. Isn particular, in what scenarios is each MO representation most suitable, and what factors should guide the choice between them?

This is a good question that we will incorporate into the paper. Some of these are speculative, as we have not explicitly performed experiments verifying them. In short:

dMO:

  • Most direct representation from DFT, so least amount of processing steps involved; faster conversion from DFT logs and smallest disk space usage.
  • No information sharing between different atoms means less sample efficiency.
  • Perhaps useful when precise shapes matter and a lot of data is available. Given the weak experimental results so far, prefer one of the other ones.
  • Unsuitable when basis sets change within dataset

cMO:

  • Not tied to basis functions centered on atoms, so could be better than uMO when there is significant MO amplitudes relatively far away from atoms
  • In the same vein, possibly better at recognizing when global MO structure is important.
  • Not structurally rotation equivariant (only approximate through augmentations), so it’s best to use it when there is a reasonable and somewhat consistent notion of canonical orientation of molecules.
  • Representation size scales with volume, so can take more disk space than other representations. Suitable when basis sets change within the dataset

uMO:

  • Fits most easily into current SotA MLFF models, so a good reasonable default choice.
  • Fully rotation equivariant, so consistent canonical pose not necessary; better when molecular structures vary significantly within the dataset
  • Basis functions are atom-centered, so scales better for larger molecules by focusing representation capabilities on where non-trivial MO amplitudes are likely to be found.
  • Suitable when basis sets change within the dataset

In general, we would recommend uMO as a default choice as it has a useful combination of properties. In specific settings as outlined above, cMO can be worth considering.

Reviewer_7mK7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to use molecular orbitals (MOs) as an extra input for molecular property prediction. The main argument is that when atom coordinates are obtained with DFT (or similar), the MOs are computed anyway as part of the geometry optimisation, so they come almost for free. The authors give an introduction to MOs for a machine learning audience and propose three representations:

  1. a direct MO representation (dMO) using the MO coefficients in the original basis, embedded per chemical element into eSEN;
  2. a cuboid representation (cMO), where the MOs are sampled on a 3D voxel grid and processed with a 3D CNN;
  3. a unified representation (uMO), where the MOs are projected onto a basis that is shared across chemical elements and added to the eSEN node features.

They handle permutation, rototranslation and sign symmetry of the MOs, and optionally add the MO energies. They also build a dataset, ChromophoreV3-DCM, of 2,336 OLED molecules with experimentally measured emission and absorption peaks and FWHMs in dichloromethane, evaluated with 10-fold cross-validation over 5 seeds. Adding MO features to DimeNet++, eSEN and a pretrained UMA model improves the results, mostly for the peak wavelengths. A low-data experiment shows better sample efficiency for the peak properties, and MO features also improve all three base models on QM9.

Critical strengths

1. The problem is clearly framed and well motivated, and the sample efficiency experiment supports it. Experimental data is expensive and scarce, and the paper makes a good case that information already sitting in standard computational chemistry pipelines is being thrown away. Section 2 makes MOs accessible without a chemistry background, and the new dataset with experimental targets is a useful contribution, since most existing benchmarks predict simulated properties. The sample efficiency experiment (Figure 2) is the most practically relevant result in the paper: since data is key in this setting, knowing how much data is needed is really useful, and the improvement is clear (UMA with cMO at 4% of the data beats UMA without MOs at 32% on Emi_peak). Together with the ablation in Table 2 and an appendix that is quite open about negative results, this makes the paper informative.

Critical weaknesses

1. The novelty is overstated. The abstract says deep learning "has largely neglected" MOs, but the paper's own Appendix A discusses several works that use orbital information as model input (e.g. OrbNet, OrbNet-Equi, OrbitAll, MOB-ML, SPAHM and King et al., whom the authors call "pioneering"), and OrbNet already did this on QM9. In addition, Molecular Orbital Learning (MōLe; Thiede et al., 2026, arXiv:2602.20232) also uses MO coefficients as input to a rotation-equivariant network and handles their sign symmetry, but it is not discussed. The contribution should be framed more narrowly (frontier MO coefficients as input to existing equivariant models, evaluated on experimental properties).

2. Several empirical claims are not sufficiently supported.

  • MO coefficients vs. MO energies. The proposed representations encode the MO coefficients (the eigenvectors of the Fock matrix), but all models in Table 3 also receive the MO energies (its eigenvalues). Table 2 shows that for the peak wavelengths the MO energies alone give most of the gain (eSEN with uMO, Emi_peak: 46.14 without any MO input, 33.58 with MO energies only, 32.73 with both). Without a "base model + MO energies only" control, the improvements cannot be attributed to the MO coefficients rather than to a few eigenvalues that have long been used as descriptors.
  • "Complementary" claim. Adding cMO (MO coefficients and energies) to eSEN improves Emi_peak by about 12 nm (45.53 → 33.39), but going from the cMO-only model (no atom positions) to eSEN with cMO only gives about 0.8 nm (34.21 → 33.39), and that also includes a larger encoder. This suggests the MOs contain most of the relevant information, rather than complementing the atom positions.
  • Baselines. The Coulomb matrix and SLATM use only nuclear charges and geometry, yet are introduced as "standard electronic structure methods". The conclusion that the equivariant models learn better from "this type of information" therefore does not follow, and the cMO model is not equivariant either.
  • Robustness to the source of MOs. This rests on a single xTB row computed on DFT geometries, where the gain shrinks and disappears for Emi_FWHM. One example is not enough for this claim.
  • QM9. The baselines are far below published numbers (DimeNet++ U0: 25.35 meV here vs. 6.32 meV in the original paper), and Table 7 shows that uMO makes UMA worse on every QM9 property, while the main text states that MOs improve "each base model on every property", which only holds for cMO.
  • Symmetries. The claim that "accommodation of symmetries is crucial for sample efficiency" is not tested by any ablation.
Questions for authors

Clarifications and changes that could affect my assessment:

  1. Can you add "base model + MO energies only" rows to Table 3, ideally also to Figure 2? This is the most important point for me.
  2. Why is there no DimeNet++ uMO result in Table 3 (and Table 7)? I assume it is because uMO (like dMO) is added to spherical-harmonic node features, which DimeNet++ does not have. If so, please state it, and consider whether an invariant version (e.g. using only the l = 0 components) would be possible. Similarly, dMO is not included in Table 3 at all. Is this because it performed worst in Table 1? Please state this explicitly in Section 5.1.
  3. How large is the fixed-size validation set in the low-data experiments? At 1% (around 21 training samples), is it larger than the training set? If so, the setting is less realistic for scarce data.
  4. Which regressor was used for the Coulomb, SPAHM and SLATM baselines, and what are their standard errors?
  5. How do the QM9 results compare to prior orbital-based methods, and how large are the MO gains on top of baselines trained to their published accuracy?
  6. Please consider rephrasing: "largely neglected", "complementary to the atom positions", "robust to different sources of MOs", "standard methods that utilize electronic structure information", and "significantly improves ... sample efficiency" (the FWHM properties are slightly worse than the baseline at 1–2% data, Appendix D).

Suggestions (would strengthen the paper, not required for my assessment):

  • I really like Figure 2, and I think it would be worth running it for the other setups too: uMO and dMO (Section 3.3 argues that sharing parameters across elements helps when data is scarce, and this is where it would show), a from-scratch eSEN or DimeNet++, and all four targets in the main text.
  • Run xTB end to end (xTB geometries + xTB MOs) and at least one other basis set or functional to support the robustness claim.
  • Minor: line 361 says "unified MO (cMO)", which should be uMO.
5 comments
AuthorsOpenReview ↗

Rebuttal (1/4)

The novelty is overstated. [...] The contribution should be framed more narrowly (frontier MO coefficients as input to existing equivariant models, evaluated on experimental properties).

We will revise the paper to make our claim narrower and clearer. For example, in the abstract we will change the wording from “deep learning” to “geometric deep learning” to emphasise that we are not referring to MOB-ML type techniques (hand-designed descriptor + ML regressor), and put more focus on property prediction to separate it from other prediction tasks. We tried to use the word “largely” in the abstract to soften our claim, but we will make this claim more precisely in the update.

Thank you for the reference to MōLe, we were not aware of it, and we will make sure to include it in an update. Having now read it, it is an example of a method that does not try to do general property prediction, but predicting excitation amplitudes from coupled cluster (CC) analysis. Our paper is addressing the most generic property prediction setting where invariance is desired, while MōLe is about the more specific problem of equivariant excitation amplitudes, so it is not directly comparable in experiments. We can add some additional detail on the tradeoffs between direct property prediction and predicting computational properties like CC metrics in the appendix as well.

MO coefficients vs. MO energies.

We agree that the paper would be strengthened with this ablation for Table 3 as well (in addition to the small model ablation in Table 2) and have now run it with the same training procedure, mean over 5 repeats of 10-fold cross-validation. We will include this updated result in the paper:

Base modelMO inputEmi_PeakEmi_FWHMAbs_PeakAbs_FWHM
DimeNet++no MO45.9319.2040.1018.33
energies only33.2418.9328.1117.61
shapes only (cMO)42.4117.5335.7916.94
energies + shapes (cMO)31.6016.9726.7616.04
eSENno MO46.1618.2240.0518.02
energies only33.4217.8128.6316.75
shapes only (cMO)44.9417.5840.5517.72
shapes only (uMO)38.8316.8036.2716.22
energies + shapes (cMO)33.5817.4828.0816.50
energies + shapes (uMO)32.6817.2128.3216.12
UMAno MO32.8016.4227.7315.55
energies only29.6716.4525.8115.19
shapes only (cMO)32.4116.2727.4915.37
shapes only (uMO)33.8415.7630.3615.13
energies + shapes (cMO)30.7316.7026.2115.43
energies + shapes (uMO)32.0317.1727.5015.51

There are a few takeaways:

  • We confirm the Table 2 result that MO energies primarily help Peak properties while MO shapes primarily help FWHM. The latter is the case regardless of shape representation for Emi_FWHM (though rather small for cMO) and true for only uMO on Abs_FWHM in the UMA rows (cMO is slightly worse here).
  • DimeNet++ improves on all 4 properties when given cMO shapes + energies over when given energies only.
  • The eSEN with energies model is always outperformed by one of the eSEN models with shape, either shape alone or shape + energy.
  • For UMA, the best results come from either energy only (Peak) or shape only (FWHM). We believe that we have identified part of the issue, which is due to the pretrained initialization not expecting MO features to be present (see QM9 table below).

In summary, we believe our experimental results are strengthened in terms of identifying cases where energies or shapes help, as well as establishing that MO shapes can provide benefits on both FWHM and Peak properties (the latter for the from-scratch models), even if the Peak properties improve through shapes to a lesser degree than through the MO energies.

AuthorsOpenReview ↗

Rebuttal (2/4)

"Complementary" claim. [...] This suggests the MOs contain most of the relevant information, rather than complementing the atom positions.

We agree that MOs contain the majority of the useful information, which is another piece of evidence that the premise of our paper is sound. We had made this claim because the base + MO models tend to beat the MO-only models, showing that there is some information that MOs alone are not able to capture. For example, cMO 29.98 -> eSEN + cMO 28.08 on Abs_Peak, with larger gaps when using DimeNet++ or UMA as base models. To be less misleading, we will change our claim in the abstract and the text to be more specifically on improving accuracy and sample efficiency of the base models, rather than being complementary.

Baselines. The Coulomb matrix and SLATM use only nuclear charges and geometry, yet are introduced as "standard electronic structure methods". The conclusion that the equivariant models learn better from "this type of information" therefore does not follow, and the cMO model is not equivariant either.

Well spotted, those are indeed writing mistakes that we will fix: we will update the descriptions to be accurate.We had incorrectly grouped them with SPAHM, which does use electronic structure. For cMO, we will revise the claim to not be about equivariant models in general (since cMO models eigenspace symmetries but not rototranslation), but our learned representations and models specifically.

Robustness to the source of MOs. This rests on a single xTB row computed on DFT geometries, where the gain shrinks and disappears for Emi_FWHM. One example is not enough for this claim.

Fair enough, we will make sure to soften this claim. We want to note that there is nothing in our architecture that is dependent on the basis set or functional being used; the uMO and cMO representations convert the existing basis set into a new basis, independent of the original one. We don't see any particular reason why our claims would only work for B3LYP/def2-svp aside from the pure quality difference in how well the resulting MOs capture the correlations with the experimental properties.

We plan to change the claim to be primarily theoretical and we will clarify that experiment using xTB MOs to be preliminary evidence for robustness. To be clear, xTB uses its own functional and basis set already.

QM9. The baselines are far below published numbers (DimeNet++ U0: 25.35 meV here vs. 6.32 meV in the original paper), and that MOs improve "each base model on every property", which only holds for cMO.

We explain this discrepancy in absolute values in appendix E (DimeNet trains separate models for each property, early stopping, 9x more samples per training run). The main reason is that scientifically, we value a like-for-like training comparison more than exactly reproducing the existing absolute numbers; it would require a substantial amount of additional computation to train one model for each property instead of the multi-property approach we took. The existence of this performance gap is well understood (see original DimeNet paper).

For the second claim, please see our updated table below.

Table 7 shows that uMO makes UMA worse on every QM9 property, while the main text states that MOs improve "each base model on every property", which only holds for cMO.

We have now investigated why uMO degrades the pretrained model and identified that we add the MO representation directly to the UMA input, which forces the parameters to adapt significantly to take it into account. We have now rerun a version where the MO representation is added with a zero-initialized scalar gate to eliminate this effect: at the start of finetuning, the model sees exactly the same input as a model without MO inputs. As a result, we observe improved results.

Modelµα⟨R²⟩ZPVEU0UHGCv
UMA0.01260.04190.8262.1618.1018.2818.4418.230.0225
UMA cMO0.01110.03830.6391.7713.8914.0714.1114.440.0203
UMA uMO0.01180.04030.7032.0016.2216.4716.5416.810.0221
UMA uMO + gate0.01170.03940.6641.8214.3314.5314.5714.970.0213

uMO performance is now fairly similar to cMO and both improve over the baseline in this updated experiment.

Note that the UMA results above reflect a unified rerun after refactoring our finetuning pipeline to ensure consistent initialization and evaluation across the QM9 and ChromophoreV3-DCM experiments. All rows in this table (and the updated Table 3 above) use the same standardized testing protocol to make sure that they are apples-to-apples comparisons. Even though the UMA becomes stronger under this update, including both cMO and gated uMO consistently improve across the 9 QM9 targets of interest.

AuthorsOpenReview ↗

Rebuttal (3/4)

Symmetries. The claim that "accommodation of symmetries is crucial for sample efficiency" is not tested by any ablation.

We will cite some relevant references that evaluate the sample efficiency of geometric deep learning in the field of molecular modeling to evidence this statement. In Table 5, we also evaluate the eigenspace sign symmetry in cMO by measuring the individual impact of the absolute value and sign channels, though not on a sample efficiency level.

Please see below for the additional sample efficiency plots, where there is a clear improvement over the base models with either structural equivariance (uMO) or approximate equivariance through augmentations (cMO). It also demonstrates that exact rotation equivariance through uMO is not strictly necessary for the sample efficiency improvements, as it performs similarly to cMO.

Can you add "base model + MO energies only" rows to Table 3, ideally also to Figure 2? This is the most important point for me.

We have now performed the requested experiments. See above for the new Table 3 results, and see below for a description of the updated Figure 2 with the energies-only variant, and the uMO and dMO variants included.

Why is there no DimeNet++ uMO result in Table 3 (and Table 7)? I assume it is because uMO (like dMO) is added to spherical-harmonic node features, which DimeNet++ does not have. If so, please state it, and consider whether an invariant version (e.g. using only the l = 0 components) would be possible.

That is correct, we will revise the paper to state it explicitly. Using l=0 components would be possible, but we decided against it because it would make the DimeNet++ model more complicated while also removing the angular structure (l=1 and above) that we believe to be important.

Similarly, dMO is not included in Table 3 at all. Is this because it performed worst in Table 1? Please state this explicitly in Section 5.1.

Yes, we indirectly state it as the last sentence in 5.1, but we can certainly revise it to make this point more clearly by mentioning that we drop it explicitly.

How large is the fixed-size validation set in the low-data experiments? At 1% (around 21 training samples), is it larger than the training set? If so, the setting is less realistic for scarce data.

For these experiments, we train for a fixed training budget with no early stopping, no model selection, and no separate hyperparameter tuning, so there is no need for a separate validation set. The test set is indeed larger than the training set (at 1%, 21 training molecules per fold, 234 held out molecules per fold), but that does not affect any training aspects; a larger test set here simply means less variance in the results reported.

Which regressor was used for the Coulomb, SPAHM and SLATM baselines, and what are their standard errors?

We use the official SPAHM implementation at https://github.com/lcmd-epfl/SPAHM, which implements these three methods. Each representation is fed into a kernel ridge regression with a Laplacian or Gaussian kernel. Exceptionally, we used the 10-fold test set performance to tune the hyperparameters to present these baselines in the best possible light; none of the other methods in our paper are tuned on the test set. The tuning is on {Gaussian, Laplacian} x [5 log scale regularizations] x [13 log scale kernel widths], yielding 130 combinations. This is the same hyperparameter sweep as in the official SPAHM code. We will include these details in an update to the paper.

We do not report standard errors for these because there is no similar notion of a random initialization seed.

How do the QM9 results compare to prior orbital-based methods, and how large are the MO gains on top of baselines trained to their published accuracy?

As we explain in Appendix E and in our response to reviewer nwY9, we do not believe that reproducing the exact training setup to be comparable to published numbers is particularly meaningful, given the substantially increased training costs. Our focus in this paper is on the ChromophoreV3-DCM dataset due to its experimental properties and thus clearer real-world applicability. We do not intend to make any SotA claims on QM9, but rather an apples-to-apples training comparison of established base models with the same models enhanced with MO information. The main purpose of the QM9 experiment is to show that MOs are beneficial to a wider variety of properties than the four tested in ChromophoreV3-DCM. Thus, we do not have such a comparison.

Reviewer_RwC7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies the effect of using the molecular orbitals (MOs) for the molecular property prediction. They give a pedagogical overview of the MO, and then introduce three different representations of MOs and corresponding encoders to embed those to machine learning models. They create ChromophoreV3-DCM dataset from the existing dataset. The experiments compare the performance of the MO features compared with other molecular features and how the MOs feature enhance the property prediction performance when combined with the other existing features in the molecular energy prediction models using the experimental ChromophoreV3-DCM dataset and simulated QM9 dataset.

Critical strengths

Targeting experimentally measured properties is a valuable direction. It is also valuable to emphasize an importance of sample efficiency in the real world setting.

Critical weaknesses
  • There are no comparisons with existing deep learning methods using MOs.
  • The contribution is limited: the key message looks as MOs are useful features for the properties prediction in one actual experimental data. This does not meet the standard of ICLR in my opinion given that there are existing works using the MOs as features.
  • The baseline in Table 1 seems very weak; the model with MOs utilizes the power of the model with rich parameters while the baseline using properties obtained from DFT is combined with the linear regression.
  • The result in Table 2 shows that the MO energies rather than MO shapes are more important features for the property predictions. This weakens the key message of this paper.
Questions for authors
  • The existing model compared is for the energy prediction models; I could not find how they are modified for the property predictions. Could you elaborate on this?
  • Could you elaborate on how to fine tune UMA?
5 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

There are no comparisons with existing deep learning methods using MOs.

Prior works using MOs develop invariant descriptors for classical ML (behind modern geometric deep learning), require multiple O(K^2) matrices (not O(N_atoms) as ours), or predict lower level properties such as orbital-level labels (not as general as ours). To our knowledge, there is no suitable method that would be a straightforward drop-in comparison for property prediction. More concretely:

CEONet, the closest and concurrent work to ours, feeds MOs into a geometric model but targets orbital-level labels (manually labeled bonding character, per-MO energies/entropies) instead of molecule-level properties. They use a single representation based on a Cartesian basis, while ours is based on spherical harmonics that are easily incorporated into current SotA MLFF approaches. Also, the impact of model and representation is not separately assessed for CEONet, while we do this for our representation and models. SIMG, SPAHM and MAOC all reduce the electronic structure to hand-designed, atom- or bond-localized invariant features derived from an NBO or density matrix analysis, rather than learning from the MO vectors themselves. The OrbNet line of research consumes full Fock, overlap, core Hamiltonian, and density matrix inputs, making its input substantially larger than MOs. This has significantly worse scaling behaviour (quadratic in number of basis functions, each atom has many basis functions) than eSEN/UMA (linear in atoms). As they design a bespoke architecture, it cannot piggyback on top of SotA MLFF methods like our method as easily. Other MOB-ML methods derive features from localized MOs but specifically target post-Hartree-Fock correlation energies, which have pairwise decompositions that do not apply to general property prediction.

The closest usable baselines we are aware of use molecular machine learning descriptors with a standard classical ML pipeline. This is what we compare to in Table 1 (Coulomb, SPAHM, SLATM with their official configuration: kernel ridge regression with Gaussian or Laplacian kernel; HOMO/LUMO and vS1/vT1 energies with a linear probe). The lack of strong baselines is also part of the premise of the paper: there is very little work specifically combining geometric deep learning and MOs for molecule-level property prediction, which is why we believe it to be important to develop these methods given the recent advances in molecular modeling and known importance in chemistry.

The contribution is limited: the key message looks as MOs are useful features for the properties prediction in one actual experimental data. This does not meet the standard of ICLR in my opinion given that there are existing works using the MOs as features.

Contribution First, we would like to clarify that our contribution is broader than that: we design three new MO representations and characterise their symmetry properties, as well as recipes for how they can be integrated with standard molecular modeling approaches. We show that they improve performance, particularly in realistic low-data settings, and that they improve state-of-the-art molecule foundation models on a standard dataset (QM9) as well as ChromophoreV3-DCM. This dataset itself is also a contribution as it features relevant experimental properties of this class of molecules. We describe in our previous answer how these are improvements over existing work.

Experimental data We want to emphasise that most existing works in this field are evaluated on simulated data only, such as QM9, MD17, MD22, which all use DFT generated coordinates and properties. To our knowledge this holds for most of the related cited works in our related works section, including ICLR and other top ML conference papers. Using experimental data in our paper addresses a key limitation of the standard DFT-only methodology.

The reason why DFT simulations are so commonplace is that they are reasonably high-quality simulations based on fundamental quantum mechanics. Properties that DFT is able to compute are often highly predictive of experimental measurements, so the prediction problems are still meaningful and should therefore not be confused with the inherent drawbacks of simulated data in other domains. In our paper, we wanted to eliminate the problem where simulation errors are correlated between DFT-calculated properties and DFT-calculated MOs, which is why we felt the need to curate this new dataset with experimental properties. This is something that many other papers in this field do not do, and we believe that this is an important aspect that makes our results stronger for real-world applicability.

AuthorsOpenReview ↗

Rebuttal (2/3)

The baseline in Table 1 seems very weak; the model with MOs utilizes the power of the model with rich parameters while the baseline using properties obtained from DFT is combined with the linear regression.

We have now added some stronger MLP baselines for models using DFT properties as input as the reviewer suggested. The results improve slightly over linear regression, but nowhere near the uMO and cMO models we have developed so the conclusions do not change; we will include this result in the updated paper.

ModelEmi_Peak ↓Emi_FWHM ↓Abs_Peak ↓Abs_FWHM ↓
HOMO/LUMO + linear42.45 ± 1.8820.34 ± 0.0443.83 ± 2.0319.88 ± 0.11
HOMO/LUMO + MLP (new)38.50 ± 0.0119.61 ± 0.0135.51 ± 0.0419.03 ± 0.02
vS1/vT1 + linear40.68 ± 0.6919.74 ± 0.1641.08 ± 0.4519.56 ± 0.29
vS1/vT1 + MLP (new)38.04 ± 0.0518.26 ± 0.0134.92 ± 0.0418.01 ± 0.02
uMO32.73 ± 0.1717.54 ± 0.1928.39 ± 0.2616.28 ± 0.08

The size of these MLPs matches the capacity of the output head of the MO models. Also note that we compare to stronger, richer baselines. Tables 3 and 4 compare against DimeNet++, eSEN and the 150M-parameter pretrained UMA model (which in principle could already have implicit knowledge about MOs through structure alone), and MO features improve on all of them. We used smaller, simpler models to compare to in Table 1 and Table 2 since our models there are smaller too.

For the Coulomb, SPAHM and SLATM baselines, they are standard molecular descriptors: we use official implementations and their published kernel/regressor pipelines (they have zero fold-to-fold variance because they are deterministic). Changing this would deviate from the published model, and our intention is to provide reasonable baselines that practitioners might already use, not develop new and improved versions of them.

The result in Table 2 shows that the MO energies rather than MO shapes are more important features for the property predictions. This weakens the key message of this paper.

Both shape and energies are part of the MO features we are arguing for in the paper. The latter is significantly easier to model (fewer equivariances to take care of), so most of the discussion in the paper is focused on modeling the former correctly. We are not claiming that shapes are always beneficial: from the underlying physics, energies are highly related to Peak-related properties because they correspond to vertical excitation gaps, while shapes encode aspects such as wavefunction delocalization, which is known to be related to widening the spectrum peaks (FWHM).

The existing Table 2 results show that energies primarily benefit Peak properties while shape primarily benefits FWHM properties, which mirrors the physics. We now provide some new results where we perform the comparison of energies only vs energies + uMO shape on top of the stronger base models in Table 3. There, we see that shape can also improve the Peak properties, even when energies are already present: for example, DimeNet++ 33.24 -> 31.60 Emi_Peak, 28.11 -> 26.76 Abs Peak; eSEN 33.42 -> 32.68 Emi_Peak, 28.63 -> 28.32 Abs_Peak. We also want to note that our new low-data figures (see response to reviewer 7mK7) show that shapes provide the majority of performance gain when data is scarce (<16%).

We consider our characterisation of the dependence a strength, not a weakening of the contribution. Please see our response to reviewer 7mK7 for the full results.

AuthorsOpenReview ↗

Rebuttal (3/3)

The existing model compared is for the energy prediction models; I could not find how they are modified for the property predictions. Could you elaborate on this?

We remove the energy prediction head and replace it with a freshly initialized 3-layer MLP with 128 channels and ReLU activations, with as many outputs as regression targets. While the energy head outputs per-atom energies that are summed (i.e. MLP applied to each atom), we first pool the representations and then apply the MLP on the molecule level, as this does not assume the properties to be additive; in initial experiments this led to improved results. While we state this for the combined models in section 5.3, we will include this information in the updated appendix for the standalone models as well. Nothing else is changed compared to the reference model.

Could you elaborate on how to fine tune UMA?

We use the published UMA-S model architecture with its public UMA-S 1.1 (~150M parameters) weights as initialization. The only modification we make is the replacement of the energy prediction head described above, as well as attaching any optional components such as the appropriate MO encoder for the representation with randomly initialized weights. All parameters are fine-tuned end-to-end without freezing any weights with a single exception: the dataset identity embedding (used by UMA to distinguish different energy distributions of a concatenated dataset) is kept fixed because we only have one dataset at a time. We use the LR schedule and training setup as specified in Appendix F.6: Adam, cosine schedule with 4% linear warmup from 6e-5 to 3e-4 down to 3e-6, weight decay 1e-3, gradient clipping 0.5. The UMA baseline without MOs is finetuned with exactly the same training recipe as the MO versions to make these valid apples-to-apples comparisons where the only factor of variation is the one we want to test (MOs vs no MOs)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes

We thank all the reviewers for their time in reviewing our paper. We believe that this constructive feedback allowed us to significantly strengthen the paper. We have also standardized and rerun all experiments under a unified protocol after some refactoring to make sure that all comparisons remain apples-to-apples. We conducted five new experiments, and revised the positioning and some inaccurate terminology.

Summary of main revisions

MO energies vs shapes (RwC7, 7mK7) New ablations for the base models across the Table 3 and sample efficiency setups show that the property performance improvements agree with physical understanding. MO energies are closely related to energy gaps that determine emission and absorption peaks, while MO shapes are linked to the broadening of the spectra, measured by FWHM, through a variety of mechanisms. MO shapes also improve Peak accuracy for the from-scratch models and are the main cause of sample efficiency gains in low data regimes.

Positioning (RwC7, 7mK7, nwY9) We narrow our claims to focus specifically on geometric deep learning for molecule-level property prediction and more clearly delineate our approach from hand-designed descriptors (MOB-ML, SPAHM, etc), architectures that scale with O(K^2) (OrbNet), and approaches that target orbital-level or coupled cluster quantities (CEONet, MōLe). We refocus our “complementary” terminology, as well as soften the xTB robustness claim.

Representation and symmetry (nwY9) We discuss the properties of the three representations in more detail and when one would choose one over another, with uMO as the default recommendation.

Details on experiments (RwC7, 7mK7, nwY9) We detail how the molecule descriptor baselines are combined with kernel ridge regression with a 130 combination hyperparameter sweep. We also clarify the reason why our QM9 results appear to be worse than published numbers: we prefer the scientific value of apples-to-apples comparisons within the same training setup than exact reproductions in the best, computationally expensive pipeline. We make it clearer why we omit dMO in later experiments, as well as the incompatibility of DimeNet++ with uMO and dMO.

New experiments

Stronger DFT property based baselines in Table 1 (RwC7) We added HOMO/LUMO + MLP and vS1/vT1 + MLP baselines that are matched in size to the MO-only output heads. They are slightly better than a linear model, but do not come close to uMO and cMO performance, so the conclusions do not change.

Full scale energies-only and shape-only ablations in Table 3 (7mK7, RwC7) This quantifies the effect of Table 2 at the full scale (when combined with the different base models) and shows that MO shape can contribute to improving both Peak and FWHM properties at this scale over a base + energies only model, either categorically for the from-scratch models (DimeNet++, eSEN), or for FWHM properties in the shape-only configuration with the pretrained model (UMA).

Sample efficiency experiments for more combinations of properties, base model, and energies vs shape (7mK7) This more precisely quantifies in what cases what types of MO information is beneficial in low-data regimes. Specifically, we identify shape as being particularly beneficial for Peak properties in the low data regime (<16% training data).

Improved UMA results for a rerun of QM9 (7mK7) This strengthens the baseline UMA as well as confirming that despite that, both uMO and cMO improve over it, not just cMO. This is partially achieved through a new gating mechanism that improves initialization behaviour when adding MOs to a pretrained model.

Rotation sensitivity analysis of cMO (nwY9) This quantifies the effect of SO(3) rotations on the model trained only on axis aligned augmentations, which indicates using a rotation-equivariant voxel approach with the cMO representation as a future direction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.