acceptodds
Under review as a conference paper at ICLR 2027

Spectrally Constrained Electronic Density Matrix Learning with Subspace Alignment

Abstract

Learning electronic-structure matrices from density functional theory (DFT) has emerged as an active area of atomistic machine learning, providing electronic information beyond the energies and forces typically targeted by machine learning interatomic potentials (MLIPs). Machine-learned Hamiltonian models (MLHs) and density-matrix models (MLDMs) are two representative approaches. MLDMs avoid the numerically sensitive conversion from predicted Hamiltonians to density matrices, but often still underperform MLHs in energy evaluation. We posit that this performance gap arises in part from how density-matrix learning treats eigenvalues and eigenspaces. We introduce a spectrally constrained framework combining Cayley rotations with an occupied-virtual (OV) loss. Cayley rotations preserve the eigenvalues of a valid initial density in an orthonormal representation, leaving the eigenspaces as the remaining degrees of freedom. The OV loss targets the first-order density errors induced by small rotations away from the target eigenspaces. On QH9, our framework outperforms QHFlow2, a state-of-the-art Hamiltonian model, in energy and real-space density accuracy, achieving a mean one-shot energy error of meV with no errors above meV. The predicted densities also accelerate self-consistent field (SCF) calculations by reducing effective iteration cost. In our scaling comparison, a 65.9M-parameter Cayley model with OV loss achieves energy accuracy comparable to a 514M-parameter model without it. Our framework generalizes to larger QM40 molecules without fine-tuning, achieving a mean energy error of meV/atom and improved SCF initialization.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Density Matrix Learning, Electronic Structure Prediction, Equivariant Neural Networks, Density Functional Theory, Hamiltonian prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YV55OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper learns electronic density matrices using a Cayley rotation of a valid initial projector. The rotation preserves electron count and occupations, while an occupied–virtual loss encourages the predicted occupied subspace to approach the target. The method is evaluated through density, energy and force errors, SCF initialization, and transfer from QH9 to larger QM40 molecules.

Critical strengths

The separation between occupation error and subspace error is useful and well motivated.

Critical weaknesses

See below

Questions for authors
  1. The learn density matrix still seems to have idempotency error? is it possible to make it as some sort of penalty term in the loss function to enforce idempotency?
  2. the authors have shown that the predicted density matrices can be used to initialize SCF calculations to reduce the number SCF iterations. But, this paper did not do this for other basis sets. I'm curious, how it looks like if you apply the model on chemical systems that used more complicated basis sets, do you still get similar or consistent speed up.
  3. For chemical systems with large basis sets that might have collinear dependence within the datasets, the basis sets overlap matrix S might not be full rank matrices. Will the method introduced here still work?
3 comments

Dear Reviewer YV55,

Thank you for your positive assessment and constructive feedback on our manuscript. We will incorporate the new experimental results and the associated discussions into the final version. If any of them address your concerns or could change your assessment, we would be glad to hear it and happy to follow up.


(Q1) Remaining idempotency error and an additional penalty

For the Cayley model, idempotency is guaranteed analytically, so the remaining error is a numerical residual of floating-point arithmetic and not a model error.

For models that predict the density matrix directly, an idempotency penalty is possible. It is a soft constraint, however, so it reduces the idempotency error to some extent but does not guarantee valid occupations, and the eigenvalues still have to be purified afterwards. The Cayley parameterization does not need such a penalty.


(Q2) Performance on more complex basis sets

We agree that this is an important question, and we tested it at the ωB97M-V/def2-TZVPD level (Table R3-1). The speed-up remains but is smaller than on QH9. Both Cayley models converge on all 128 evaluated structures and reduce the SCF iteration cost by 8% (ERIC 0.921) and the SCF end-to-end time by about 13%.

For this test, we trained our models on 41,709 closed-shell OMol25 electrolyte structures (25 elements, up to 150 atoms and 4,000 basis functions). Each model was trained for 3 epochs (40 to 50 H100 GPU-hours) from the same PTB-derived initial density (Grimme et al., 2023), projected into def2-TZVPD and completed with atomic core orbitals. Because ORCA, used to generate the reference data, does not support GPU acceleration, we evaluated with GPU4PySCF, and small differences between the two programs are included in the errors. We evaluated the models on 1,024 validation structures with compositions unseen in training, and ran the DFT evaluation of the energy and SCF on a random subset of 128 of them.

This experiment is meant to show that our method also works when the basis set is extended. The one-shot energy error (median 10.36 meV/atom with the OV loss, 15.08 without) is still far from practical accuracy, and we believe closing this gap is an important direction for future work.

Table R3-1. OV-block error (1,024 structures), one-shot absolute energy error in meV/atom, and SCF behavior (128 structures) on OMol25 electrolytes at ωB97M-V/def2-TZVPD.

Model∥Pov∥F\lVert P_{\mathrm{ov}}\rVert_F ↓MAE ↓Median [25%, 75%] ↓SCF convergedERIC ↓SCF end-to-end time ↓ (s)
PTB-derived initial density0.627894.43700.31 [555.87, 966.31]128/1281.00099.81
Cayley, OV00.15098.2815.08 [9.86, 41.53]128/1280.92186.40
Cayley, OV100.13685.7210.36 [6.83, 28.15]128/1280.92186.18
ΔD\Delta D + SR, OV02.83045,421.0321,850.67 [12,959.75, 45,533.52]110/1282.125180.56
ΔD\Delta D + SR, OV101.22110,437.791,395.02 [984.63, 3,243.71]126/1281.396139.28

Here, ERIC and SCF end-to-end time are averaged over structures where the model and the PTB-derived initialization converge to the same SCF solution.

Reference: S. Grimme, M. Müller, and A. Hansen, J. Chem. Phys. 158, 124111 (2023).


(Q3) Linearly dependent basis sets

The current implementation uses Cholesky orthogonalization and therefore assumes a positive-definite overlap matrix. We did not encounter rank-deficient overlap matrices in the def2-TZVPD experiments above, so this assumption held in all settings we evaluated.

When the overlap matrix is rank-deficient, the method can be applied in the linearly independent subspace that remains after removing the redundant basis directions. Established approaches for this, such as canonical orthogonalization and partial Cholesky orthogonalization, are available in PySCF (canonical_orth_ and partial_cholesky_orth_ in pyscf.scf.addons). The Cayley rotation then acts within this subspace and preserves the occupations in the same way. We have not tested this extension yet.

Reference: https://pyscf.org/_modules/pyscf/scf/addons.html

Dear Reviewer YV55,

Thank you again for your positive assessment of our work. We were wondering whether our responses have addressed your questions. If anything remains unclear, we would be glad to follow up during the discussion period. For convenience, we summarize the main points of our responses below:

(1) Idempotency error (Q1). For the Cayley model, idempotency is guaranteed analytically, so the remaining error is a floating-point residual and not a model error. An idempotency penalty is possible for models that predict the density matrix directly, but it is a soft constraint and does not guarantee valid occupations.

(2) More complex basis sets (Q2). We trained on OMol25 electrolytes with 25 elements at ωB97M-V/def2-TZVPD (up to 4,000 basis functions). The speed-up remains but is smaller than on QH9: Cayley converges on all 128 evaluated structures and reduces the SCF iteration cost by 8% and the SCF end-to-end time by about 13% (Table R3-1).

(3) Linearly dependent basis sets (Q3). We did not encounter rank-deficient overlap matrices in the def2-TZVPD experiments. When they occur, the method can be applied in the linearly independent subspace obtained by canonical or partial Cholesky orthogonalization, where the Cayley rotation preserves the occupations in the same way.

Thank you again for your time and for the positive assessment.

Dear Reviewer YV55,

Thank you again for your thoughtful review. With the author-reviewer discussion period closing today (~ 12hr left), we would appreciate knowing whether these have addressed your concerns, or whether any points still need clarification.

Thank you

Reviewer_ujHLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the direct prediction of electronic density matrices within Kohn–Sham density functional theory (DFT). The authors address two limitations of conventional density-matrix learning: violations of physically valid orbital occupations and misalignment of the predicted occupied subspace with the reference solution. The proposed framework combines a Cayley-transform parameterization with an occupied–virtual (OV) loss. Starting from a valid MINAO-derived initial density matrix, the model learns rotations of the occupied subspace while preserving the electron count, occupation spectrum, and idempotency by construction. The auxiliary OV loss explicitly penalizes subspace misalignment alongside the conventional density-matrix reconstruction loss. The method is evaluated on QH9 against additive density-matrix models with McWeeny purification and machine-learned Hamiltonian models, including QHNet and QHFlow2. The experiments assess density-matrix and real-space density accuracy, energies, forces, and SCF initialization. The proposed model achieves a mean one-shot energy error of 0.184 meV on QH9 and reduces the effective SCF iteration cost. The authors also investigate inversion symmetry, model scaling, and numerical stability. Generalization is evaluated on 2,243 larger QM40 molecules without fine-tuning, yielding a mean energy error of approximately 0.98 meV/atom and improved SCF initialization.

Critical strengths
  • Physically motivated approach. The combination of Cayley rotations and OV supervision addresses two distinct limitations of density-matrix learning: invalid orbital occupations and occupied-subspace misalignment. Enforcing spectral constraints by construction provides a principled alternative to post-processing and improves downstream accuracy.
  • The effectiveness of OV-loss. The Cayley model, with a significantly smaller number of parameters and OV-loss, achieves comparable energy efficiency to a model with a larger number of parameters without OV-loss.
Critical weaknesses

Basis dependence of the OV loss.

  • The OV loss in Eq. (37) includes an elementwise absolute-value penalty that is not invariant under orthogonal rotations within the occupied and virtual subspaces. Consequently, the training objective depends on an arbitrary choice of orbital basis, even when the underlying physical subspaces remain unchanged. The authors should justify this choice and compare it with a basis-invariant alternative.

Limited improvement over simpler alternatives and limited generalization.

  • On QM40, spectral retraction reduces the energy error of the ΔD baseline to 1.009 meV/atom, compared with 0.985 meV/atom for Cayley (Table 18), leaving only a 2.4% improvement. Moreover, Cayley and ΔD with McWeeny purification achieve the same SCF iteration metric (ERIC = 0.70) on QM40. Since ERIC excludes density preparation and inference time, the practical computational advantage remains unclear. A comparison with spectral retraction in SCF initialization and end-to-end runtime measurements would better establish the benefit of the proposed parameterization.
  • The evaluation is restricted to closed-shell molecules containing H, C, N, O, and F, with the same DFT functional and basis set used for the target calculations. The QM40 experiment demonstrates molecular-size extrapolation but does not establish transfer to different chemical compositions, open-shell systems, or strongly distorted geometries. Furthermore, the fixed occupation spectrum of the Cayley parameterization limits its direct applicability to systems requiring fractional occupations.
Questions for authors
  1. Computational efficiency. Does the proposed method provide an end-to-end computational advantage when initial density construction, neural network inference, and post-processing are included? The reported ERIC metric excludes some of these costs.

  2. Limits of applicability. How could the proposed parameterization be extended to open-shell systems or systems requiring fractional orbital occupations? Additionally, to what extent do the authors expect the observed improvements to generalize beyond equilibrium geometries and the B3LYP/def2-SVP setting?

  3. Physical self-consistency of predicted densities. The Cayley parameterization guarantees electron count and idempotency but does not enforce Kohn–Sham stationarity. How close are the predicted densities to satisfying the self-consistency condition [F[P], P] = 0? Would the occupied–virtual Fock residual provide a different assessment of prediction quality than the reported density and energy errors?

  4. Consistency of predicted forces. Do the reported one-shot forces include the geometry dependence of the predicted density matrix, or are they evaluated with the density held fixed? Have the authors verified their consistency with finite-difference derivatives of the corresponding one-shot energies?

  5. SCF convergence robustness. Can the authors demonstrate cases where their predicted density enables SCF convergence while conventional initialization (e.g., MINAO) fails? Beyond reducing iteration counts, it would be valuable to establish whether the proposed method improves convergence reliability for challenging molecular systems under identical solver settings and convergence criteria.

11 comments
AuthorsOpenReview ↗

Response to Reviewer ujHL (1/5)

Dear Reviewer ujHL,

Thank you for your time and constructive feedback on our manuscript. We will incorporate the new experimental results and the associated discussions into the final version. If any of them address your concerns or could change your assessment, we would be glad to hear it and happy to follow up.

(W1) Basis dependence of the current OV loss

We agree with your concern about the basis dependence of the OV loss. Our OV loss in Eq. (37) combines an L1 and an L2 term, and the L1 term can be affected by rotations of the eigenvectors within the occupied and virtual subspaces.

To check this effect, we trained a Cayley model in which the L1 term is replaced by an RMS term, so that only invariant terms remain, and evaluated it on 1,024 QH9 test molecules (Table R2-1). It reaches the same energy error as Eq. (37) (0.25 meV, against 0.50 meV without an OV loss), at the cost of a higher DD MAE (41.00 vs. 30.44).

This dependence does not change the observation that subspace alignment improves the energy, since the invariant loss shows the same energy improvement. We also agree that the invariant loss is more principled and will add this experiment to the appendix.

Table R2-1. Cayley models on QH9 with different OV losses.

OV losswOVw_{\mathrm{OV}}DD MAE ↓Trace ↓Idemp. ↓∥Pov∥F\lVert P_{\mathrm{ov}}\rVert_F ↓ρNMAE\rho_{\mathrm{NMAE}} ↓EE ↓FF ↓
Unit10−610^{-6}10−6e10^{-6}e10−610^{-6}10−310^{-3}10−610^{-6}meVmeV/Å
None031.430.200.002.2965.410.502.33
Eq. (37)1030.440.200.001.3255.390.252.19
Basis-invariant (RMS + MSE)1041.000.200.001.4260.140.252.25

(W2-1) Limited improvement over SR

We understand that the small difference on QM40 can look as if the advantage of Cayley disappears for larger molecules. Our investigation shows that the advantage is masked by a size-generalization gap that affects all models. Once this gap is removed with a small amount of data, the difference between Cayley and ΔD\Delta D + SR grows from about 3% to about 32%, while valid occupations and the OV loss remain important.

We fine-tuned on 64 QM40 molecules and 72 QMugs molecules for 3,000 updates, about 1% of the pre-training steps, and tested on QM40 (up to 40 heavy atoms, Table R2-2). To also see the effect at larger sizes, we additionally tested on QMugs, a dataset of drug-like molecules from which we use those with 40 to 80 heavy atoms (Table R2-3).

On QM40, the error of Cayley with OV10 drops from 0.98 to 0.19 meV/atom and that of ΔD\Delta D + SR from 1.01 to 0.28 meV/atom, which gives the 32% difference above. The SCF acceleration of the two models is similar (ERIC 0.676 vs. 0.680), so the advantage of Cayley lies in the one-shot energy.

Table R2-2. QM40 (2,243 matched molecules). Original QH9 models and models fine-tuned on QM40 + QMugs molecules.

ModelFine-tuningDD MAE ↓Trace ↓Idemp. ↓∥Pov∥F\lVert P_{\mathrm{ov}}\rVert_F ↓EE ↓E/NE/N ↓ERIC ↓
Unit10−610^{-6}10−6e10^{-6}e10−610^{-6}10−310^{-3}meVmeV/atomratio
ΔD\Delta D, OV0, rawOriginal193.8763325.90324.2544.63848.8315.680.767
QM40 + QMugs142.8217906.57202.5026.50418.758.630.741
ΔD\Delta D, OV0, +SROriginal190.400.000.0044.64166.052.960.732
QM40 + QMugs128.930.000.0026.5246.250.890.701
ΔD\Delta D, OV10, rawOriginal197.0871953.88290.9026.49677.4112.020.737
QM40 + QMugs175.6119882.64234.1515.36437.038.620.719
ΔD\Delta D, OV10, +SROriginal152.430.000.0026.4956.911.010.703
QM40 + QMugs121.440.000.0015.3714.440.280.680
Cayley, OV0Original121.610.310.0031.2184.311.510.710
QM40 + QMugs83.640.310.0017.4922.450.440.686
Cayley, OV10Original118.450.310.0024.1955.510.980.707
QM40 + QMugs83.280.310.0011.309.200.190.676
AuthorsOpenReview ↗

Response to Reviewer ujHL (2/5)

(W2-1) Cont - Limited improvement over SR

On QMugs, the size-generalization gap is larger. Without fine-tuning, the original Cayley models do not accelerate the SCF (ERIC 1.048 at OV0 and 1.105 at OV10), while ΔD\Delta D + SR gives 12% and 10% acceleration (0.879 and 0.901). After fine-tuning, the energy error of Cayley with OV10 drops from 522.00 to 0.65 meV/atom and that of ΔD\Delta D + SR from 498.27 to 0.87 meV/atom, so Cayley is again more accurate, by about 25%. The ERIC of the two models is comparable (0.726 vs. 0.718).

Table R2-3. QMugs (50 matched molecules). Original QH9 models and models fine-tuned on QM40 + QMugs molecules.

ModelFine-tuningDD MAE ↓Trace ↓Idemp. ↓∥Pov∥F\lVert P_{\mathrm{ov}}\rVert_F ↓EE ↓E/NE/N ↓ERIC ↓
Unit10−610^{-6}10−6e10^{-6}e10−610^{-6}10−310^{-3}meVmeV/atomratio
ΔD\Delta D, OV0, rawOriginal608.422.52×1062.52\times10^{6}1143.65408.6227162.49209.021.295
QM40 + QMugs163.051.00×1051.00\times10^{5}224.2880.672447.8019.690.772
ΔD\Delta D, OV0, +SROriginal650.240.000.00405.4817370.85124.300.879
QM40 + QMugs154.320.000.0080.61407.443.260.742
ΔD\Delta D, OV10, rawOriginal1204.791.87×1061.87\times10^{6}2469.58702.221.17×1051.17\times10^{5}814.431.144
QM40 + QMugs179.0687648.20213.5344.913295.0825.450.750
ΔD\Delta D, OV10, +SROriginal1256.310.000.00687.3769723.12498.270.901
QM40 + QMugs137.210.000.0044.86109.040.870.718
Cayley, OV0Original460.280.480.00438.7226042.28190.011.048
QM40 + QMugs108.890.470.0057.93225.491.790.729
Cayley, OV10Original583.740.470.00661.4870786.01522.001.105
QM40 + QMugs103.070.470.0037.5582.580.650.726

That a small amount of training on larger molecules removes most of the error suggests that the error before fine-tuning is dominated by the shift from QH9 to larger molecules, for example by edges at interaction distances that never appear in QH9 (up to 9 heavy atoms). Valid occupations and the OV loss remain important after fine-tuning. On QM40, the raw ΔD\Delta D model stays far behind its SR counterpart (8.62 vs. 0.28 meV/atom), and the OV loss lowers the OV error and the energy error. We will add these results to the main text.


(W2-1, Q1) End-to-end computational efficiency

ERIC counts only the SCF steps, including the Fock build for the initialization, and excludes other costs such as neural-network inference and post-processing. We therefore measured the end-to-end wall-clock time to a converged SCF on QM40 and QMugs, including every step, using the fine-tuned models from W2-1 (Table R2-4).

All learned initializations are faster than MINAO. Cayley reduces the total time by 26% on QM40 and 17% on QMugs, comparable to ΔD\Delta D + SR. Because inference is no longer negligible on these larger molecules, this gain is smaller than the reduction in SCF time alone (38% and 34%).

In both ERIC and wall-clock time, the two models show a similar and practical level of acceleration, while Cayley with the OV loss has the more accurate one-shot energy (0.19 vs. 0.28 meV/atom on QM40 and 0.65 vs. 0.87 meV/atom on QMugs).

Table R2-4. One-shot energy error and end-to-end time to SCF convergence per molecule (fine-tuned OV10 models).

DatasetMethodE/NE/N ↓ (meV/atom)Initial densityInferencePost-processingSCFTotal ↓Speedup vs. MINAO ↑
QM40MINAO-0.113-0.001810.76410.8781.00
ΔD\Delta D8.621.0300.3440.00057.1388.5121.28
ΔD\Delta D + McW20.291.0280.3440.00146.7598.1321.34
ΔD\Delta D + SR0.281.0280.3440.01046.7488.1311.34
Cayley0.191.0300.3540.00056.7048.0891.35
QMugsMINAO-0.574-0.004120.79021.3681.00
ΔD\Delta D25.452.5261.5150.001714.15818.2011.18
ΔD\Delta D + McW20.902.5211.5150.004113.46717.5071.22
ΔD\Delta D + SR0.872.5371.5150.029513.39917.4801.23
Cayley0.652.5361.5680.001613.63117.7371.21
AuthorsOpenReview ↗

Response to Reviewer ujHL (3/5)

(W2-2, Q2) Limited generalization and limits of applicability

We agree that our evaluation has this scope, and we will add a limitations paragraph to the revised manuscript. Below we answer each point, separating what depends on the training data from what is a fundamental limitation of the Cayley parameterization and the OV loss.

Chemical composition, functional, and basis set. Neither the Cayley parameterization nor the OV loss depends on a particular composition, functional, or basis set. Extending to larger bases and more elements is nevertheless non-trivial in practice. Even for Hamiltonian learning, HELM (Kaniselvan et al., 2026) reports errors on OMol data in def2-TZVPD that are about 16 times higher than on ∇2\nabla^2DFT (def2-SVP) for the same elements and more than 30 times higher for all 58 elements.

To test this setting directly, we trained our models on 41,709 closed-shell OMol25 electrolyte structures with 25 elements and up to 150 atoms and 4,000 basis functions, at the ωB97M-V/def2-TZVPD level (Table R2-5). Each model was trained for 3 epochs (40 to 50 H100 GPU-hours) from the same PTB-derived initial density (Grimme et al., 2023), projected into def2-TZVPD and completed with atomic core orbitals. Because ORCA, used to generate the reference data, does not support GPU acceleration, we evaluated with GPU4PySCF, and small differences between the two programs are included in the errors. We evaluated the models on 1,024 validation structures with compositions unseen in training, and ran the DFT evaluation of the energy and SCF on a random subset of 128 of them.

In Table R2-5, both Cayley models converge in SCF on all 128 evaluated structures. Relative to the PTB-derived initial density, they reduce the SCF iteration cost by 8% (ERIC 0.921) and the SCF end-to-end time by about 13%. OV supervision lowers the OV-block error (0.150 to 0.136), and it lowers the energy error of Cayley on 127 of the 128 structures (median 15.08 to 10.36 meV/atom). The ΔD\Delta D + SR models are much less accurate and fail to converge on 18 and 2 structures, although part of this gap may come from their different training setup.

We therefore see no indication that the Cayley parameterization or the OV loss is limited to small basis sets or to H, C, N, O, and F. The errors still have a long tail (MAE 85.72 vs. median 10.36 meV/atom) and are far from practical accuracy. We see reducing these errors to a practical level and extending the study to more elements as important directions for future work.

Table R2-5. OV-block error (1,024 structures), one-shot absolute energy error in meV/atom, and SCF behavior (128 structures) on OMol25 electrolytes at ωB97M-V/def2-TZVPD.

Model∥Pov∥F\lVert P_{\mathrm{ov}}\rVert_F ↓MAE ↓Median [25%, 75%] ↓SCF convergedERIC ↓SCF end-to-end time ↓ (s)
PTB-derived initial density0.627894.43700.31 [555.87, 966.31]128/1281.00099.81
Cayley, OV00.15098.2815.08 [9.86, 41.53]128/1280.92186.40
Cayley, OV100.13685.7210.36 [6.83, 28.15]128/1280.92186.18
ΔD\Delta D + SR, OV02.83045,421.0321,850.67 [12,959.75, 45,533.52]110/1282.125180.56
ΔD\Delta D + SR, OV101.22110,437.791,395.02 [984.63, 3,243.71]126/1281.396139.28

ERIC and SCF end-to-end time are averaged over structures where the model and the PTB-derived initialization converge to the same SCF solution.

(Continued in the next comment)

Reviewer_Y5mSOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1(−1)
Presentation
2(−1)
Soundness
1
Read review
Paper summary

The paper presents a framework that combines the supervision of the Grassmannian energy gradient (OV-loss) with generator prediction to learn to transport an initial density matrix (after one SCF step) to the ground state (target). This enforces charge and idempotency of the predicted matrices. This model (Cayley) is then compared to SOTA Hamiltonian matrix prediction models and density models with posthock purification on QH9 and OOD tested on QM40 (size extrapolation).

Critical strengths
  • Principled approach

    • Enforcing physics contraints
    • I find the "inversion equivariance by group averaging" insight paticularly interesting.
  • Elegant combination of Grassmannian energy gradient loss (from previous work) with target ground state descriptors (only computed once for the train set)

  • OOD size evaluation on QM40

Critical weaknesses
  • Majority of experiments on saturated QH9
    • energy errors on the order of 10 micro Hatree of both baseline and the proposed method, which is already orders of magnitude below the accuracy of B3LYP and typical numerical errors.
    • On QM40 the effect of the proposed Cayley treatment vanishes almost entirely (table 18):
MethodwOVw_{\text{OV}}D↓\mathbf{D}\downarrow (10−610^{-6})POV↓\mathbf{P}_{\text{OV}}\downarrow (10−310^{-3})ρNMAE↓\rho_{\text{NMAE}}\downarrow (10−610^{-6})EE Mean ↓\downarrow [meV]
ΔD+SR\Delta\mathbf{D} + \text{SR}1010152.430152.43026.49026.490560.22560.2256.91556.915
Cayley1010118.446118.44624.19024.190801.61801.6155.51155.511
  • Weak baselines (see questions below)

  • 2 Fock builds required for Force and energy evalutations, one for D0D_0 from MINAO one for the energetic evaluations themselves, the first could be avoided.

    • (You also might want to mention this in your runtime section in the appendix)
    • While both the proposed model and the baseline start from this one SCF-step density the baseline does not require the initial projection and hence could equally well allocate this additional compute cost after Δ\Delta-MINAO matrix prediction closer to the solution.
Questions for authors

Q1: Weak baselines

  • Why is the much stronger baseline with "fixed rank spectral retraction" only in supplementary Table 18 and not in the main text?
  • Charge conservation is easily enforced via scalar rescaling of the predicted matrix?
  • The baseline based on McWeeny purification could profit from an autodifferentiable training time approach.
  • Have you tried these (they seem to be implemented in the submitted code)?
  • How do the achieved energies and Forces compare to MLIPs, given that MLH and MLDM energy and force evaluations are much more expensive (in an entirely different complexity class)?

Q2: Fock builds and initial projection

  • What about a Fock-build-free delta learning approach?
  • Also a baseline which allocates the extra fock build at the end would be interesting.

Q3: Robustness of SCF Acceleration

Could you test acceleration on a handful of large QMugs molecules?

Minor Additional Remarks

  • The equal parameter focus seems to be beside the point given, that memory is not the bottleneck, and also compute is dominated by the Fock builds, so why not scale up the model expressivity until gains for baselines and proposed method plateau?
    • And the Cayley model requires twice the compute due to the symmetrization under space inversion
    • The way you construct the O(3) models, also doubles the compute. Could you also provide an equal compute baseline to strengthen the "empirical support for the importance of inversion symmetry in density-matrix learning" claim?
12 comments
AuthorsOpenReview ↗

Response to Reviewer Y5mS (1/6)

Dear Reviewer Y5mS,

Thank you for your constructive ideas and review. We will incorporate the new results and discussions into the final version. If any of them address your concerns or could change your assessment, we would be glad to hear from you and are happy to follow up.

Philosophy of our experiments. Before answering, we briefly explain the design of our experiments, since it underlies several of the answers below. We set out to understand why density-matrix prediction fails and to separate two possible causes: (1) invalid occupations and (2) a misaligned occupied–virtual subspace. For the first, our claim concerns valid occupations themselves, whether guaranteed by construction (Cayley, our main approach) or restored by post-processing (McW2, and SR as the reviewer pointed out). The OV loss targets the second. The additional experiments below follow the same design.


(W1-1) Majority of experiments on QH9

We hope the experiments below, in particular the QM40 and QMugs results after fine-tuning on a small number of molecules, ease the concern about evaluating mostly on QH9. We will also add the QMugs experiments to the main text in the revision.

We would also like to explain why studying this problem on QH9 is not trivial. First, the baselines you refer to already follow our message. They combine an occupation correction (McW2 or SR) with the OV loss, and this is what brings them to the 10 μEh\mu E_h (≈ 0.272 meV) level. Without these, the unconstrained ΔD\Delta D gives 100.93 meV, whereas Cayley with OV10 reaches 0.18 meV. Second, the failure modes we identify, namely the inversion failure and the Hamiltonian failure, give a deeper understanding of the problem. How well B3LYP agrees with experiment is a separate question from how well a model reproduces its reference density matrix, which is what we study.


(W1-2) Recovering the advantage of Cayley on larger molecules with a small amount of data

We understand that the small difference on QM40 can look as if the advantage of Cayley disappears for larger molecules. Our investigation shows that the advantage of the proposed method is masked by a size-generalization gap that affects all models. Once this gap is removed with a small amount of data, the difference between Cayley and ΔD\Delta D + SR grows from about 3% to about 32%, while valid occupations and the OV loss remain important.

We fine-tuned on unseen 64 QM40 molecules and 72 QMugs molecules (40 to 80 heavy atoms) for 3,000 updates, about 1% of the pre-training steps, and tested on QM40 (Table R1-1). The error of Cayley with OV10 drops from 0.98 to 0.19 meV/atom and that of ΔD\Delta D + SR from 1.01 to 0.28 meV/atom, which gives the 32% difference above. The SCF acceleration of the two models is similar (ERIC 0.676 vs. 0.680), so the advantage of Cayley lies in the one-shot energy.

These results show that a small amount of training on larger molecules removes most of the error. This suggests that the error before fine-tuning is dominated by the shift from QH9 to larger molecules, for example by edges at interaction distances that never appear in QH9 (up to 9 heavy atoms). Valid occupations and the OV loss remain important after fine-tuning. The raw ΔD\Delta D model stays far behind its SR counterpart (8.62 vs. 0.28 meV/atom), and the OV loss lowers the OV error and, with it, the energy error (for Cayley, 17.49 to 11.30 ×10−3\times 10^{-3} and 0.44 to 0.19 meV/atom).

Table R1-1. QM40 (2,243 molecules). Original QH9 models and models fine-tuned on QM40 + QMugs molecules.

ModelFine-tuningDD MAE ↓Trace ↓Idemp. ↓∥Pov∥F\lVert P_{\mathrm{ov}}\rVert_F ↓EE ↓E/NE/N ↓ERIC ↓
Unit10−610^{-6}10−6e10^{-6}e10−610^{-6}10−310^{-3}meVmeV/atomratio
ΔD\Delta D, OV0, rawOriginal193.8763325.90324.2544.63848.8315.680.767
QM40 + QMugs142.8217906.57202.5026.50418.758.630.741
ΔD\Delta D, OV0, +SROriginal190.400.000.0044.64166.052.960.732
QM40 + QMugs128.930.000.0026.5246.250.890.701
ΔD\Delta D, OV10, rawOriginal197.0871953.88290.9026.49677.4112.020.737
QM40 + QMugs175.6119882.64234.1515.36437.038.620.719
ΔD\Delta D, OV10, +SROriginal152.430.000.0026.4956.911.010.703
QM40 + QMugs121.440.000.0015.3714.440.280.680
Cayley, OV0Original121.610.310.0031.2184.311.510.710
QM40 + QMugs83.640.310.0017.4922.450.440.686
Cayley, OV10Original118.450.310.0024.1955.510.980.707
QM40 + QMugs83.280.310.0011.309.200.190.676
AuthorsOpenReview ↗

Response to Reviewer Y5mS (2/6)

(Q3) Acceleration on large QMugs

The size-generalization gap is larger at the QMugs scale, but the same small fine-tuning recovers the SCF acceleration (Table R1-2). Without fine-tuning, the original Cayley models do not accelerate the SCF (ERIC 1.048 at OV0 and 1.105 at OV10), while ΔD\Delta D + SR gives 12% and 10% acceleration (0.879 and 0.901). With the fine-tuned models from Table R1-1, the ERIC of Cayley reaches 0.729 and 0.726. The ΔD\Delta D + SR baseline reaches a comparable ERIC (0.742 and 0.718), but Cayley with OV10 has the lower energy error (0.65 vs. 0.87 meV/atom).

Table R1-2. QMugs (50 matched molecules). Original QH9 models and models fine-tuned on QM40 + QMugs molecules.

ModelFine-tuningDD MAE ↓Trace ↓Idemp. ↓∥Pov∥F\lVert P_{\mathrm{ov}}\rVert_F ↓EE ↓E/NE/N ↓ERIC ↓
Unit10−610^{-6}10−6e10^{-6}e10−610^{-6}10−310^{-3}meVmeV/atomratio
ΔD\Delta D, OV0, rawOriginal608.422.52×1062.52\times10^{6}1143.65408.6227162.49209.021.295
QM40 + QMugs163.051.00×1051.00\times10^{5}224.2880.672447.8019.690.772
ΔD\Delta D, OV0, +SROriginal650.240.000.00405.4817370.85124.300.879
QM40 + QMugs154.320.000.0080.61407.443.260.742
ΔD\Delta D, OV10, rawOriginal1204.791.87×1061.87\times10^{6}2469.58702.221.17×1051.17\times10^{5}814.431.144
QM40 + QMugs179.0687648.20213.5344.913295.0825.450.750
ΔD\Delta D, OV10, +SROriginal1256.310.000.00687.3769723.12498.270.901
QM40 + QMugs137.210.000.0044.86109.040.870.718
Cayley, OV0Original460.280.480.00438.7226042.28190.011.048
QM40 + QMugs108.890.470.0057.93225.491.790.729
Cayley, OV10Original583.740.470.00661.4870786.01522.001.105
QM40 + QMugs103.070.470.0037.5582.580.650.726
AuthorsOpenReview ↗

Response to Reviewer Y5mS (3/6)

(W2, Q1) Weak baselines

We understand the concern that the main text did not cover more design choices for the occupation treatment. Our claim concerns valid occupations themselves and not one particular treatment, so we compared the two ways of guaranteeing them, by post-processing (McW2, SR) and by construction (Cayley), each with and without the OV loss. In both cases, valid occupations consistently improve over the unconstrained baseline, and the OV loss further improves the energy, so our conclusions do not depend on the specific treatment.

However, we agree that testing more design choices strengthens both the evaluation and the contribution of the paper. We will move (Q1-1) SR into the main text as a main baseline, and we added (Q1-2) charge conservation by scalar rescaling and (Q1-3) training through differentiable McWeeny purification. In (Q1-5), we also compare the energies and forces with MLIPs and computation time.

All new models in Q1-2 and Q1-3 are trained from scratch with the same parameter count and configuration. Because of the short rebuttal period, they are evaluated on a fixed random subset of 1,024 test molecules, and we will report results on the full test set in the final manuscript.


(Q1-1) SR baselines

We agree that SR is a strong test for understanding the effects of exact occupations and the OV loss, and we will move it into the main text and the main tables. We would still like to point out two things. First, on QH9, the gap between McW2 and SR is very small (Table 18 in the paper). Second, on larger molecules, a small amount of fine-tuning recovers the performance gap between Cayley and SR, as shown above (Tables R1-1 and R1-2). We hope these points address your concern.


(Q1-2) Charge conservation via scalar rescaling

Scalar rescaling alone is not sufficient for our purpose, because it fixes the total electron count but does not impose the occupation spectrum. To address your concern, we added a baseline that rescales ΔD\Delta D. Here, rescaling removes the trace error but leaves the idempotency error, and its energy error increases compared to the native one.

wOVw_{\mathrm{OV}}OutputDD MAE ↓Trace ↓Idemp. ↓∥Pov∥F\lVert P_{\mathrm{ov}}\rVert_F ↓ρNMAE\rho_{\mathrm{NMAE}} ↓EE ↓FF ↓
Unit10−610^{-6}10−6e10^{-6}e10−610^{-6}10−310^{-3}10−610^{-6}meVmeV/Å
0ΔD\Delta D67.073548.84111.654.38200.13104.999.53
0NeN_e rescaling67.120.00111.734.38207.20256.817.14
10ΔD\Delta D95.821598.67158.092.06137.2543.835.58
10NeN_e rescaling95.830.00158.102.06139.21133.454.63

(Q1-3) Autodiff McW2 during training

Our question is what happens when the model is designed to satisfy exact occupations, and including McWeeny purification in the model output addresses this question less directly. McWeeny purification does not impose the exact occupation spectrum on arbitrary predictions (Appendix D). It yields valid occupations only if the eigenvalues of the prediction are already close to 0 or 1, with the number of eigenvalues near one equal to the electron count. For this reason, we did not adopt it in our design, which aims to guarantee exact occupations during training.

Following your suggestion, we nevertheless tested it. We trained a model through two differentiable McWeeny steps (ΔD\Delta D (AD)) with OV0 and OV10, and evaluated it both directly and with two additional McWeeny steps after prediction (four in total).

The results show that training through McW2 and applying further purification is also a strong design choice. With four McWeeny steps in total, the energy error drops from 1.69 to 0.61 meV at OV0 and from 0.46 to 0.26 meV at OV10, close to Cayley (0.50 and 0.25 meV). Its direct output, however, still violates the occupation constraints, with energy errors above 100 meV.

wOVw_{\mathrm{OV}}MethodDD MAE ↓Trace ↓Idemp. ↓∥Pov∥F\lVert P_{\mathrm{ov}}\rVert_F ↓ρNMAE\rho_{\mathrm{NMAE}} ↓EE ↓FF ↓
Unit10−610^{-6}10−6e10^{-6}e10−610^{-6}10−310^{-3}10−610^{-6}meVmeV/Å
0ΔD\Delta D + McW259.440.540.004.38132.041.693.44
0ΔD\Delta D (AD)36.694250.314.322.6196.71124.117.98
0ΔD\Delta D (AD) + McW236.620.000.002.6176.140.612.71
0Cayley31.430.200.002.2965.410.502.33
10ΔD\Delta D + McW252.720.240.002.0675.780.462.48
10ΔD\Delta D (AD)35.406448.806.411.40118.15223.5511.71
10ΔD\Delta D (AD) + McW234.930.000.001.4059.850.262.36
10Cayley30.440.200.001.3255.390.252.19

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.