- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PhysDFTBench, a tool designed to evaluate whether neural exchange-correlation functional models satisfy specific physical exact conditions inherent to Density Functional Theory (DFT). First, the authors directly test whether published model weights satisfy these properties; for instance, they observe that the energy output of the deployed DM21 model changes when spin channels are swapped for a fixed density—a finding that leads them to conclude the released DM21 models fail to satisfy exact spin-permutation symmetry. Second, the authors train their own neural correction models, leveraging various strategies: some properties (such as M4) are guaranteed through specific construction, while others are approximated by incorporating penalties into the loss function. Experiments demonstrate that conditions guaranteed by construction remain satisfied, whereas conditions enforced via the loss function do not necessarily meet the target criteria. Finally, the authors discuss the circumstances under which the SCF response can be ignored during training, categorizing scenarios into "protected," "needs no protection," and "unprotected." This analysis rectifies an implementation error found in an earlier version of their own implementation.
Critical strengths
- The physics-informed condition test proposed by the authors is simple and straightforward. For instance, it requires calculating the energy twice for the same fixed density, simply by swapping the two spin channels. This test requires no reference energy, no retraining, and no additional SCF run. Furthermore, the authors found that the results for all four sets of released DM21 weights exceeded the round-off level; thus, at least one point is clear: the released DM21 models do not satisfy exact spin-permutation symmetry.
- The paper is careful to clarify exactly what different tests demonstrate. Some tests correspond to exact identities, while others are conducted within the constraints of a finite basis, finite , or a one-sided condition. The paper avoids simply labeling these cases as satisfying an "exact condition." For example, in the case of a one-sided test, merely falling within a certain numerical range does not constitute proof that the exact condition holds.
Critical weaknesses
-
The paper's core conclusion is quite straightforward, and the experiments largely verify things that could have been anticipated from the outset. A condition guaranteed by construction is fundamentally different from a condition incorporated into the training loss as a penalty. If a specific identity is guaranteed by construction for any set of parameters, the training process will not violate it. Conversely, adding a term like to the loss function offers no guarantee that the final solution will satisfy . Thus, it is hardly surprising that a condition included in the loss fails to meet the corresponding criterion in the final model. Similarly, the statement ``an architecture is not a warranty'' is quite direct and can already be anticipated without these experiments. I also have reservations regarding the M4 results. The DM21 supplementary material describes both strict two-pass symmetrization and approximate symmetrization based on random spin flipping, yet the paper under review does not clarify which method was actually used for the four sets of released weights. Consequently, while the M4 results demonstrate that the released DM21 models lack exact spin-permutation symmetry, a non-zero M4 residual is to be expected if the approximate scheme was employed during training. It is therefore unclear how the released weights tested here correspond to the strict or approximate symmetrisation described in the original DM21 paper.
-
Section 5 essentially involves re-applying a standard result to the authors' own early implementation.
The paper itself acknowledges that using the envelope theorem to neglect the SCF response—under appropriate conditions—is a standard shortcut. The issues identified in Section 5 stem from the authors' own earlier training implementation: they applied this shortcut without satisfying the necessary stationarity conditions, thereby omitting the response term. Furthermore, their proposed A/B/C classification appears to derive directly from the expression . Class A corresponds to ; Class B applies when the density is independent of the parameter with respect to which the derivative is taken; and Class C covers cases where neither condition holds. Properly verifying the assumptions of the chain rule and the envelope theorem naturally leads to these scenarios.
Moreover, it is unclear what the A/B/C classification actually adds compared to the correct application of the existing envelope theorem. The failure case explicitly demonstrated in the paper stems from the authors' own previous implementation. Furthermore, the training run in which the authors corrected the gradient also involved changes to the label set; consequently, the specific impact of the gradient correction on the final training result was not isolated or measured.
Questions for authors
The questions stem from the previously mentioned weakness:
-
How should the results for M4 be interpreted? The DM21 supplementary material describes both "strict two-pass symmetrization" and "approximate symmetrization based on random spin flipping." Which of these methods was actually used for the four sets of released weights tested in this paper? If this is currently unknown, then aside from demonstrating that the released DM21 implementation is not exactly spin-permutation symmetric, what else does M4 reveal?
-
Why is the A/B/C classification necessary? The paper notes that neglecting the SCF response via the envelope theorem is a standard shortcut. If the assumptions of the envelope theorem are properly verified, the A, B, and C scenarios appear to follow directly. Does this work introduce any new sufficient or necessary conditions? Apart from the authors' own previous implementation, has any existing neural DFT work been shown to suffer from the same gradient problem? Can these findings be used to improve those existing studies?
Thanks for the authors' responses. I will keep my initial score.