- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces Force Delta (), the mean absolute difference between direct and gradient-based force predictions to measure stability and uncertainty in direct force MLIPs. This approach is motivated by showing direct force predictions cause energy drift even with low pointwise force errors. On 50 MPtrj systems, has high correlation to an exact Jacobian conservatively measure and to downstream energy drift. As an UQ metric, is shown to be superior or competitive over some systems against ensembles and other UQ metrics, including ones requiring training/fitting. On other systems there is low correlation between the metric and the error to DFT. The authors demonstrate using for efficient finetuning with an active learning setup, showing reduced force error and energy drift using up to 200 structures.
Critical strengths
Immediate UQ for direct force MLIPs. The method is simple, works out-of-the-box for direct force MLIPs in a single model, and requires no additional training. High correlation with exact non-conservativity measure and simulation drift. On an Orb model with 50 different materials systems, the paper shows high correlation between to \lambda and simulation energy drift, offering a cheaper way to measure non-conservativity. The comparisons reported against other UQ metrics are strong, showing is competitive across MLIP architectures. Active Learning. The empirical results of the active learning are very good, showing substantial decrease in test error and nearly removing the drift using only 100 structures over 2 generations (200 total).
Critical weaknesses
Instability limited to drift. is validated as a stability metric against energy drift, while instability can manifest in a multitude of different ways (unphysical geometries, temperature drift, structural collapse, etc.). Correspondingly, showing just energy drift with accurate direct force predictions in section 4.1 is unsurprising. It is established that low pointwise force error does not guarantee energy conservation, including drift in direct-force models even with lower force errors [1]. While section 4.1 mentions the authors saw increased temperature fluctuations in NVT simulations using the direct forces, it is established in prior work [2] and this analysis is limited to that section only. The authors should show instabilities other than just drift or restrict claims about using the metric to drift and not stability overall.
Correlation with \lambda measure is limited to Orb. The high correlation with the exact non-conservativity measure \lambda is only done with two Orb models and MPtrj systems, so it is not established whether these results generalize to other architectures and systems.
Weak UQ results on certain systems. Certain systems in the adversarial attack don’t have a strong correlation with (Table 4). This is not reported in the main text, which just states “ retains a strong positive correlation with across structures”, referring only to Figure 3. This also does not hold up with the results presented in Table 4. The reasoning given in the appendix, that this is due to initial inaccuracy of the model, seems to only hold partially in Figure 7, as some of the systems with low correlation also have high errors. If it is indeed the reasoning, show stronger evidence, and discuss it in the main text. Additionally, the comparison against trained UQ (Table 2) is done only over ammonia, while with other systems it might not perform as well.
Active learning ablations. While the active learning results are impressive, without ablations it is unclear whether the improved energy drift (and test accuracy) can be attributed to using the 200 -maximized structures, or to the additional training, particularly due to supervising the conserving forces. This is especially true for the second generation results. Including an ablation study (different methods for choosing the fine tuned structures, supervising different combinations of conserving and direct forces, number of generations, etc.) would strengthen this.
Active learning drift measurement. Measuring drift for 10 ps can be insufficient and is inconsistent with other drift measurements done in the paper. The authors should use a consistent justifiable drift measurement protocol throughout the paper.
Aligned direct and conservative force errors. The authors acknowledge that a limitation of is when the two forces differ in the same direction and magnitude from the ground truth, however they do not characterize this mode of failure (diagnosis, dependence on system? architecture?).
Non-equivariant direct force head for MACE. In the MACE setup, the added Orb-style direct force prediction head breaks the equivariance of the model, weakening the claim that works well on equivariant architectures.
[1] Zhang, D., Bi, H., Dai, FZ. et al. Pretraining of attention-based deep learning potential model for molecular simulation. npj Comput Mater 10, 94 (2024). https://doi.org/10.1038/s41524-024-01278-7 [2] Bigi, F., Langer, M. F., & Ceriotti, M. (2025). The dark side of the forces: assessing non-conservative force models for atomistic machine learning. Forty-Second International Conference on Machine Learning. Retrieved from https://openreview.net/forum?id=OEl3L8osas
Questions for authors
What is the motivation for using mean absolute difference and not metrics (e.g., root mean square difference)? In Figure 2, can you color the points markers by the system and/or model? Any outliers? What is the spearman correlation of each system? In the MACE setup, why did you use the Orb-style non-equivariant direct force head and not an equivariant alternative? Can you show how performs with MACE using an equivariant head? In the active learning fine tuning of Orb, does each setting use the resulting model of the previous setting? So does the finetuning of and together use the model that was trained on the same 100 structures of finetuning only, and does the second generation use the model that was trained on both? Can you show the computational efficiency of the method compared with the baseline model and other methods? Is there a way to diagnose when a user can trust that a small is attributed to a small force error and not to the two predictions deviating in the same direction? Which systems are the results in Table 1 reported over? In section 4.6, what is the timestep used and MD protocol used?
8 comments
Replies to Reviewer bi7z (Part 2)
Weak UQ results on certain systems, and aligned force errors. We agree with the reviewer that the weak correlations in Table 4 and the aligned-error failure mode deserve further investigation than a few sentences in the appendix, and we thank the reviewer for pointing these out. We accordingly extended the adversarial attack experiments to five additional pretrained eqV2 models and two MACE models trained from scratch on an MPtrj subset, one direct-force-only and one hybrid, both with the exactly equivariant head described below, all validated against DFT. Entries are , , :
| System | eqV2-dens-86M-mp | eqV2-dens-153M-mp | eqV2-31M-mp | eqV2-31M-omat-mp-salex | eqV2-86M-mp | MACE direct | MACE hybrid |
|---|---|---|---|---|---|---|---|
| ice | 0.98, 0.99, 0.95 | 0.92, 1.00, 0.94 | 0.88, 1.00, 0.90 | 0.87, 0.98, 0.83 | 0.92, 0.97, 0.93 | 0.96, 0.92, 0.96 | 1.00, 0.97, 0.97 |
| Mg17Al12 | 0.86, 0.98, 0.92 | 0.95, 0.99, 0.93 | 0.95, 0.99, 0.91 | 0.88, 1.00, 0.85 | 0.89, 0.98, 0.92 | 1.00, 1.00, 1.00 | 1.00, 1.00, 1.00 |
| LGPS | 0.88, 1.00, 0.85 | 0.91, 0.97, 0.93 | 0.97, 1.00, 0.99 | 0.93, 0.98, 0.92 | 1.00, 1.00, 1.00 | 1.00, 1.00, 1.00 | 0.79, 0.87, 0.93 |
| CaPd-NH$_2$ | 0.35, 1.00, 0.33 | 0.25, 0.98, 0.16 | 0.99, 0.98, 1.00 | 0.98, 0.98, 0.96 | 1.00, 0.98, 0.97 | 0.98, 0.98, 0.97 | 1.00, 0.95, 0.95 |
| aspirin | 0.58, 0.97, 0.60 | 0.71, 0.89, 0.70 | 0.95, 0.99, 1.00 | 0.98, 1.00, 0.97 | 0.95, 0.97, 0.96 | 0.99, 1.00, 0.99 | 0.88, 0.93, 0.91 |
| paracetamol | 0.07, 0.99, 0.15 | 0.87, 0.97, 0.73 | 0.75, 0.99, 0.82 | 1.00, 1.00, 1.00 | 1.00, 1.00, 1.00 | 1.00, 1.00, 1.00 | 0.94, 1.00, 0.94 |
| MOF-5 | 0.13, 0.99, 0.09 | 0.24, 0.98, 0.17 | 0.89, 0.99, 0.87 | 0.96, 1.00, 0.96 | 0.92, 1.00, 0.93 | 1.00, 1.00, 1.00 | 1.00, 1.00, 1.00 |
Together with the 22 pairs of Table 4, this gives 71 model–system pairs. The correlation with the direct-force error is strong across most of the set: the median is 0.92 over all 71 pairs, and 0.95 over the 51 pairs from models not trained with a denoising objective (DeNS) [4], with 40 of those at 0.85 or above. The weak correlations concentrate in the DeNS checkpoints, whose median is 0.51 over 20 pairs and which improves monotonically with model capacity: 0.33 at 31M, the checkpoint used in Table 4, then 0.58 at 86M and 0.87 at 153M.
One possible explanation is that DeNS trains the per-atom output head to predict the added position noise and the per-structure energy head to predict the energy of the uncorrupted structure from corrupted input [4], so the two targets are not related by differentiation as they are under standard direct-force training. This may make the two heads behave differently enough that their errors decouple and cannot capture both errors at once, with extra capacity partially absorbing the difference.
Whether the two force errors move together, , also predicts how well ranks : the two correlations agree at Pearson . Both heads read the same representation, so out of distribution their errors usually rise together and tracks both errors well. On a few paths, DeNS-trained models in particular, the two heads behave differently enough that the attack on raises the error of one force while the other (mostly the trained direct force head) stays robust, decoupling the two errors and weakening .
Aligned errors, in which both forces deviate in the same direction with large errors while stays small, are rare: 1 of 71 paths, namely MOF-5 under orb-d3-xs-v2, the smallest and only non-equivariant model in the set, on the most structurally heterogeneous system we test. This suggests that the consensus failure mode may occur when a system is so far from the training prior, especially for a model of small capacity, that the model stops responding to the distribution shift and outputs a constant small while the error keeps increasing as the ground truth shifts further from both forces. This failure mode is inherent to disagreement-based UQ, as with deep ensembles, which are known to underestimate errors that all members share in the OOD regime [6]. Our results nonetheless suggest practical guidelines for avoiding it: prefer larger models, avoid denoising-trained checkpoints where is to be used for the direct force, and treat systems far from the training distribution with caution, for instance by labeling the true error of a few low- structures with DFT. We have added this discussion to the revision and the full results to the appendix.
Replies to Reviewer bi7z (Part 3)
Trained-UQ benchmarks beyond ammonia, and an equivariant direct-force head for MACE. The reviewer is right that the MLP head in the submission breaks equivariance of MACE. We used the Orb-style head to keep the MACE and Orb architectures comparable, but we agree with the reviewer and have implemented an exactly equivariant direct-force head: , where are the 128 equivariant features of atom from MACE's first interaction layer and the weights are predicted by an MLP from the invariant () features of both layers. Invariant weights multiplying equivariant vectors give exactly; a rotation test gives a violation of on ammonia and on silica, against for the MLP head, as the reviewer correctly points out.
We evaluated this head on ammonia and on two further benchmarks: condensed-phase SiO2 from Tan et al. [5], with large periodic cells and an OOD test set of high configurational complexity from high-deformation-rate trajectories, and a multi-system MACE trained on a 20k subset of MPtrj structures and tested on OMat24 rattled structures. Across all of them, remains superior or competitive against ensembles and other UQ methods while being the only out-of-the-box single-model method available from standard direct-force MLIP training, without additional fitting or specialized training. As with the ammonia benchmark in the submission, its utility also extends to hybrid models in which both forces are trained and can be used.
-
Ammonia with the equivariant direct-force head. remains competitive against the deep ensemble and outperforms GMM, while MVE remains unstable owing to its hard-to-optimize specialized loss. Submitted results are included for comparison. (AUROC):
Method Orb MACE, equivariant MACE, MLP head 0.832 ± 0.026 (0.969 ± 0.007) 0.874 ± 0.031 (0.982 ± 0.002) 0.861 ± 0.055 (0.966 ± 0.003) GMM 0.819 ± 0.070 (0.962 ± 0.023) 0.841 ± 0.042 (0.979 ± 0.006) 0.898 ± 0.030 (0.982 ± 0.003) MVE 0.692 ± 0.120 (0.897 ± 0.076) 0.702 ± 0.157 (0.862 ± 0.124) 0.675 ± 0.241 (0.791 ± 0.129) Deep ensemble 0.825 (0.982) 0.933 (0.987) 0.873 (0.976) (baseline suggested by Reviewer B3Fn) 0.723 ± 0.024 (0.891 ± 0.005) 0.731 ± 0.072 (0.882 ± 0.100) 0.773 ± 0.050 (0.902 ± 0.001) -
Condensed-phase SiO2 [5]. (AUROC):
Method Orb MACE, equivariant MACE, MLP head 0.917 ± 0.012 (0.986 ± 0.006) 0.935 ± 0.031 (0.975 ± 0.008) 0.908 ± 0.029 (0.982 ± 0.004) GMM 0.784 ± 0.030 (0.967 ± 0.010) 0.775 ± 0.018 (0.914 ± 0.009) 0.796 ± 0.022 (0.921 ± 0.012) MVE 0.635 ± 0.183 (0.827 ± 0.125) 0.712 ± 0.243 (0.851 ± 0.243) 0.681 ± 0.103 (0.867 ± 0.098) Deep ensemble 0.951 (0.990) 0.953 (0.988) 0.932 (0.985) 0.762 ± 0.014 (0.831 ± 0.003) 0.744 ± 0.004 (0.865 ± 0.002) 0.722 ± 0.012 (0.867 ± 0.003) -
Multi-system training from scratch, MACE with the equivariant head trained on an MPtrj subset, with an OMat24 rattled OOD test set:
Method AUROC 0.864 ± 0.023 0.898 ± 0.002 Deep ensemble 0.902 0.926 0.718 ± 0.032 0.810 ± 0.021 -
Hybrid models, in which both forces are supervised, as in the MTS scheme of [2]. Practitioners can use either force in these models, and correlates well with both errors, serving as a UQ metric for both simultaneously:
Method / System : : AUROC : : AUROC / Ammonia 0.908 ± 0.065 0.989 ± 0.003 0.953 ± 0.027 0.989 ± 0.003 GMM / Ammonia 0.847 ± 0.019 0.975 ± 0.005 0.873 ± 0.027 0.975 ± 0.005 Deep ensemble / Ammonia 0.906 0.987 0.960 0.984 / Ammonia 0.791 ± 0.062 0.894 ± 0.078 0.832 ± 0.042 0.926 ± 0.025 / MPtrj 0.934 ± 0.001 0.980 ± 0.002 0.923 ± 0.002 0.980 ± 0.002 Deep ensemble / MPtrj 0.914 0.972 0.901 0.962 / MPtrj 0.826 ± 0.001 0.896 ± 0.005 0.834 ± 0.002 0.903 ± 0.005
Replies to Reviewer bi7z (Part 1)
We thank the reviewer for thoroughly reviewing our work and for giving us many valuable suggestions, which led to new experiments that substantially improve the generality of the paper and further support the UΔ utility claims in the original submission. We address each point below.
Instability limited to energy drift. We agree with the reviewer and have restricted our stability claims to energy drift in the revised version. We focus on energy drift because Bigi et al. [2] show that it is the root cause of the other instabilities, including the structural and dynamical property deviations that follow from thermostat interference, and that correcting the drift also recovers them. We have also retained our NVT measurement as a confirmation of their result, as originally intended.
We also thank the reviewer for pointing us to Zhang et al. [1], which, together with Bigi et al. [2], shows that a conservative force can give more stable dynamics than a more accurate direct force. We have reframeed Section 4.1 as a confirmation of this finding, which our setting extends: in our work F^c is never trained and has large error, unlike the trained conservative force in [1, 2], so exact conservativity matters more than static accuracy for energy conservation even when the conservative force is substantially less accurate. Section 4.1 has also been presented as the motivation for UΔ as a cheap proxy for non-conservativity and energy drift in the revised version, in place of the exact Jacobian-based measure λ, which requires 3N backward passes per structure.
Correlation with λ limited to Orb, and per-system statistics. The reviewer is right about the limited scope. We accordingly extended the Section 4.2 experiments to four architecture/dataset settings spanning Orb and the equivariant eqV2 evaluated on MPtrj and OMat24 [3], with every correlation recomputed within each trajectory so that each value represents a single system. We also extended the NVE simulation time from 5 ps to 50 ps for consistency throughout the paper as suggested by the reviewer, and performed DFT calculations on 100 evenly sampled frames along each trajectory to track the force error εnc.
Median [interquartile range] over 50 trajectories per setting:
UΔ still tracks λ, energy drift, and also the force error well within each trajectory of a single system, across all four settings. This also explains the active learning results below, where finetuning on high-UΔ structures improves energy drift and accuracy together, and considerably more than the MD baseline. UΔ can therefore be interpreted as a practical joint indicator of per-material non-conservativity and force error within an MD trajectory, without DFT labels or the expensive Jacobian-based λ.
A few trajectories fall below the median, in which the drift stays small throughout and the correlation is correspondingly noisy, though still positive:
[1] D. Zhang, H. Bi, F.-Z. Dai, W. Jiang, X. Liu, L. Zhang, and H. Wang. Pretraining of attention-based deep learning potential model for molecular simulation. npj Computational Materials, 10:94, 2024.
[2] F. Bigi, M. F. Langer, and M. Ceriotti. The dark side of the forces: assessing non-conservative force models for atomistic machine learning. In ICML, PMLR 267, 2025.
[3] L. Barroso-Luque, M. Shuaibi, X. Fu, B. M. Wood, M. Dzamba, M. Gao, A. Rizvi, C. L. Zitnick, and Z. W. Ulissi. Open Materials 2024 (OMat24) inorganic materials dataset and models. arXiv:2410.12771, 2024.
[4] Y.-L. Liao, T. Smidt, M. Shuaibi, and A. Das. Generalizing denoising to non-equilibrium structures improves equivariant force fields. Transactions on Machine Learning Research, 2024.
[5] A. R. Tan, S. Urata, S. Goldman, J. C. B. Dietschreit, and R. Gómez-Bombarelli. Single-model uncertainty quantification in neural network potentials does not consistently outperform model ensembles. npj Computational Materials, 9:225, 2023.
[6] L. Kahle and F. Zipoli. Quality of uncertainty estimates from neural network potential ensembles. Physical Review E, 105:015311, 2022.