acceptodds
Under review as a conference paper at ICLR 2027

Bayesian Linear Last Layer for Machine Learning Interatomic Potentials

Abstract

Reliable uncertainty quantification is essential for deploying Machine Learning Interatomic Potentials (MLIPs), especially when molecular dynamics or materials simulations encounter configurations outside the training distribution. Deep ensembles are among the strongest practical baselines for MLIP uncertainty, but training and storing several copies of a modern pretrained model is often prohibitively expensive. We show that Bayesian Linear Last Layers (BLLs) provide a scalable alternative for MLIPs: a single pretrained backbone supplies atomic features, while exact Bayesian inference over the final force-prediction layer gives predictive uncertainties; the construction extends directly to equivariant force heads. However, BLLs are known to be overconfident, so we analyze the sources of this miscalibration and introduce a simple post-hoc recalibration to address the issue. We show that BLLs recalibrated on in-distribution examples are competitive with LoRA ensembles on several foundation backbones for molecules and materials.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Uncertainty Quantification, Machine Learning Interatomic Potentials, Bayesian Last Layer, Post-hoc rescaling, Molecular Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tK8LOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper uses a Bayesian last layer for uncertainty quantification in Machine Learning Interatomic Potentials (MLIPs). The paper also investigates recalibration and comparison to other calibration and finetuning approaches like LoRA. The paper consider one MLIP and one dataset. The validation is well-executed.

Critical strengths

Important topic to know what you don't know when you perform surrogate modelling. Analysis is sound.

Critical weaknesses

Not super novel paper. Does not bring anything fundamentally new to the table. Also, more more models and datasets could have been considered.

Questions for authors

What is the novelty? Why is one model and one dataset enough to conclude form?

Reviewer_rJNdOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The work proposes an uncertainty estimation method for non-conservative machine learning interatomic potentials (MLIPs) by replacing the final force-prediction layer on a pretrained MLIP with a Bayesian linear model. The approach aims to provide a computationally efficient uncertainty estimation alternative to expensive ensemble methods. The work also proposes a post-hoc two parameter rescaling calibration correction for models with frozen backbones. The authors evaluate the method on MPtrj and OMol25 on Orb-v3, eSEN-em, eSEN-md, and OrbMol backbones and compare to other uncertainty quantification techniques including LoRA ensembles, BLIP, SWAG, and last-layer Laplace approximations.

Critical strengths

Application of Bayesian last-layer to foundation potentials: The work demonstrates the application of BLL to pretrained foundation MLIPs.

Empirical evaluation across multiple potentials: The work evaluates several UQ methods, BLL, LoRA ensembles, BLIP, SWAG, and last-layer Laplace approximations, across multiple foundation backbones, Orb-v3-direct, eSEN-sm, eSEN-md, OrbMol, on two datasets, MPtrj and OMol25.

Critical weaknesses

Mixed performance relative to LoRA ensembles: The experimental results do not establish that BLL consistently outperforms LoRA ensembles. For eSEN-md LoRA is better on ECE, Spearman, and NLL. For OrbMol LoRA is better across all reported metrics.

The OOD evaluation on OMOL25 is not genuinely OOD for the backbone: The OMol25 foundation backbones are pretrained on the full OMol25 training set, including configurations later designated as OOD-size and OOD-force for the BLL. Furthermore, the BLL are fit using entirely frozen features from these pretrained backbones. The experiments cannot establish how the methods handle genuine OOD data.

Lack of UQ validation in MD or active learning: The motivation emphasizes the importance of reliable UQ for molecular dynamics or active learning, however, the experiments primarily evaluate static force predictions.

Limited applicability to direct force MLIPs: The formulation only applies to direct force prediction architectures which can have problematic MD instabilities.

Inconsistent training protocols of BLL: The experimental setup differs without explanation substantially between MPtrj and OMol25. On MPtrj, the authors optimize the penultimate force projection jointly with the BLL whereas on OMol25 they fit BLL using entirely frozen features.

Fairness of LoRA ensemble comparison: The LoRA ensembles presented as baseline models modify the message-passing layers while BLL only modifies the force head or its final projection. These differences affect the learned representations and force prediction accuracy potentially affecting the UQ comparison. The results when BLL performs better on UQ tasks (eSEN-sm, orb-v3-direct) the presented LoRA models show significant MAE errors. The comparison may reflect differences in underlying force predictors rather than UQ methods alone.

Unclear and inconsistent calibration protocol: It is difficult to understand what was used for the calibration datasets in the experiments. In particular, Table 1 evaluates the BLL models and the ensemble of three pretrained backbones on OMol25 Neutral Val. The ensemble of three pretrained backbones was recalibrated on the id test while Appendix H states that the BLL recalibration parameters are fit on Neutral Val. This indicates that the BLL models are evaluated on the same dataset used for calibration then compared to an ensemble calibrated on a different dataset. This introduces bias in the ECE and NLL comparison and undermines any conclusion about BLL providing better results than the ensemble.

Lack of clarity: The paper has unclear explanation of calibration datasets, evaluation splits, and baselines. There are also grammatical errors and lack of variable definition. These make it difficult to assess the validity of the experiments.

Minor: L036: missing space L056: define N L090: define N L078: define MCMC acronym L124: define D L129: define E, reuse of N L196: “either part of all of it” did you mean “either part or all of it”? L205: missing space L329: space before comma Should define OOD-Mid and OOD-Far in maintext Use of OOD-Size/OOD-size/OOD Size Numbering of equations in the appendix instead of the main text

Questions for authors

How does the method depend on the availability of representative calibration data? Could the authors provide experiments measuring sensitivity to calibration-set size and composition? Uncertainty-error correlation on static datasets does not necessarily translate to the motivating UQ tasks of detecting MD instability and active learning. Could the authors use the proposed UQ method in an active learning loop or MD failure-detection benchmark? What is the inference cost comparison across UQ models and how much does inference cost increase with the addition of the BLL? Since the OMol25 OOD configurations were included in the backbone pretraining, can the authors evaluate BLL on configurations not included in the training of the backbone and Bayesian layer? The work presents two strategies for fitting the BLL: jointly optimizing the penultimate force projection and BLL parameters (MPtrj) or fitting the BLL on frozen features (OMol25). Why do the authors select different strategies for each dataset instead of applying both methods? Why are SWAG and last-layer Laplace evaluated only on MPtrj and OMol25, respectively? What is the motivation for comparing BLL against an ensemble of three different pretrained foundation models (eSEN-sm, eSEN-md, and OrbMol) in Table 1? Such ensembles may capture differences in architecture, model capacity, and training rather than provide a controlled comparison of UQ methods. Why is the ensemble of three different pretrained foundation models (eSEN-sm, eSEN-md, and OrbMol) in Table 1 calibrated using the ID test set, inconsistent with all other model calibration? What does the shading on the lines indicate in Figure 2? How does fitting the BLL affect model accuracy on MPtrj? Table 5 reports this comparison for OMol25 but no equivalent results are provided for MPtrj. The discussion in the appendix states that the recalibration procedure fails on rMD17. Why are these results not shown in the main text? Appendix Table 9 reports additional comparison of UQ methods on other datasets. Why are these evaluations not discussed in the main text?

Reviewer_itLLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: The work evaluates the Bayesian linear last layer (BLL) method for efficient uncertainty quantification in atomic machine learning potentials and force fields. A generalization of BLLs for equivariant force heads is introduced. Sources of uncertainty miscalibration are investigated and a simple post-hoc recalibration approach is introduced. The BLL approach is compared in empirical evaluations against different methods such as LoRA ensembles, variational-dropout BLIPs and the SWAG method. The results from the experiments are promising, with BLLs consistently being at or near the top-performing method.

Critical strengths

Positives: I think the paper deals with an important open question. Cheap and reliable uncertainty quantification methods in machine learning force fields would be of great benefit for foundation models, where the computational overhead of ensemble-based methods poses a significant challenge. Further, the results of BLLs look good in the comparisons, which is a novel and significant result.

Critical weaknesses

Weaknesses: I think the structure and clarity of the writing could be significantly improved. Most importantly, the authors need to more clearly establish the novel contributions of the work and clearly distinguish which elements are established methodology, which are adaptations to this setting, and which constitute new methodological advances relative to the closest prior work. From what I understand, BLL methods are a well-established method in uncertainty quantification for deep learning models and the authors themselves write that “the construction extends directly to equivariant force heads”. Uncertainty calibration is also performed with standard methodology. Therefore, the methodological contribution appears very limited. The main novelty and contribution, in my estimation, is the evaluation performed. The evaluation looks decent overall, but I think two things could improve it. First of all, perhaps the most important property of the uncertainty metric is the ability to detect structures with a large prediction error. This is often not captured well by averaged metrics like log-likelihoods. If, on the evaluated dataset, most structures have a small prediction error, then the contribution to the mean log-likelihoods of the low-error structures can outweigh the contribution of the higher-error structures, even if each high-error structure individually has a higher individual contribution. While the effect of distribution shift was investigated, I think evaluating something like ROC-AUC scores for detecting force errors above some threshold would provide significant additional value. Further, because the evaluation is the main contribution of the paper, I think a broader comparison against additional cheap uncertainty quantification methods should be considered or it should be discussed why they are not relevant here. For example, several methods in the following publications seem relevant: https://www.nature.com/articles/s41524-023-01180-8 , https://www.nature.com/articles/s41524-024-01254-1.

Questions for authors

Additional Remarks Independent of my Recommendation: If the paper deals with direct force field architectures, is Machine Learning Interatomic Potential (MLIP) the correct terminology, since the forces are not derived from a potential energy and may not even be conservative? Maybe Machine Learning Force Fields (MLFFs) would be a better description. Additionally, I am not sure if this conference has the optimal readership for this paper. I think it can fit in, but most of the research in this field, in particular application-focused research, is published in domain-specific journals, which might be better suited to get this paper read by the relevant research community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.