acceptodds
Withdrawn from ICLR 2027

Multi-Fidelity Training Reshapes Uncertainty Decomposition in Atomistic Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-fidelity training, which combines abundant low-fidelity data with scarce high-fidelity labels, is increasingly common in atomistic modeling, but its effect on predictive uncertainty remains largely unexplored. We investigate this question using deep ensembles of mean-variance MACE models trained on ANI-1x, contrasting high-fidelity training from scratch with low-fidelity pretraining followed by fine-tuning. We characterize aleatoric, epistemic, and total uncertainty throughout training and under two distribution shifts: unseen molecular systems and higher-energy configurations. We find that pretraining reduces prediction error and improves uncertainty-based ranking across all test sets, but does not yield well-calibrated predictive variances; calibration deteriorates more severely under energy extrapolation. Training-dynamics analysis localizes most of this calibration degradation to high-fidelity optimization, during which the learned aleatoric variance becomes overconfident on held-out data. Moreover, multi-fidelity fine-tuning produces a more effective uncertainty-based acquisition function than high-fidelity training from scratch. These findings show that uncertainty ranking and calibration capture distinct properties and must be assessed separately in multi-fidelity atomistic models, especially under extrapolation.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Atomistic Models, Multi-Fidelity, Uncertainty Decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Q2LTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how multi-fidelity training affects predictive uncertainty in atomistic machine learning models. Using mean–variance MACE deep ensembles, the authors compare high-fidelity training from scratch with low-fidelity pretraining followed by high-fidelity fine-tuning. They evaluate prediction accuracy, uncertainty ranking, and calibration under both in-distribution settings and two types of distribution shift: unseen molecular systems and higher-energy configurations.

The main findings suggest that low-fidelity pretraining improves predictive accuracy and uncertainty-based error ranking, but does not consistently improve uncertainty calibration, particularly under energy extrapolation. By analyzing training dynamics, the authors show that calibration degradation occurs primarily during high-fidelity optimization and is associated with increasingly overconfident aleatoric uncertainty estimates. The paper further demonstrates that uncertainty estimates from pretrained models can provide more effective guidance for uncertainty-based data acquisition.

Critical strengths
  1. Systematic empirical investigation of uncertainty in multi-fidelity learning.

The paper addresses an interesting and practically relevant question that has received relatively limited attention. Rather than evaluating uncertainty only at convergence, the authors examine its evolution throughout pretraining and fine-tuning. The separate analysis of uncertainty ranking, calibration, and aleatoric and epistemic components provides a more detailed understanding of model behavior. The evaluation across two types of distribution shift and five independent data splits further strengthens the empirical analysis.

  1. Interesting findings with practical implications.

The observation that improved prediction accuracy and uncertainty ranking do not necessarily lead to better calibration is important for reliable scientific machine learning. The paper also identifies distinct behaviors of aleatoric and epistemic uncertainty during high-fidelity optimization. Furthermore, the uncertainty-guided acquisition experiment demonstrates the practical usefulness of uncertainty ranking, even when the predicted uncertainty magnitudes are not well calibrated.

Critical weaknesses
  1. Limited experimental validation of the proposed explanation for uncertainty degradation.

Although the paper provides a detailed empirical characterization of uncertainty evolution, the underlying mechanism is not sufficiently established. The authors suggest that the learned aleatoric variance becomes overconfident because the variance head follows decreasing training residuals while errors on held-out data do not decrease accordingly. While this explanation is plausible and supported by the observed training dynamics, the paper lacks targeted ablation experiments to verify it. For example, resetting the variance head during fine-tuning or investigating alternative training objectives could help distinguish this explanation from other effects related to optimization and overfitting. Consequently, the work provides valuable empirical observations, but the mechanistic understanding remains somewhat limited.

  1. Limited generalizability across models, uncertainty estimators, and multi-fidelity training strategies.

The experiments are primarily conducted using a single model architecture (MACE), one dataset (ANI-1x), and one uncertainty estimation approach (mean–variance deep ensembles). In addition, the multi-fidelity experiments focus on a sequential pretraining–fine-tuning strategy. It remains unclear whether the observed uncertainty behavior is a general property of multi-fidelity learning or is specific to this combination of model architecture, uncertainty estimator, and training procedure. Other multi-fidelity strategies, such as joint training, multi-head learning, or alternating optimization across fidelity levels, might exhibit different uncertainty dynamics. Investigating or discussing these alternatives would help clarify the scope of the conclusions.

  1. Novelty Clarification.

Several aspects of the reported findings are closely related to prior observations on the distinction between uncertainty ranking and calibration, the unreliability of aleatoric uncertainty under distribution shift, and the optimization difficulties of Gaussian NLL-based variance estimation. While the analysis of uncertainty evolution during low-to-high-fidelity training appears to offer additional empirical insights, the manuscript would benefit from a clearer explanation of which findings are specific to the multi-fidelity setting and how they extend existing knowledge. In particular, Bilbrey et al. (2025) reported related overconfidence behavior in ensemble-based uncertainty estimates for MACE foundation models and appears relevant to this discussion.

Questions for authors

Similar to the weakness:

  1. Regarding the mechanism of calibration degradation: Could the authors provide additional ablation experiments to better validate the proposed explanation for the overconfidence of aleatoric uncertainty? For example, would resetting the variance prediction head at the beginning of high-fidelity fine-tuning, or using a different training objective, change the observed calibration dynamics?

  2. Regarding the generality of the findings: To what extent do the observed uncertainty behaviors depend on the sequential low-fidelity pretraining and high-fidelity fine-tuning strategy? Would similar ranking–calibration trade-offs be expected under joint multi-fidelity training or multi-head approaches? Additional comparisons, or a more detailed discussion of the expected differences, would help establish whether these findings reflect a broader property of multi-fidelity learning.

1 comment

I will maintain my score. And I personally suggest improving the presentation of Figures 2 and 3, as the current plots do not effectively communicate the key findings. Alternative visualizations, such as heatmaps or dumbbell plots for Figure 2 and delta plots for Figure 3, could make the comparisons more intuitive and highlight the main results more clearly.

Reviewer_j1p7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates uncertain assessment for models trained on low and high fidelity data in the surrogate molecular energy modeling. The paper is not proposing any new methodology or datasets. Instead it is addressing an important question about the combination of different levels of theory data and how that affects uncertainty assessment and calibration. The validation is thorough although as always more and different could have been done.

Critical strengths

A timely investigation and thorough of the combination of different quality datasets.

Critical weaknesses

Only one model (MACE) and only two datasets used. Not clear that the training setup: low fidelity pretraining and high fidelity finetuning is the right approach. The two datasets come from different distributions with systematic biases and if I understand correctly, the fidelity label is not given as input to the model. This seems to be a missed opportunity because it would allow the model to learn more and also potentially be better calibrated overall.

Questions for authors

Better arguments for the limitations in the model, data and training strategy choices.

Reviewer_ge6XOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines how pretraining atomistic energy models on low-fidelity data affects uncertainty estimates through ensembling. In particular, they train an ensemble of MACE models with variance heads on a dataset containing DFT energy estimates and, for a smaller subset, more accurate coupled-cluster (CC) estimates. Then they compare using CC data alone vs. pretraining using DFT and fine-tuning using CC on OOD systems and OOD configurations of ID systems. They report that pretraining generally improves uncertainty ranking but not calibration. They also monitor these metrics during training and compare them for uncertainty-based active acquisition.

Critical strengths

The write-up is readable and avoids the clarity issues typical of heavily LLM-assisted writing.

The results are clearly presented, and it makes good use of figures.

The choice of dataset for the multi-fidelity study seems reasonable, and the data splits make sense.

The experiments using uncertainty in active learning are a nice addition.

Critical weaknesses

The study's motivation is weak, and its scope is quite narrow. I may have missed a trend in this area, but I don’t think using high-fidelity data for fine-tuning is common right now, so the study of ensemble uncertainty in this setting is not well motivated, and on top of this, the reported findings are not unexpected.

The scope of experiments is limited to a specific model (MACE) on a specific dataset.

All experiments use only energies, and they ignore forces, which are a stronger training signal and are widely used to train MLIPs. So the conclusions are further limited to this less common training setting.

The models being compared (DFT+CC vs CC) are contrasted under a specific rather arbitrary training protocol: a model is pretrained for a certain number of epochs, with optional fine-tuning at a lower learning rate on a smaller set of higher-fidelity data. A proper study could match these settings in total compute budget (training steps), consider various model sizes given data availability, use optimal learning rates, and also report pretrain-only results.

Given that this is a deterministic task, reporting aleatoric uncertainty metrics doesn’t make much sense. The paper correctly notes that this variance component could model approximation or other errors, yet it still reports it as aleatoric uncertainty throughout.

Minor: The champion figure has a white band in two of my PDF viewers but I’m not sure if this is a problem on my side.

Questions for authors

LF-only results are currently not reported. Is LF-only significantly worse than LF+HF or HF-only?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.