- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how multi-fidelity training affects predictive uncertainty in atomistic machine learning models. Using mean–variance MACE deep ensembles, the authors compare high-fidelity training from scratch with low-fidelity pretraining followed by high-fidelity fine-tuning. They evaluate prediction accuracy, uncertainty ranking, and calibration under both in-distribution settings and two types of distribution shift: unseen molecular systems and higher-energy configurations.
The main findings suggest that low-fidelity pretraining improves predictive accuracy and uncertainty-based error ranking, but does not consistently improve uncertainty calibration, particularly under energy extrapolation. By analyzing training dynamics, the authors show that calibration degradation occurs primarily during high-fidelity optimization and is associated with increasingly overconfident aleatoric uncertainty estimates. The paper further demonstrates that uncertainty estimates from pretrained models can provide more effective guidance for uncertainty-based data acquisition.
Critical strengths
- Systematic empirical investigation of uncertainty in multi-fidelity learning.
The paper addresses an interesting and practically relevant question that has received relatively limited attention. Rather than evaluating uncertainty only at convergence, the authors examine its evolution throughout pretraining and fine-tuning. The separate analysis of uncertainty ranking, calibration, and aleatoric and epistemic components provides a more detailed understanding of model behavior. The evaluation across two types of distribution shift and five independent data splits further strengthens the empirical analysis.
- Interesting findings with practical implications.
The observation that improved prediction accuracy and uncertainty ranking do not necessarily lead to better calibration is important for reliable scientific machine learning. The paper also identifies distinct behaviors of aleatoric and epistemic uncertainty during high-fidelity optimization. Furthermore, the uncertainty-guided acquisition experiment demonstrates the practical usefulness of uncertainty ranking, even when the predicted uncertainty magnitudes are not well calibrated.
Critical weaknesses
- Limited experimental validation of the proposed explanation for uncertainty degradation.
Although the paper provides a detailed empirical characterization of uncertainty evolution, the underlying mechanism is not sufficiently established. The authors suggest that the learned aleatoric variance becomes overconfident because the variance head follows decreasing training residuals while errors on held-out data do not decrease accordingly. While this explanation is plausible and supported by the observed training dynamics, the paper lacks targeted ablation experiments to verify it. For example, resetting the variance head during fine-tuning or investigating alternative training objectives could help distinguish this explanation from other effects related to optimization and overfitting. Consequently, the work provides valuable empirical observations, but the mechanistic understanding remains somewhat limited.
- Limited generalizability across models, uncertainty estimators, and multi-fidelity training strategies.
The experiments are primarily conducted using a single model architecture (MACE), one dataset (ANI-1x), and one uncertainty estimation approach (mean–variance deep ensembles). In addition, the multi-fidelity experiments focus on a sequential pretraining–fine-tuning strategy. It remains unclear whether the observed uncertainty behavior is a general property of multi-fidelity learning or is specific to this combination of model architecture, uncertainty estimator, and training procedure. Other multi-fidelity strategies, such as joint training, multi-head learning, or alternating optimization across fidelity levels, might exhibit different uncertainty dynamics. Investigating or discussing these alternatives would help clarify the scope of the conclusions.
- Novelty Clarification.
Several aspects of the reported findings are closely related to prior observations on the distinction between uncertainty ranking and calibration, the unreliability of aleatoric uncertainty under distribution shift, and the optimization difficulties of Gaussian NLL-based variance estimation. While the analysis of uncertainty evolution during low-to-high-fidelity training appears to offer additional empirical insights, the manuscript would benefit from a clearer explanation of which findings are specific to the multi-fidelity setting and how they extend existing knowledge. In particular, Bilbrey et al. (2025) reported related overconfidence behavior in ensemble-based uncertainty estimates for MACE foundation models and appears relevant to this discussion.
Questions for authors
Similar to the weakness:
-
Regarding the mechanism of calibration degradation: Could the authors provide additional ablation experiments to better validate the proposed explanation for the overconfidence of aleatoric uncertainty? For example, would resetting the variance prediction head at the beginning of high-fidelity fine-tuning, or using a different training objective, change the observed calibration dynamics?
-
Regarding the generality of the findings: To what extent do the observed uncertainty behaviors depend on the sequential low-fidelity pretraining and high-fidelity fine-tuning strategy? Would similar ranking–calibration trade-offs be expected under joint multi-fidelity training or multi-head approaches? Additional comparisons, or a more detailed discussion of the expected differences, would help establish whether these findings reflect a broader property of multi-fidelity learning.
I will maintain my score. And I personally suggest improving the presentation of Figures 2 and 3, as the current plots do not effectively communicate the key findings. Alternative visualizations, such as heatmaps or dumbbell plots for Figure 2 and delta plots for Figure 3, could make the comparisons more intuitive and highlight the main results more clearly.