acceptodds
Under review as a conference paper at ICLR 2027

Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?

Abstract

While deep ensembles are widely considered to be the default method for uncertainty quantification in deep learning, their effectiveness for graph-structured data is often simply assumed based on successes in domains like computer vision. We investigate standard deep ensembles specifically for message-passing graph neural networks. Benchmarking across eight datasets representing varied tasks and complexities, we reveal that ensembles provide only marginal improvement over a single model. These marginal gains stem primarily from averaging out optimization noise in point predictions rather than from meaningfully better uncertainty estimates. Through an aleatoric-epistemic decomposition, we identify epistemic collapse: independently trained networks consistently converge to overly similar predictions. Because disagreement is the fundamental mechanism through which ensembles capture epistemic uncertainty, this lack of diversity neutralizes their key advantage. Towards explaining this phenomenon, we derive a bound on epistemic variance in terms of the graph topology and Dirichlet energy budget, linking epistemic collapse to the smoothing bias of message passing. Our results suggest that deep ensemble success does not seamlessly transfer to graph machine learning.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph representation learning, graph neural networks, deep ensembles, uncertainty

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_f5NdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates GNN ensembles in node regression, classification and regression in terms of their NLL on a test set and show that they do not improve over a single model in this metric. The authors explore the hypothesis that this is because individual ensemble members collapse to the same hypothesis and identify the smoothing bias of GNNs as the cause.

Critical strengths
  • The focuses also on understudied problems like node regression to evaluate uncertainty.
  • The paper formally highlights smoothing as a mechanism that forces individual ensembles members to collapse onto the same hypothesis. This is a novel insight that can enable future work on ensembles to be more effective on uncertainty quantification for graphs.
Critical weaknesses
  • The insight that ensembles lag behind their counterpart in i.i.d. domains is well known in the literature for uncertainty on graphs [1,2,3,4]. Hence, the main question the paper asks "Do Deep Ensembles Actually Capture Uncertainty" is answered as a "no" by previous work already. The paper also does not acknowledge any of the previous works (the references I provided are just a few examples) that include DEs in their benchmarks and arrive at this conclusion.
  • The main proxy for evaluating the quality of an uncertainty estimate is NLL while it is common practice in the literature to instead look at the surrogate tasks of OOD detection under distribution shift and misclassification detection. For example, under distribution shift, it is not clear whether the desired behavior of a model is to output high or low NLL: When the model, for example by chance, makes a correct prediction, NLL will be lower, while a good uncertainty estimator should still flag distribution shift as predictions are unreliable. NLL is not completely detached from uncertainty performance, but using NLL is an uncommon choice in the literature for both i.i.d. and graph data. OOD AUC-ROC should be the main metric for uncertainty estimation unless the paper makes a convincing case as to why to deviate from this well-established practice.

References:

  • [1] Stadler, Maximilian, et al. "Graph posterior network: Bayesian predictive uncertainty for node classification." Advances in Neural Information Processing Systems 34 (2021): 18033-18048.
  • [2] Fuchsgruber, Dominik, et al. "Uncertainty for active learning on graphs." arXiv preprint arXiv:2405.01462 (2024).
  • [3] Fuchsgruber, Dominik, Tom Wollschläger, and Stephan Günnemann. "Energy-based epistemic uncertainty for graph neural networks." Advances in Neural Information Processing Systems 37 (2024): 34378-34428.
  • [4] Trivedi, Puja, et al. "Accurate and scalable estimation of epistemic uncertainty for graph neural networks." International Conference on Learning Representations. Vol. 2024. 2024.
Questions for authors
  • The Theorem in L.420 bounds the total epistemic uncertainty in terms of the spectral gap. For the non-distribution shift experiments where uncertainty is only evaluated on in-distribution test data, this explains why for these test nodes the smoothing necessarily enforces similar hypothesis (predictions). In, for example, a leave-out-class setting, out-of-distribution nodes cluster strongly together and the connectivity to anchor nodes will be low. Empirically however, 1-4 find poor performance of DEs also on these. Do the authors have any idea or guess as to why that could be? Is there a reason to suspect epistemic collapse here as well or are other principles candidate explanations?
Reviewer_8R5yOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper identifies a failure cases of deep ensembles (a widely used uncertainty quantification method) for graph data using message-passing graph neural networks (GNNs). Deep ensembles are shown to perform worse than a trivial baseline on a road network benchmark, and are benchmarked on node-level classification and regression datasets (plus one graph-level dataset for classification and regression each) where they are shown to offer limited improvement to single-model methods. The gains they do provide are shown to be mainly from greater smoothness and accuracy of point prediction rather than better-calibrated uncertainty, and the models' predictions tend to converge far more than has been observed on computer visions tasks (which the paper calls epistemic collapse). The epistemic collapse is linked to a smoothness-based bound on disagreement between the ensembled GNN predictions.

Critical strengths

S1. The finding that a technique used in other domains (e.g. vision) doesn't carry over to graphs is a compelling one, and supported by some graph-specific arguments. Precise findings from other domains are contrasted with the largely negative findings in the paper's study.

S2. Care is taken to distinguish better uncertainty quantification from better point estimates, and also decompose the former by kinds of uncertainty.

Critical weaknesses

W1. The paper presents a limited failure case for a certain type of graph task, benchmark, etc. and as things stand should not overclaim that graphs or GNNs themselves are the problem. The empirical results cover standard ensembling and focus on node-level prediction (the only setting where the theoretical results are currently scoped for). While the benchmarked improvements of an ensemble versus a single model are substantially lower than what is reported in the vision domain, there are many possible confounds not currently controlled for including but not limited to different training set sizes, baseline quality, architecture capacity, regularization, ensemble size, etc.

W2. It is unclear that the proposed claim is a true failure case, or that the theoretical analysis explains the observed phenomenon. Predictions that do not widely differ does not necessarily mean epistemic collapse; it could also mean that the data explain the prediction very well. 2) It is not clear how tight the smoothness bound on prediction differences is.

Questions for authors

Q1. How tight are the bounds of the theorem (Eq. 4) in practice? Can the energy budget and anchored spectral gap distinguish settings where ensembles succeed vs collapse? Q2. Do we see epistemic uncertainty fail to rise even in a setting when it provably should (e.g. task-specific information is missing)?

Reviewer_iuQHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper investigates whether standard DEs provide meaningful uncertainty estimates for message-passing graph neural networks. Through experiments across eight graph datasets, the authors find that DEs provide limited improvements over single models, with most gains attributed to averaging point predictions rather than improved uncertainty estimation.
  • The authors identify epistemic collapse, a phenomenon in which independently initialized GNNs produce highly similar predictions. To explain this behavior, the paper derives an upper bound on ensemble predictive variance using graph connectivity and Dirichlet energy, linking limited ensemble diversity to the smoothing bias of message passing.
Critical strengths
  1. Meaningful problem formulation: The paper critically examines standard deep ensembles, which are frequently used as uncertainty quantification baselines. Reassessing their effectiveness specifically for GNNs addresses an important practical question.

  2. Systematic experimental analysis: The evaluation covers diverse node-level and graph-level classification and regression tasks. The DE-R, DE-A, and Jensen's gap analyses are particularly useful for distinguishing improvements in point predictions from uncertainty-related benefits.

  3. Theoretical insight: The proposed variance bound provides an interesting connection between graph topology, Dirichlet energy, and ensemble disagreement. This offers a potentially useful explanation for how the smoothing bias of message passing may limit ensemble diversity.

Critical weaknesses
  1. Interpretation of epistemic collapse: Low disagreement among ensemble members does not necessarily imply poor uncertainty estimation. Models may legitimately agree when sufficient information is available. Although the experiments convincingly demonstrate limited disagreement, they provide less direct evidence that this reflects underestimation of the true epistemic uncertainty.

  2. Limited empirical validation of the theory: The proposed bound relies on assumptions about prediction smoothness and training-node fit. However, the Dirichlet energy budget, anchored spectral gap, and tightness of the bound are not quantitatively evaluated on the benchmark datasets. Consequently, the proposed connection between smoothing bias and the observed epistemic collapse remains insufficiently validated empirically.

  3. Limitations of the Jensen's gap analysis: The Jensen's gap measures disagreement in the probability assigned to the ground-truth class, rather than disagreement across the full predictive distribution. Therefore, a small gap alone does not establish that ensemble members produce nearly identical predictive distributions. Additional direct diversity metrics would strengthen this conclusion.

  4. Limited analysis of causal factors and generalizability: Although the experiments cover multiple datasets, the effects of architectural choices that control smoothing, such as GNN depth and residual connections, are not systematically investigated. Moreover, the theoretical analysis primarily addresses node-level predictions, leaving its relevance to graph-level tasks less clear. Comparisons with bootstrap or architecturally diverse ensembles would also help identify the conditions under which epistemic collapse occurs.

Questions for authors
  1. Can the authors provide stronger evidence that low ensemble disagreement reflects underestimated epistemic uncertainty rather than appropriate agreement among well-trained models? For example, how does uncertainty respond to reduced training data or regions with higher prediction errors?

  2. Can the authors estimate the Dirichlet energy budget, minimum energy, and anchored spectral gap on the benchmark datasets? Comparing the resulting bounds with observed epistemic variance would help establish their practical relevance.

  3. How does ensemble disagreement change when smoothing is systematically controlled through GNN depth or residual connections while keeping the dataset and base architecture fixed?

  4. Beyond Jensen's gap, could the authors report pairwise KL divergence, Jensen-Shannon divergence, or other metrics that directly measure disagreement across the complete predictive distributions?

  5. Under matched ensemble sizes and computational budgets, can bootstrap ensembles or architecturally diverse GNN ensembles mitigate epistemic collapse? Such comparisons would clarify whether the limitation is specific to standard same-architecture ensembles rather than ensembling in general.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.