- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper evaluates GNN ensembles in node regression, classification and regression in terms of their NLL on a test set and show that they do not improve over a single model in this metric. The authors explore the hypothesis that this is because individual ensemble members collapse to the same hypothesis and identify the smoothing bias of GNNs as the cause.
Critical strengths
- The focuses also on understudied problems like node regression to evaluate uncertainty.
- The paper formally highlights smoothing as a mechanism that forces individual ensembles members to collapse onto the same hypothesis. This is a novel insight that can enable future work on ensembles to be more effective on uncertainty quantification for graphs.
Critical weaknesses
- The insight that ensembles lag behind their counterpart in i.i.d. domains is well known in the literature for uncertainty on graphs [1,2,3,4]. Hence, the main question the paper asks "Do Deep Ensembles Actually Capture Uncertainty" is answered as a "no" by previous work already. The paper also does not acknowledge any of the previous works (the references I provided are just a few examples) that include DEs in their benchmarks and arrive at this conclusion.
- The main proxy for evaluating the quality of an uncertainty estimate is NLL while it is common practice in the literature to instead look at the surrogate tasks of OOD detection under distribution shift and misclassification detection. For example, under distribution shift, it is not clear whether the desired behavior of a model is to output high or low NLL: When the model, for example by chance, makes a correct prediction, NLL will be lower, while a good uncertainty estimator should still flag distribution shift as predictions are unreliable. NLL is not completely detached from uncertainty performance, but using NLL is an uncommon choice in the literature for both i.i.d. and graph data. OOD AUC-ROC should be the main metric for uncertainty estimation unless the paper makes a convincing case as to why to deviate from this well-established practice.
References:
- [1] Stadler, Maximilian, et al. "Graph posterior network: Bayesian predictive uncertainty for node classification." Advances in Neural Information Processing Systems 34 (2021): 18033-18048.
- [2] Fuchsgruber, Dominik, et al. "Uncertainty for active learning on graphs." arXiv preprint arXiv:2405.01462 (2024).
- [3] Fuchsgruber, Dominik, Tom Wollschläger, and Stephan Günnemann. "Energy-based epistemic uncertainty for graph neural networks." Advances in Neural Information Processing Systems 37 (2024): 34378-34428.
- [4] Trivedi, Puja, et al. "Accurate and scalable estimation of epistemic uncertainty for graph neural networks." International Conference on Learning Representations. Vol. 2024. 2024.
Questions for authors
- The Theorem in L.420 bounds the total epistemic uncertainty in terms of the spectral gap. For the non-distribution shift experiments where uncertainty is only evaluated on in-distribution test data, this explains why for these test nodes the smoothing necessarily enforces similar hypothesis (predictions). In, for example, a leave-out-class setting, out-of-distribution nodes cluster strongly together and the connectivity to anchor nodes will be low. Empirically however, 1-4 find poor performance of DEs also on these. Do the authors have any idea or guess as to why that could be? Is there a reason to suspect epistemic collapse here as well or are other principles candidate explanations?