Bimodal Knowledge Distillation of Bayesian Deep Neural Networks with Mixtures of Gaussian Processes

Bayesian neural networks provide a probabilistic framework for uncertainty quantification through a posterior over model parameters. In this work, we study the predictive structure induced by this posterior and examine how this structure can be used for prediction and uncertainty quantification. From SGLD posterior samples, we observe that, for some inputs, predictive logits form two separated modes that support different classes. This structure identifies the class pair underlying the predictive ambiguity and indicates whether the uncertainty comes from disagreement between these two classes rather than broad ambiguity across classes. Posterior distillation enables efficient inference by compressing posterior samples, but a unimodal approximation can lose this mode structure. To preserve this structure, we propose MoDGF, a posterior distillation method based on a mixture of two Deep Gaussian Factor (DGF) components. Using this component-wise structure, we propose component-guided pairwise specialist inference and within-component epistemic OOD detection. Extensive experiments show that MoDGF preserves SGLD predictive performance across image and language tasks, while its component-wise information improves specialist inference and OOD detection.