acceptodds
Withdrawn from ICLR 2027

Bimodal Knowledge Distillation of Bayesian Deep Neural Networks with Mixtures of Gaussian Processes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Bayesian neural networks provide a probabilistic framework for uncertainty quantification through a posterior over model parameters. In this work, we study the predictive structure induced by this posterior and examine how this structure can be used for prediction and uncertainty quantification. From SGLD posterior samples, we observe that, for some inputs, predictive logits form two separated modes that support different classes. This structure identifies the class pair underlying the predictive ambiguity and indicates whether the uncertainty comes from disagreement between these two classes rather than broad ambiguity across classes. Posterior distillation enables efficient inference by compressing posterior samples, but a unimodal approximation can lose this mode structure. To preserve this structure, we propose MoDGF, a posterior distillation method based on a mixture of two Deep Gaussian Factor (DGF) components. Using this component-wise structure, we propose component-guided pairwise specialist inference and within-component epistemic OOD detection. Extensive experiments show that MoDGF preserves SGLD predictive performance across image and language tasks, while its component-wise information improves specialist inference and OOD detection.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian neural networks, predictive multimodality, knowledge distillation, mixture of Gaussian processes, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dxf1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers posterior distillation while preserving the predictive structure. The previous DGF model approximates the teacher distribution by a single Gaussian process and thus cannot explicitly represent bimodal predictive structure. Therefore the authors propose MoDGF, which uses a mixture of two DGF components as the student model to represent bimodal predictive structure in the distilled distribution.

The motivation is clear.

For the experimental part, the authors first evaluate whether MoDGF preserves the classification performance of stochastic gradient Langevin dynamics (SGLD) on image classification and language reasoning tasks. Second, they evaluate whether component-guided pairwise specialist inference improves classification performance. Third, they examine whether within-component epistemic uncertainty provides better signals for OOD detection.

Critical strengths

None.

Critical weaknesses

However, the experiments did not provide sufficient evidence on the performance superiority. From Table 1 and Table 2, the performance improvements of MoDGF with respect to single DGF is quite minor. Probably, more datasets should be used to validate the effectiveness of MoDGF.

It seems that Table 4 shows a larger improvement over single DGF, but only with two similar datasets.

Questions for authors

As shown in Figure 1(a), predictive bimodality is frequently observed (more than 10% in the test data) across CIFAR-10, CIFAR-100, and Tiny-ImageNet. My question is: Is the phenomenon of predictive bimodality only been seen with SGLD?

Reviewer_uRwEOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper explores how to condense multiple posterior samples generated by SGLD into a more compact student model. The authors observed that certain posterior logits of the input present a bimodal structure, for instance, one part of the model supports category A, while the other part supports category B. To preserve this structure, the authors proposed a hybrid model MoDGF consisting of two DGFs, and used the information from the two components for category selection by experts and OOD detection.

Critical strengths

The accuracy, ECE, and NLL of MoDGF on image and language tasks are close to those of SGLD; The two-component-guided classification improves the accuracy of experts on CIFAR-10 and CIFAR-100; When using within-component epistemic uncertainty for OOD detection, the AUROC improves on the four data combinations tested in the paper;

Critical weaknesses

The comparisons are unfair in multiple experiments, I believe the authors deliberately conceal this fact. For instance, Single DGF and MoDGF use different shared parameters in their main experiments, yet the differences are only reported in the appendix. This is highly unprofessional and dishonest.

The writing lacks rigor and contains inconsistencies throughout, both in the methods and in the experimental results. The differences between definition of bimodal and its actual usage is particularly confusing, and the inconsistencies in the data make the results difficult to believe.

Questions for authors

Table 9 is the supplementary data for the same experiment as in Table 4. Why are there discrepancies between the two sets of data? In Table 4, MoDGF (Epistemic) -cifar10-tinyimagenet = 86.76, while in Table 9, it is = 86.16.

In the results reported in the main paper, Single DGF uses a DGF with r=8, while MoDGF uses two DGFs each with r=16. This is an unfair comparison. Moreover, the fact that Single DGF uses a DGF with r=8 was quietly mentioned in the appendix and not presented in the main paper.

Three different expert construction methods were used, making it difficult to prove the contribution of MoDGF component routing itself.

The actual reasoning delay of MoDGF was not reported.

Reviewer_i3u9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper started from an oboservation that, in SGLD posterioro samples, a great nuber of the predictions has bimodes. To preserve these bimodal predictions, the paper proposed, MoDGF, a mixture of two DGFs with a structured mean-field EM training strategy. Based on this, the paper also proposed a pairwise specialist component for the cases that two DGF diasagree with each oher. Further more, the paper proposed a new EU decompostition beyond classific uncertainty decompostion and provide semantic interpretation for them. Experiemnts across image classification and language tasks show competitive performance.

Critical strengths
  1. The proposed EU decomposition has meaningful interpretation and application. The chain rule of mutual information is trivial, but the paper provises an interesting interpretation: (1) EU_between is the disagreement between distinct predictive component; (2) EU_within is more close a true OOD singal. And experiments show that different EUs perform differently on downstream tasks, like EU_within is good for OOD, while raw EU performs best in active learning.

  2. The experiments are well-designed and comprehensive with promising results.

Critical weaknesses
  1. Limited novelty and insufficient justification. The method design are theoretical results are not new in the field. The general problem of preserving multimodal predictive distributions during distillations has been extensive studiedd in the field; and it is shown that flow-based methods can learn more comples distributions with multi modes. The proposed method limites the student to two gaussian components without explicitly explaining why this particular representation is better than Gaussian mixture or more flexible distribution representations.

I can see the DGF can model cross-input correlations, but the experiments did not evaluate this ability. And the teacher was restricted to SGLD, which is a single-model based bayesian method, this teacher may naturally stuck in one mode, and so the student.

  1. Key experiments are not evaulated fairly, and lacking important evaluations. (1) in the main experiments, MoDGF uses r=16, wile single DGF only uses r =8; combining Figure 6, it seems that MoDGF does not show better results than single DGF when r = 8. (2) The baseline EDFM may not be well-tuned, which is even worse than single DGF on CIfar-100 and ARC-C.. (3) see "Questions for Authors"
Questions for authors
  1. see weakness

  2. The authors should compare component-guided routing with a simple top-2 prediction-based routing strategy using the exact same specialist backbone and binary heads

  3. A controlled experiment with known predictive modes and independently adjustable within-mode variability and between-mode separation would help verify whether MoDGF recovers the intended uncertainty components.

  4. The authros are expected to show generalized results with difffernt teacher models. And the bimode was discovered and defined only on SGLD, which may not be a strong phenomonen acrossing different bayesian models

  5. More baseline methods are needed, like deep ensemble/other bayesian models for most of the tasks; and for specific tasks, like MSP, feature-based, distance-based methods for OOD. If the main evidence is MoDGF matches SGLD and improves over single DGF, the contribution appears really limited. And even the improvement over single DGF can be explained by larger latent dimension or model capacity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.