- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper considers posterior distillation while preserving the predictive structure. The previous DGF model approximates the teacher distribution by a single Gaussian process and thus cannot explicitly represent bimodal predictive structure. Therefore the authors propose MoDGF, which uses a mixture of two DGF components as the student model to represent bimodal predictive structure in the distilled distribution.
The motivation is clear.
For the experimental part, the authors first evaluate whether MoDGF preserves the classification performance of stochastic gradient Langevin dynamics (SGLD) on image classification and language reasoning tasks. Second, they evaluate whether component-guided pairwise specialist inference improves classification performance. Third, they examine whether within-component epistemic uncertainty provides better signals for OOD detection.
Critical strengths
None.
Critical weaknesses
However, the experiments did not provide sufficient evidence on the performance superiority. From Table 1 and Table 2, the performance improvements of MoDGF with respect to single DGF is quite minor. Probably, more datasets should be used to validate the effectiveness of MoDGF.
It seems that Table 4 shows a larger improvement over single DGF, but only with two similar datasets.
Questions for authors
As shown in Figure 1(a), predictive bimodality is frequently observed (more than 10% in the test data) across CIFAR-10, CIFAR-100, and Tiny-ImageNet. My question is: Is the phenomenon of predictive bimodality only been seen with SGLD?