- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose a way to set the prior precision/variances when training Bayesian Neural Networks with variational inference.
The main idea is that starting from a deterministic solution for the parameters,
the authors estimate the influence of each individual parameter to the negative log-likelihood loss.
The intuition is that: the parameters that have higher influence on the loss are assigned higher variance
allowing them to move more freely away from the pretrained solution
Less sensitive parameters are tightly more anchored
The sensitivity score are then mapped to exact precision scores.
Experimentally their method outperforms other types of precision initialization in various benchmarks
Critical strengths
- The idea of using influence (pruning sensitivity ) as a source of information when designing a prior is up to my knowledge quite novel
- The method is presented clearly, with appropriate theoretical motivation. The local analysis in Section 2.2 explains how prior precision controls both the posterior mean displacement and the posterior variance.
- The experimental evaluation is adequate. It covers CNN and ViT architectures, in and out of distribution datasets up to ImageNet-1K, and it reports several metrics as well.
Critical weaknesses
Dependence on the mean-field posterior
- The posterior and the prior choice in variational inference interact strongly. All the experiments use solely MFVI. While someone will logically assume that the benefits of the influence-guided prior will curry to mere correlated posteriors, it might also not be the case, I encourage the authors to adopt a correlated posterior as well
- Tomczak et al., Efficient Low Rank Gaussian Variational Inference for Neural Networks (NeurIPS 2020)
Restriction to variational inference
- The approximation framework used throughout the paper is solely variational inference. While VI is quite poplar, it would be beneficial to show that the proposed prior generalizes to to other inference methods as well. The Laplace approximation is a natural candidate: the prior is already centered at a pretrained solution and the GNN is already computed.
This will also provide the marginal-likelihood approach as a stronger EB baseline for prior precision selection
- Immer, Alexander, et al., Scalable marginal likelihood estimation for model selection in deep learning. (ICML 2021)
Missing discussion of function-space priors
- The related-work discussion is fairly comprehensive on parameter-space priors, but it does not mention the line of work on specifying priors in function space rather than parameter space. This is directly relevant, since the paper’s motivation is that meaningful priors are hard to express at the level of individual weights. I suggest the authors discuss, for example:
- Rudner et al., Function-Space Regularization in Neural Networks: A Probabilistic Perspective (ICML 2023)
- Tran et al., All You Need is a Good Functional Prior for Bayesian Deep Learning (JMLR 2022)
MAP reference in the main results.
- The pretrained deterministic (MAP) solution, which is also the prior mean, should be included as a reference row in the main tables. The paper does not aim for SOTA results, which is fine, but a MAP baseline is needed to verify that the VI models are reasonably trained and to show what the Bayesian treatment adds over the point estimate.
Recommendation: I would be willing to raise my score if the authors adequately address the weaknesses raised above.
Questions for authors
How do the learned posterior variances relate to the sensitivity ranks after training? Do high-sensitivity parameters actually retain larger variance, as the motivation in Section 2.2 suggests?
7 comments
Response to Reviewer Pjus (2-1)
Comment 2: Restriction to Variational Inference
Answer.
We thank the reviewer for this valuable suggestion. Extending our prior design beyond variational inference is a question we had also considered during the development of this work, particularly regarding the mathematical consistency of applying Laplace approximation to a pretrained solution.
1. Mathematical Consideration
Unlike MFVI, which directly optimizes a variational posterior, Laplace approximation constructs a Gaussian approximation around a maximum a posteriori (MAP) solution. Therefore, applying our empirical prior to Laplace approximation requires ensuring that the expansion point corresponds to a stationary point of the resulting posterior.
To clarify this issue, let the negative log-likelihood be .
The corresponding negative log-posterior can be expressed, up to an additive constant, as
where denotes the regularization term induced by the prior distribution.
The MAP solution is defined as , which must satisfy the first-order stationary condition .
However, our original empirical prior is centered at the pretrained solution :
For this Gaussian prior, the regularization term becomes
Consequently, evaluating the negative log-posterior gradient at gives
\nabla_{\mathbf{w}}\mathcal{J}(\mathbf{w}^{\ast}) \=\nabla_{\mathbf{w}}\mathcal{L}_{\mathrm{NLL}}(\mathbf{w}^{\ast})+\nabla_{\mathbf{w}}\mathcal{R}_{\mathrm{ours}}(\mathbf{w}^{\ast})\ \=\nabla_{\mathbf{w}}\mathcal{L}_{\mathrm{NLL}}(\mathbf{w}^{\ast})+\Lambda_{\mathrm{ours}}(\mathbf{w}^{\ast}-\mathbf{w}^{\ast})\ \=\underbrace{\nabla_{\mathbf{w}}\mathcal{L}_{\mathrm{NLL}}(\mathbf{w}^{\ast})}_{\mathbf{g}}. \end{aligned}$$ Importantly, $\mathbf{g}$ does not necessarily vanish, since deterministic pretraining typically involves regularization in addition to the negative log-likelihood. Consequently, the pretrained solution $\mathbf{w}^{\ast}$ is not necessarily a stationary point of the posterior defined by our original empirical prior. We therefore identified two possible approaches to construct a mathematically consistent Laplace approximation: **Prior mean correction**: Adjust the prior mean so that the pretrained solution $\mathbf{w}^{\ast}$ remains a stationary point of the resulting posterior. **MAP reoptimization**: Retain the original prior mean $\mathbf{w}^{\ast}$ and reoptimize the MAP solution under the proposed prior before constructing the Laplace approximation. Both approaches are mathematically valid. However, the second requires additional optimization and generally moves the posterior mode away from the original pretrained solution. We therefore consider prior mean correction a more practical choice for post-hoc inference, as it allows us to preserve the pretrained solution without additional MAP optimization.Response to Reviewer Pjus (2-2)
2. Prior Mean Correction
To preserve as the expansion point, we introduce a corrected prior mean , while keeping the proposed prior precision matrix unchanged.
Specifically, we define the corrected prior as
We require the negative log-posterior gradient to vanish at : .
Substituting the corrected prior gives
Using , we obtain .
Therefore, the corrected prior mean is
This correction ensures that the prior gradient exactly cancels the negative log-likelihood gradient at , making the pretrained solution a stationary point of the resulting posterior.
Under a local quadratic approximation of the negative log-likelihood around , the corrected negative log-posterior becomes
where denotes the Hessian of the negative log-likelihood at . Using the positive semidefinite GGN approximation in place of , the corresponding local Gaussian approximation is
For the full-network diagonal Laplace approximation used in our experiments, this reduces to
Importantly, this formulation preserves our sensitivity-based precision allocation while ensuring that the pretrained solution is a consistent center for the local Laplace approximation.
3. Experimental Results
Following this formulation, we evaluated our method under full-network diagonal Laplace approximation using the linearized (GLM) predictive distribution with 30 Monte Carlo samples.
We conducted experiments on CIFAR-10 and ImageNet-1K across CNN and ViT architectures. As a baseline, we used the post-hoc empirical Bayes (EB) approach implemented in Laplace Redux (Daxberger et al., 2021), which optimizes Gaussian prior precision hyperparameters through marginal likelihood maximization while retaining a zero-mean prior. In contrast, our approach uses the proposed prior precision with the prior mean correction derived above.
| Dataset | Architecture | Method | Acc ↑ | NLL ↓ | ECE ↓ | AUROC ↑ | FPR95 ↓ |
|---|---|---|---|---|---|---|---|
| CIFAR-10 | CNN | Ours | 0.915 | 0.255 | 0.017 | 0.864 | 0.634 |
| EB | 0.900 | 0.535 | 0.240 | 0.859 | 0.648 | ||
| CIFAR-10 | ViT | Ours | 0.790 | 0.616 | 0.040 | 0.756 | 0.791 |
| EB | 0.773 | 0.809 | 0.195 | 0.773 | 0.786 | ||
| ImageNet-1K | CNN | Ours | 0.801 | 1.420 | 0.407 | 0.699 | 0.859 |
| EB | 0.757 | 1.582 | 0.445 | 0.308 | 0.991 | ||
| ImageNet-1K | ViT | Ours | 0.721 | 1.220 | 0.120 | 0.792 | 0.798 |
| EB | 0.709 | 1.462 | 0.277 | 0.733 | 0.868 |
The results show that our method improves accuracy, NLL, and ECE over the EB baseline across all four settings. For example, on CIFAR-10 CNN, our method reduces NLL from 0.535 to 0.255 and ECE from 24.0% to 1.7%. Similar improvements are observed on ImageNet-1K, where our method reduces NLL from 1.462 to 1.220 for ViT and from 1.582 to 1.420 for CNN. Improvements in OOD detection are also observed in three of the four settings, although they are not uniform across all architectures.
Response to Reviewer Pjus (1)
We thank the reviewer for the thoughtful and constructive feedback, as well as for recognizing the novelty of our approach and the strengths of our experimental evaluation. We have carefully considered the concerns raised and revised the manuscript to address them. All changes in the revised manuscript are highlighted in red. Below, we provide point-by-point responses and describe the corresponding revisions.
Comment 1: Generalization Beyond Mean-Field Variational Inference
Answer.
Following the low-rank Gaussian variational inference framework of Tomczak et al. (2020), we consider a correlated Gaussian posterior with covariance Σϕ=D+UU⊤, where U∈Rd×r. We evaluate two configurations: (i) a learnable diagonal covariance D and (ii) a fixed diagonal covariance D=e−20I, with rank r=8. Both configurations are evaluated on CIFAR-10 using CNN and ViT architectures. The results are reported in Appendix A.6 (Tables A5 and A6).
1. Learnable Diagonal Covariance
As shown in Table A5, our method achieves the lowest NLL and ECE for both architectures while maintaining competitive accuracy and OOD detection performance. For CNN, the proposed prior also achieves the highest AUROC and lowest FPR95. For ViT, the OOD results are comparable to those of λmaxI, with only marginal differences. These results suggest that the proposed prior precision allocation remains effective under correlated Gaussian posteriors with learnable diagonal covariance.
Table A5: Correlated Gaussian posterior (r=8) with learnable diagonal covariance on CIFAR-10. OOD detection uses predictive entropy on TinyImageNet.
2. Fixed Diagonal Covariance
We further examine a more constrained covariance parameterization by fixing D=e−20I, while retaining the low-rank component. Table A6 reports the corresponding results.
Under the fixed diagonal configuration, the relative performance changes. For CNN, EB-Diag achieves the lowest NLL and highest AUROC, while the correlated prior achieves the lowest ECE. The proposed prior maintains competitive accuracy but no longer achieves the best NLL or calibration. For ViT, the proposed prior achieves the highest accuracy and lowest NLL, although the NLL improvement over λmaxI is small. Its ECE is also higher than that of the competing priors.
Table A6: Correlated Gaussian posterior (r=8) with fixed diagonal covariance on CIFAR-10. OOD detection uses predictive entropy on TinyImageNet.
Overall, the favorable relative performance of our prior is largely preserved under correlated Gaussian posteriors with learnable diagonal covariance. However, introducing additional low-rank parameters provides only marginal benefits over MFVI. In contrast, fixing the diagonal covariance substantially changes the relative performance across prior configurations.
These results suggest that learning parameter-wise posterior variances plays an important role in realizing the benefits of the proposed prior, beyond introducing posterior correlations alone.