acceptodds
Under review as a conference paper at ICLR 2027

Empirical Priors for Bayesian Neural Networks via Weight Pruning Sensitivity

Abstract

Bayesian neural networks provide a principled framework for uncertainty estimation, yet their performance is highly dependent on the choice of prior over network parameters. Designing informative priors for modern neural networks remains challenging because their large parameter spaces make it difficult to characterize the functional importance of individual parameters in a computationally tractable way. In this work, we propose an empirical prior for Bayesian neural networks that uses continuous pruning sensitivity to allocate Gaussian prior precision across individual parameters. Through extensive experiments across architectures, datasets, and prior configurations, we demonstrate that our approach achieves consistent performance gains across model families and dataset scales, suggesting that network pruning can be used not only for model compression, but also as a useful tool for designing informative priors in Bayesian neural networks.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian neural network, pruning, prior, variational inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PjusOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a way to set the prior precision/variances when training Bayesian Neural Networks with variational inference. The main idea is that starting from a deterministic solution for the parameters, the authors estimate the influence of each individual parameter to the negative log-likelihood loss. The intuition is that: the parameters that have higher influence on the loss are assigned higher variance allowing them to move more freely away from the pretrained solution Less sensitive parameters are tightly more anchored
The sensitivity score are then mapped to exact precision scores. Experimentally their method outperforms other types of precision initialization in various benchmarks

Critical strengths
  • The idea of using influence (pruning sensitivity ) as a source of information when designing a prior is up to my knowledge quite novel
  • The method is presented clearly, with appropriate theoretical motivation. The local analysis in Section 2.2 explains how prior precision controls both the posterior mean displacement and the posterior variance.
  • The experimental evaluation is adequate. It covers CNN and ViT architectures, in and out of distribution datasets up to ImageNet-1K, and it reports several metrics as well.
Critical weaknesses

Dependence on the mean-field posterior

  • The posterior and the prior choice in variational inference interact strongly. All the experiments use solely MFVI. While someone will logically assume that the benefits of the influence-guided prior will curry to mere correlated posteriors, it might also not be the case, I encourage the authors to adopt a correlated posterior as well
    • Tomczak et al., Efficient Low Rank Gaussian Variational Inference for Neural Networks (NeurIPS 2020)

Restriction to variational inference

  • The approximation framework used throughout the paper is solely variational inference. While VI is quite poplar, it would be beneficial to show that the proposed prior generalizes to to other inference methods as well. The Laplace approximation is a natural candidate: the prior is already centered at a pretrained solution and the GNN is already computed. This will also provide the marginal-likelihood approach as a stronger EB baseline for prior precision selection
    • Immer, Alexander, et al., Scalable marginal likelihood estimation for model selection in deep learning. (ICML 2021)

Missing discussion of function-space priors

  • The related-work discussion is fairly comprehensive on parameter-space priors, but it does not mention the line of work on specifying priors in function space rather than parameter space. This is directly relevant, since the paper’s motivation is that meaningful priors are hard to express at the level of individual weights. I suggest the authors discuss, for example:
    • Rudner et al., Function-Space Regularization in Neural Networks: A Probabilistic Perspective (ICML 2023)
    • Tran et al., All You Need is a Good Functional Prior for Bayesian Deep Learning (JMLR 2022)

MAP reference in the main results.

  • The pretrained deterministic (MAP) solution, which is also the prior mean, should be included as a reference row in the main tables. The paper does not aim for SOTA results, which is fine, but a MAP baseline is needed to verify that the VI models are reasonably trained and to show what the Bayesian treatment adds over the point estimate.

Recommendation: I would be willing to raise my score if the authors adequately address the weaknesses raised above.

Questions for authors

How do the learned posterior variances relate to the sensitivity ranks after training? Do high-sensitivity parameters actually retain larger variance, as the motivation in Section 2.2 suggests?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Pjus (1)

We thank the reviewer for the thoughtful and constructive feedback, as well as for recognizing the novelty of our approach and the strengths of our experimental evaluation. We have carefully considered the concerns raised and revised the manuscript to address them. All changes in the revised manuscript are highlighted in red. Below, we provide point-by-point responses and describe the corresponding revisions.


Comment 1: Generalization Beyond Mean-Field Variational Inference

Answer.

Following the low-rank Gaussian variational inference framework of Tomczak et al. (2020), we consider a correlated Gaussian posterior with covariance Σϕ=D+UU⊤\Sigma_\phi = D + UU^{\top}, where U∈Rd×rU \in \mathbb{R}^{d \times r}. We evaluate two configurations: (i) a learnable diagonal covariance DD and (ii) a fixed diagonal covariance D=e−20ID=e^{-20}I, with rank r=8r=8. Both configurations are evaluated on CIFAR-10 using CNN and ViT architectures. The results are reported in Appendix A.6 (Tables A5 and A6).

1. Learnable Diagonal Covariance

As shown in Table A5, our method achieves the lowest NLL and ECE for both architectures while maintaining competitive accuracy and OOD detection performance. For CNN, the proposed prior also achieves the highest AUROC and lowest FPR95. For ViT, the OOD results are comparable to those of λmax⁡I\lambda_{\max}I, with only marginal differences. These results suggest that the proposed prior precision allocation remains effective under correlated Gaussian posteriors with learnable diagonal covariance.

Table A5: Correlated Gaussian posterior (r=8) with learnable diagonal covariance on CIFAR-10. OOD detection uses predictive entropy on TinyImageNet.

ModelPriorAcc. ↑NLL ↓ECE ↓AUROC ↑FPR95 ↓
CNNλmin I.885±.005.388±.016.085±.002.852±.006.662±.000
λmax I.916±.001.347±.004.050±.001.854±.002.689±.003
EB-Global.833±.012.631±.013.183±.032.802±.015.742±.038
EB-Diag.881±.004.543±.018.207±.009.830±.002.706±.008
Ours.916±.001.251±.001.007±.001.868±.003.619±.005
ViTλmin I.606±.0291.103±.079.047±.002.672±.022.869±.018
λmax I.798±.010.603±.010.047±.008.766±.007.780±.009
EB-Global.626±.0291.044±.083.044±.002.684±.013.860±.008
EB-Diag.647±.052.989±.148.049±.006.693±.033.847±.031
Ours.812±.003.553±.001.033±.001.767±.001.777±.004

2. Fixed Diagonal Covariance

We further examine a more constrained covariance parameterization by fixing D=e−20ID=e^{-20}I, while retaining the low-rank component. Table A6 reports the corresponding results.

Under the fixed diagonal configuration, the relative performance changes. For CNN, EB-Diag achieves the lowest NLL and highest AUROC, while the correlated prior achieves the lowest ECE. The proposed prior maintains competitive accuracy but no longer achieves the best NLL or calibration. For ViT, the proposed prior achieves the highest accuracy and lowest NLL, although the NLL improvement over λmax⁡I\lambda_{\max}I is small. Its ECE is also higher than that of the competing priors.

Table A6: Correlated Gaussian posterior (r=8) with fixed diagonal covariance on CIFAR-10. OOD detection uses predictive entropy on TinyImageNet.

ModelPriorAcc. ↑NLL ↓ECE ↓AUROC ↑FPR95 ↓
CNNλmin I.875±.006.415±.015.049±.005.833±.002.688±.008
λmax I.913±.005.367±.002.054±.001.852±.002.693±.014
EB-Global.905±.005.333±.014.039±.003.854±.007.647±.024
EB-Diag.916±.001.294±.002.039±.001.862±.002.641±.005
Correlated.900±.004.337±.009.034±.003.847±.005.657±.019
Ours.916±.002.334±.004.049±.002.856±.001.677±.003
ViTλmin I.695±.014.898±.029.069±.003.701±.004.839±.002
λmax I.798±.004.621±.003.059±.006.761±.003.784±.004
EB-Global.697±.016.887±.033.065±.006.700±.007.838±.001
EB-Diag.712±.006.851±.011.069±.007.711±.005.830±.003
Ours.808±.001.618±.003.070±.000.763±.001.783±.006

Overall, the favorable relative performance of our prior is largely preserved under correlated Gaussian posteriors with learnable diagonal covariance. However, introducing additional low-rank parameters provides only marginal benefits over MFVI. In contrast, fixing the diagonal covariance substantially changes the relative performance across prior configurations.

These results suggest that learning parameter-wise posterior variances plays an important role in realizing the benefits of the proposed prior, beyond introducing posterior correlations alone.



AuthorsOpenReview ↗

Response to Reviewer Pjus (2-1)

Comment 2: Restriction to Variational Inference

Answer.

We thank the reviewer for this valuable suggestion. Extending our prior design beyond variational inference is a question we had also considered during the development of this work, particularly regarding the mathematical consistency of applying Laplace approximation to a pretrained solution.

1. Mathematical Consideration

Unlike MFVI, which directly optimizes a variational posterior, Laplace approximation constructs a Gaussian approximation around a maximum a posteriori (MAP) solution. Therefore, applying our empirical prior to Laplace approximation requires ensuring that the expansion point corresponds to a stationary point of the resulting posterior.

To clarify this issue, let the negative log-likelihood be LNLL(w)=−log⁡p(D∣w)\mathcal{L}_{\mathrm{NLL}}(\mathbf{w})=-\log p(\mathcal{D}\mid\mathbf{w}).

The corresponding negative log-posterior can be expressed, up to an additive constant, as

J(w)=LNLL(w)+R(w),\mathcal{J}(\mathbf{w})=\mathcal{L}_{\mathrm{NLL}}(\mathbf{w})+\mathcal{R}(\mathbf{w}),

where R(w)\mathcal{R}(\mathbf{w}) denotes the regularization term induced by the prior distribution.

The MAP solution is defined as wMAP=arg⁡min⁡wJ(w)\mathbf{w}_{\mathrm{MAP}}=\arg\min_{\mathbf{w}}\mathcal{J}(\mathbf{w}), which must satisfy the first-order stationary condition ∇wJ(wMAP)=0\nabla_{\mathbf{w}}\mathcal{J}(\mathbf{w}_{\mathrm{MAP}})=\mathbf{0}.

However, our original empirical prior is centered at the pretrained solution w∗\mathbf{w}^{\ast}:

pours(w)=N(w∣w∗,Λours−1).p_{\mathrm{ours}}(\mathbf{w})=\mathcal{N}\left(\mathbf{w}\mid\mathbf{w}^{\ast},\Lambda_{\mathrm{ours}}^{-1}\right).

For this Gaussian prior, the regularization term becomes

Rours(w)=12(w−w∗)⊤Λours(w−w∗).\mathcal{R}_{\mathrm{ours}}(\mathbf{w})=\frac{1}{2}(\mathbf{w}-\mathbf{w}^{\ast})^{\top}\Lambda_{\mathrm{ours}}(\mathbf{w}-\mathbf{w}^{\ast}).

Consequently, evaluating the negative log-posterior gradient at w∗\mathbf{w}^{\ast} gives

\nabla_{\mathbf{w}}\mathcal{J}(\mathbf{w}^{\ast}) \=\nabla_{\mathbf{w}}\mathcal{L}_{\mathrm{NLL}}(\mathbf{w}^{\ast})+\nabla_{\mathbf{w}}\mathcal{R}_{\mathrm{ours}}(\mathbf{w}^{\ast})\ \=\nabla_{\mathbf{w}}\mathcal{L}_{\mathrm{NLL}}(\mathbf{w}^{\ast})+\Lambda_{\mathrm{ours}}(\mathbf{w}^{\ast}-\mathbf{w}^{\ast})\ \=\underbrace{\nabla_{\mathbf{w}}\mathcal{L}_{\mathrm{NLL}}(\mathbf{w}^{\ast})}_{\mathbf{g}}. \end{aligned}$$ Importantly, $\mathbf{g}$ does not necessarily vanish, since deterministic pretraining typically involves regularization in addition to the negative log-likelihood. Consequently, the pretrained solution $\mathbf{w}^{\ast}$ is not necessarily a stationary point of the posterior defined by our original empirical prior. We therefore identified two possible approaches to construct a mathematically consistent Laplace approximation: **Prior mean correction**: Adjust the prior mean so that the pretrained solution $\mathbf{w}^{\ast}$ remains a stationary point of the resulting posterior. **MAP reoptimization**: Retain the original prior mean $\mathbf{w}^{\ast}$ and reoptimize the MAP solution under the proposed prior before constructing the Laplace approximation. Both approaches are mathematically valid. However, the second requires additional optimization and generally moves the posterior mode away from the original pretrained solution. We therefore consider prior mean correction a more practical choice for post-hoc inference, as it allows us to preserve the pretrained solution without additional MAP optimization.
AuthorsOpenReview ↗

Response to Reviewer Pjus (2-2)

2. Prior Mean Correction

To preserve w∗\mathbf{w}^{\ast} as the expansion point, we introduce a corrected prior mean mcorr\mathbf{m}_{\mathrm{corr}}, while keeping the proposed prior precision matrix Λours\Lambda_{\mathrm{ours}} unchanged.

Specifically, we define the corrected prior as

pcorr(w)=N(w∣mcorr,Λours−1).p_{\mathrm{corr}}(\mathbf{w})=\mathcal{N}\left(\mathbf{w}\mid\mathbf{m}_{\mathrm{corr}},\Lambda_{\mathrm{ours}}^{-1}\right).

We require the negative log-posterior gradient to vanish at w∗\mathbf{w}^{\ast}: ∇wJ(w∗)=0\nabla_{\mathbf{w}}\mathcal{J}(\mathbf{w}^{\ast})=\mathbf{0}.

Substituting the corrected prior gives

∇wLNLL(w∗)+Λours(w∗−mcorr)=0.\nabla_{\mathbf{w}}\mathcal{L}_{\mathrm{NLL}}(\mathbf{w}^{\ast})+\Lambda_{\mathrm{ours}}(\mathbf{w}^{\ast}-\mathbf{m}_{\mathrm{corr}})=\mathbf{0}.

Using g=∇wLNLL(w∗)\mathbf{g}=\nabla_{\mathbf{w}}\mathcal{L}_{\mathrm{NLL}}(\mathbf{w}^{\ast}), we obtain Λours(w∗−mcorr)=−g\Lambda_{\mathrm{ours}}(\mathbf{w}^{\ast}-\mathbf{m}_{\mathrm{corr}})=-\mathbf{g}.

Therefore, the corrected prior mean is

mcorr=w∗+Λours−1g.\mathbf{m}_{\mathrm{corr}}=\mathbf{w}^{\ast}+\Lambda_{\mathrm{ours}}^{-1}\mathbf{g}.

This correction ensures that the prior gradient exactly cancels the negative log-likelihood gradient at w∗\mathbf{w}^{\ast}, making the pretrained solution a stationary point of the resulting posterior.

Under a local quadratic approximation of the negative log-likelihood around w∗\mathbf{w}^{\ast}, the corrected negative log-posterior becomes

J(w)≈J(w∗)+12(w−w∗)⊤(H+Λours)(w−w∗),\mathcal{J}(\mathbf{w})\approx\mathcal{J}(\mathbf{w}^{\ast})+\frac{1}{2}(\mathbf{w}-\mathbf{w}^{\ast})^{\top}(H+\Lambda_{\mathrm{ours}})(\mathbf{w}-\mathbf{w}^{\ast}),

where HH denotes the Hessian of the negative log-likelihood at w∗\mathbf{w}^{\ast}. Using the positive semidefinite GGN approximation GG in place of HH, the corresponding local Gaussian approximation is

qLA(w)≈N(w∣w∗,(G+Λours)−1).q_{\mathrm{LA}}(\mathbf{w})\approx\mathcal{N}\left(\mathbf{w}\mid\mathbf{w}^{\ast},(G+\Lambda_{\mathrm{ours}})^{-1}\right).

For the full-network diagonal Laplace approximation used in our experiments, this reduces to

qLA(w)≈N(w∣w∗,[diag⁡(G)+Λours]−1).q_{\mathrm{LA}}(\mathbf{w})\approx\mathcal{N}\left(\mathbf{w}\mid\mathbf{w}^{\ast},[\operatorname{diag}(G)+\Lambda_{\mathrm{ours}}]^{-1}\right).

Importantly, this formulation preserves our sensitivity-based precision allocation while ensuring that the pretrained solution is a consistent center for the local Laplace approximation.

3. Experimental Results

Following this formulation, we evaluated our method under full-network diagonal Laplace approximation using the linearized (GLM) predictive distribution with 30 Monte Carlo samples.

We conducted experiments on CIFAR-10 and ImageNet-1K across CNN and ViT architectures. As a baseline, we used the post-hoc empirical Bayes (EB) approach implemented in Laplace Redux (Daxberger et al., 2021), which optimizes Gaussian prior precision hyperparameters through marginal likelihood maximization while retaining a zero-mean prior. In contrast, our approach uses the proposed prior precision with the prior mean correction derived above.

DatasetArchitectureMethodAcc ↑NLL ↓ECE ↓AUROC ↑FPR95 ↓
CIFAR-10CNNOurs0.9150.2550.0170.8640.634
EB0.9000.5350.2400.8590.648
CIFAR-10ViTOurs0.7900.6160.0400.7560.791
EB0.7730.8090.1950.7730.786
ImageNet-1KCNNOurs0.8011.4200.4070.6990.859
EB0.7571.5820.4450.3080.991
ImageNet-1KViTOurs0.7211.2200.1200.7920.798
EB0.7091.4620.2770.7330.868

The results show that our method improves accuracy, NLL, and ECE over the EB baseline across all four settings. For example, on CIFAR-10 CNN, our method reduces NLL from 0.535 to 0.255 and ECE from 24.0% to 1.7%. Similar improvements are observed on ImageNet-1K, where our method reduces NLL from 1.462 to 1.220 for ViT and from 1.582 to 1.420 for CNN. Improvements in OOD detection are also observed in three of the four settings, although they are not uniform across all architectures.

Reviewer_GWhnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Motivated by the need for better priors in BNNs, the authors devise a data-informed way to select the precision parameters of a fully factorised Gaussian prior. Starting with the weights vector of a deterministically pre-trained network, the authors define each weight's pruning sensitivity score as the difference in training-set log-likelihood caused by removing the weight. They propose a Hessian-based approach to estimate them in a scalable manner. Since the posterior should be tighter around weights that have a higher sensitivity to pruning, the authors propose a ranking of the prior precisions that matches that of the pruning sensitivity scores, and after selecting the minimum and maximum precisions in a sensible way (minimum of 1.0, maximum of {number of training points} / {maximum learning rate} ) they assign precision values that are linearly spaced within the bounding values, in according with the ranking assignments. For both their approach and all baselines (except in one experiment), the authors choose to consider BNNs whose prior mean is set as the pre-trained weights vector.

The authors compare their approach against a sensible collection of baseline approaches (for selecting the Gaussian prior's precision parameters) and the results are favourable.

Critical strengths

S1 The paper is very well written and all exposition is clear.

S2 The problem of how to find better priors for BNNs is both important and interesting, and the authors' idea strikes me as rather elegant!

S3 The experiments have been carried out thoroughly. The authors have posed a nice variety of sensible research questions and they have chosen their experiments well for the purposes of answering them. The results are very promising.

Critical weaknesses

W1: Double Conditioning. The prior is essentially a (rather coarse) Laplace approximation to the usual posterior p(w∣D)p(\mathbf{w}|\mathcal{D}) obtained under a N(0,σI)\mathcal{N}(\boldsymbol{0},\sigma\mathbf{I}) prior (where σ\sigma depends on how regularisation was performed for the deterministic pretrained networks, currently this detail is missing from the paper), meaning the authors' approximate posterior q(w∣D)q(\mathbf{w}|\mathcal{D}) is approximately targeting p(w∣D)p(D∣w)p(D)=p(w∣D,D)\frac{p(\mathbf{w}|\mathcal{D})p(\mathcal{D}|\mathbf{w})}{p(\mathcal{D})} = p(\mathbf{w}|\mathcal{D},\mathcal{D}). While this looks highly unusual, I believe it is the same thing as the tempered posterior at temperature T=0.5T=0.5 which achieves the effect of counting each datapoint twice: pT=0.5(w∣D):∝p(w)p(D∣w)0.5p_{_{T=0.5}}(\mathbf{w}|\mathcal{D}) :\propto p(\mathbf{w})p(\mathcal{D}|\mathbf{w})^{0.5}. Since tempered MFVI is already a widely adopted and investigated approach in the BNN community, please can the authors

  1. mention this connection in the next revision,
  2. comment here and in their revision on the relative pros and cons between their approach and tempered MFVI.

I leave it up to the authors to decide whether to include a tempered MFVI baseline in their experiments in the revision; for me this inclusion is not a requirement for acceptance.

Since the prior could serve as an approximate posterior, the authors should also

  1. include their prior itself as a baseline in the experiments in their next revision

as this would help readers understand how close to a usable approximate posterior the authors' prior already is. Finally, the authors also need to

  1. include the pre-trained solution as a baseline in the experiments in their next revision

as this would answer the question of whether any of the approximate inference methods can surpass the trivial solution, which is an especially pertinent question since all baselines are regularised against differing too much from this solution. Note that whether or not the pre-trained solution outperforms the authors' approach has absolutely no bearing on my acceptance/rejection recommendation at all; but including the deterministic network as a baseline will push me closer to acceptance.

Questions for authors

Further Requests

R1 In the discussions surrounding existing approaches for constructing empirical priors, there are some highly relevant works that have been left out and that readers would be benefit from knowing about. These include paper 1, paper 2, paper 3, paper 4, and paper 5.

R2 The description of the two uniform prior setups on lines 245-247 is not clear to me. Please can the authors clarify how the per-weight uniform distributions are assigned here. Each uniform distribution is centred on the corresponding wi∗w_i^*, but what exactly is the (half) width of each distribution in terms of λmin\lambda_\text{min} or λmax\lambda_\text{max}? At the bottom of page 5, the authors write that the λminI\lambda_\text{min}\mathbf{I} prior corresponds to a unit-variance Gaussian. Whether these priors are uniform or Gaussian needs to be confirmed by the authors and clarified in the paper.

R3 Regarding the introduction of LtestBMA\mathcal{L}_\text{test}^\text{BMA} on lines 368-371, this paragraph can be made considerably shorter and clearer by using the standard term "log posterior predictive". This metric should be a function of μ,σ\boldsymbol{\mu}, \boldsymbol{\sigma} in the same way that Ltest(μ,σ)\mathcal{L}_\text{test}(\boldsymbol{\mu},\boldsymbol{\sigma}) is$\textemdash$why have the authors chosen notation that obscures this fact? It is also unusual that the authors compute the expected per-datapoint log posterior predictive, rather than the much more natural log-posterior predictive density of the full test set, which is more sensitive to the correlation structure of prediction samples. The latter quantity is computed as log⁡(Ew∼qϕ[∏x,y∈Dtestp(y∣x,w)])\log \Biggl( \mathbb{E}_{\mathbf{w}\sim q_\phi} \Bigl[ \prod_{x,y\in\mathcal{D}_\text{test}}p(y|x,\mathbf{w})\Bigr]\Biggr), and is amenable to test-set-size-normalisation simply by dividing it by the number of test points (if the per-datapoint scale is what the authors desire). The metric the authors currently consider only considers the marginals of the posterior predictive, which is needlessly limiting.

R4 For the experiment in Section 5.3, please can the authors make it clearer (here and in the manuscript) whether their method uses w∗\mathbf{w}^* or 0\boldsymbol{0} as the prior mean? In other words, does their method only use a pre-trained solution to obtain the precisions, or also for the mean?

R5 Is the approach presented in equation 6 prior work? If so, this should be made clearer/more explicit. Furthermore, the claimed weakness of this approach is supported with neither a citation nor a pointer to relevant experiment conducted by the authors (e.g. in an appendix). The authors should fix this issue in order to bolster their claim, because equation 6's approach being flawed is what ultimately motivates the entire paper.

Minor Corrections

C1 On line 207 the authors state "the KL term in the negative ELBO is scaled by 1N\frac{1}{N}. But this is wrong according to both 1.) me, and 2.) the authors (equation 2). Please can the authors either fix this or justify it.

C2 On lines 244-245, "as described Section 3.4" should read "as described in Section 3.4".

7 comments
AuthorsOpenReview ↗

Response to Reviewer GWhn (1)

We thank the reviewer for the thoughtful and constructive feedback, as well as for recognizing the strengths of our work. We have carefully addressed the concerns raised and revised the manuscript accordingly. All changes in the revised manuscript are highlighted in red. Below, we provide point-by-point responses and describe the corresponding revisions.


Comment 1: Double Conditioning and Connection to Tempered MFVI

Answer

We agree that our empirical prior is data-dependent, as both the pretrained weights and pruning sensitivities are obtained from the same training data subsequently used for MFVI. This data reuse is part of our problem formulation, and we appreciate the opportunity to clarify its relationship to tempered inference. We also agree that the reviewer's interpretation can be connected to cold posterior inference with T=0.5T=0.5, where the likelihood is raised to the power 1/T=21/T=2.

1. Mathematical Relationship

Following the reviewer's interpretation, let p0(w)=N(0,Λ0−1)p_0(\mathbf{w})=\mathcal{N}(\mathbf{0},\boldsymbol{\Lambda}_0^{-1}) denote the reference Gaussian prior associated with deterministic pretraining, and let the usual posterior under this prior be

p(w∣D)=p0(w)p(D∣w)Z0.p(\mathbf{w}\mid\mathcal{D})=\frac{p_0(\mathbf{w})p(\mathcal{D}\mid\mathbf{w})}{Z_0}.

Our empirical prior is defined as pours(w)=N(w∗,Λours−1)p_{\mathrm{ours}}(\mathbf{w})=\mathcal{N}(\mathbf{w}^{\ast},\boldsymbol{\Lambda}_{\mathrm{ours}}^{-1}), and the variational posterior q(w∣D)q(\mathbf{w}\mid\mathcal{D}) approximates the normalized distribution proportional to p(D∣w)pours(w)p(\mathcal{D}\mid\mathbf{w})p_{\mathrm{ours}}(\mathbf{w}). By multiplying and dividing the unnormalized target by the usual posterior, we obtain

\frac{p_{\mathrm{ours}}(\mathbf{w})}{p(\mathbf{w}\mid\mathcal{D})} \propto \underbrace{p_0(\mathbf{w})p(\mathcal{D}\mid\mathbf{w})^2}_{\text{Tempered posterior}} \underbrace{\frac{p_{\mathrm{ours}}(\mathbf{w})}{p(\mathbf{w}\mid\mathcal{D})}}_{C(\mathbf{w})}.$$ Here, $C(\mathbf{w})$ represents the difference between our empirical prior and the usual posterior. This decomposition is an exact algebraic identity for the unnormalized target. As the reviewer suggests, if $p_{\mathrm{ours}}(\mathbf{w})=p(\mathbf{w}\mid\mathcal{D})$, then $C(\mathbf{w})=1$, and the resulting target is exactly equivalent to likelihood tempering with $T=0.5$. **2. Comparison with Tempered MFVI** Motivated by the reviewer's observation, we additionally evaluated tempered MFVI with $T\in{1.0,0.5,0.1}$ on CIFAR-10 using CNN and ViT. | CIFAR-10 | Acc ↑ | NLL ↓ | ECE ↓ | AUROC ↑ | FPR95 ↓ | |---|---:|---:|---:|---:|---:| | CNN (T=1.0) | .883±.003 | .388±.008 | .081±.004 | .846±.004 | .657±.007 | | CNN (T=0.5) | .886±.005 | .356±.009 | .049±.002 | .852±.007 | .647±.010 | | CNN (T=0.1) | .886±.002 | .343±.005 | .028±.004 | .842±.006 | .664±.006 | | **CNN (Ours)** | **.916±.001** | **.247±.002** | **.007±.001** | **.861±.002** | **.632±.002** | | ViT (T=1.0) | .597±.013 | 1.130±.042 | .045±.005 | .669±.009 | .878±.010 | | ViT (T=0.5) | .614±.020 | 1.074±.046 | .033±.008 | .678±.001 | .872±.002 | | ViT (T=0.1) | .617±.058 | 1.075±.015 | **.028±0.04** | .672±.003 | .872±.004 | | **ViT (Ours)** | **.814±.001** | **.552±.002** | .034±.002 | **.767±.001** | **.771±.001** | AUROC and FPR95 are evaluated on TinyImageNet. We observe that reducing the temperature improves standard MFVI. Nevertheless, our method at $T=1$ achieves higher accuracy, lower NLL, and better OOD detection performance than the evaluated tempered baselines on both architectures. These results indicate that global tempering alone does not reproduce the predictive performance of our method under the evaluated settings, although the comparison does not isolate the proposed precision allocation from other differences in prior specification. The detailed experimental results are provided in Appendix A.5. **3. Pros and Cons Relative to Tempered MFVI** Compared with tempered MFVI, which globally adjusts the relative contribution of the likelihood and prior, our method assigns parameter-wise prior precisions based on pruning sensitivity. This enables selective regularization, allowing greater posterior flexibility for sensitive parameters while constraining less sensitive ones. Empirically, our method achieves better accuracy, NLL, and OOD detection performance than the evaluated tempered MFVI baselines, even at $T=1$. However, our method requires a pretrained deterministic network and an additional sensitivity estimation step, whereas standard tempered MFVI does not require these components. This introduces additional computational costs, particularly when a pretrained checkpoint is unavailable.
AuthorsOpenReview ↗

Response to Reviewer GWhn (2)

Comment 2: Prior-Only Evaluation as an Approximate Posterior

Answer.

We thank the reviewer for this insightful suggestion. Evaluating the prior directly through sampling is a simple yet informative way to assess its predictive performance, which we had not previously considered.

Following this suggestion, we evaluated the predictive distribution induced by sampling directly from the proposed prior, without MFVI optimization, across CIFAR-10, ImageNet-100, and ImageNet-1K. For CNN architectures, sampled weights may induce activation distributions that differ from the pretrained BatchNorm (BN) running statistics. Therefore, we report results both with and without BN recalibration. For ViT architectures, BN recalibration is not applied since they use LayerNorm (LN).

The results show that the proposed prior can already provide reasonable predictive performance without variational optimization, particularly for CIFAR-10 CNN. However, prior-only inference does not consistently match MFVI, especially in terms of predictive NLL and calibration. For example, on ImageNet-1K ResNet50-v2, MFVI reduces NLL from 1.359 to 0.817 and ECE from 34.7% to 3.0% compared with the BN-recalibrated prior.

Similar benefits are observed for ViT architectures. On CIFAR-10 ViT, MFVI improves accuracy from 76.9% to 81.4% and reduces NLL from 0.673 to 0.552. On ImageNet-100 ViT, MFVI reduces NLL from 1.280 to 0.408 and ECE from 43.3% to 4.9%. Notably, on ImageNet-1K ViT, MFVI reduces NLL from 1.390 to 1.116 and ECE from 22.3% to 0.9%, while maintaining the same accuracy of 71.6%. Although these improvements are not uniform across all metrics, they demonstrate that variational optimization provides additional benefits beyond sampling directly from the empirical prior.

DatasetArchitectureEvaluationAcc ↑NLL ↓ECE ↓AUROC ↑
CIFAR-10CNNPrior Only0.914‾\underline{0.914}0.248‾\underline{0.248}0.0130.862
Prior Only + BN0.914‾\underline{0.914}0.2470.011‾\underline{0.011}0.862
MFVI0.9160.2470.0070.861‾\underline{0.861}
ViTPrior Only0.769‾\underline{0.769}0.673‾\underline{0.673}0.0230.747‾\underline{0.747}
MFVI0.8140.5520.034‾\underline{0.034}0.767
ImageNet-100CNNPrior Only0.8280.6680.0650.861
Prior Only + BN0.838‾\underline{0.838}0.628‾\underline{0.628}0.059‾\underline{0.059}0.866‾\underline{0.866}
MFVI0.8400.6060.0280.868
ViTPrior Only0.829‾\underline{0.829}1.280‾\underline{1.280}0.433‾\underline{0.433}0.884‾\underline{0.884}
MFVI0.8830.4080.0490.904
ImageNet-1KCNNPrior Only0.784‾\underline{0.784}1.5680.4420.778
Prior Only + BN0.7801.359‾\underline{1.359}0.347‾\underline{0.347}0.794‾\underline{0.794}
MFVI0.7910.8170.0300.850
ViTPrior Only0.7161.390‾\underline{1.390}0.223‾\underline{0.223}0.806
MFVI0.7161.1160.0090.792‾\underline{0.792}

These findings suggest that our empirical prior provides a useful predictive distribution around the pretrained solution, while MFVI remains important for refining predictive behavior, particularly uncertainty calibration.

We have included these additional experiments and the corresponding discussion in Section 5.3 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer GWhn (3)

Comment 3: Pretrained Deterministic Network as a Baseline

Answer.

We have explicitly included the pretrained deterministic solution (w∗\mathbf{w}^{\ast}) as a baseline in Tables 1 and 2 of the revised manuscript, reporting its predictive performance and OOD detection results alongside the MFVI methods. While the pretrained DNN remains competitive and outperforms MFVI baselines in certain experimental settings, our method achieves lower predictive NLL, better calibration, and stronger OOD detection in other evaluated settings.


Comment 4: Missing Related Work

Answer.

We have expanded the second and third paragraphs of the Introduction to provide a more comprehensive discussion of existing approaches to prior construction, including function-space priors, empirical Bayes, and meta-learning approaches. We have also clarified how our method differs from these approaches by focusing on parameter-wise prior precision allocation guided by pruning sensitivity when a pretrained solution is available.


Comment 5: Clarification of Uniform Prior Settings

Answer.

We apologize for the ambiguity in our description. Our intention was to indicate that the same prior precision is uniformly assigned to all parameters, rather than to imply a uniform probability distribution. Specifically, both baselines use isotropic Gaussian priors centered at the pretrained weights, with precision matrices λmin⁡I\lambda_{\min}I and λmax⁡I\lambda_{\max}I, respectively.

In contrast, our method assigns lower prior precision to weights with higher pruning sensitivity, allowing greater posterior flexibility, while less sensitive weights receive higher prior precision to remain closer to the pretrained solution.

We have revised the manuscript to use the term "isotropic Gaussian priors" to avoid this ambiguity.

Reviewer_K2ahOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a novel method for specifying an informative prior distribution over the parameters of a Bayesian neural network (BNN) when a pretrained neural network is available. Their approach considers a special case of mean-field variational inference (MFVI) for BNNs in which independent normal priors are specified for each BNN weight, with the prior means set to the corresponding weights of a pretrained deterministic neural network. The primary contribution is the choice of the priors' precision parameters based on the sensitivity of an approximate negative log-likelihood (NLL) of the BNN parameters to parameter-wise perturbations.

More specifically, the authors consider a locally quadratic approximation of the NLL around the pretrained weights and use the magnitude of its change imposed by a parameter-specific perturbation (pruning sensitivity score; PSS) as an indicator of the effect of removing that weight. These PSSs are transformed into prior precision values by first mapping them to normalized ranks that determine their final magnitude in an affine interpolation between 1 and the maximum precision value given by (size of the training data set) / (maximal learning rate). Consequently, weights with higher PSSs are assigned lower prior precision values. The approach is evaluated on CIFAR-10, ImageNet-100, and ImageNet-1K using CNN (ResNet) and Vision Transformer (DeiT-Tiny) architectures to demonstrate performance improvements over Gaussian isotropic, correlated, and empirical Bayes baselines in both in-distribution and out-of-distribution metrics.

Critical strengths

The paper provides a clear problem formulation within the MFVI framework, demonstrates the precision divergence problem of a marginal likelihood-based approach in this setting, and proposes the innovative solution of choosing prior precision parameters based on the PSS.

The Experiment and Analysis sections support the proposed approach. There are two points I find worth highlighting:

  • The posterior predictive decomposition of the test NLL supports the hypothesis that this approach induces posterior diversity that can be exploited in Bayesian model averaging, whereas isotropic high-precision priors constrain posterior variation around the pretrained solution (Table 3).
  • The analyses of the importance of a PSS-order-preserving choice of prior precisions (see Table 4) and the partitioning of the loss landscape into low- and high-sensitivity groups (see Figure 1) provide evidence that the PSS-based approach induces reasonable estimates.
Critical weaknesses

The scope of the paper and its performance claims are restricted to a specific case of MFVI for BNNs. This should be stated more explicitly.

  • The authors state that marginal likelihood maximization is a natural approach to prior precision estimation and cite two papers (Immer et al., 2021, and Daxberger et al., 2021) that use a Laplace approximation of the posterior distribution over BNN parameters (lines 123-124). However, unlike in the paper under review, that Laplace approximation entails a quadratic approximation of the NLL that does not include a first-order term since the gradient is assumed to approximately vanish at the pretrained network parameters. Moreover, according to the cited literature, if the Laplace approximation is based on a Generalized Gauss-Newton approximation of the network weights' Hessian matrix, the appropriate posterior predictive distribution for these BNNs corresponds to a network with linearized features (which also yields improved predictive performance). The authors should more clearly differentiate their setting from the cited literature.
  • The comparison to the pretrained deterministic solutions (which the authors state are competitive and outperform their MFVI counterparts in some settings; compare Appendix A.7) puts this restriction into perspective.

Although the PSS-based selection of prior precision parameters is intuitively appealing and competitive in the paper's experiments, there is no clear statistical motivation that connects the PSS to desirable properties of the BNNs' posterior (predictive) distributions. Providing this motivation would meaningfully strengthen the paper.

Questions for authors

None.

4 comments
AuthorsOpenReview ↗

Response to Reviewer K2ah

We thank the reviewer for the thoughtful and constructive feedback, as well as for recognizing the strengths of our formulation and experimental analysis. We have carefully considered the concerns raised and revised the manuscript to improve its clarity and technical precision. All changes in the revised manuscript are highlighted in red. Below, we provide point-by-point responses and describe the corresponding revisions.


Comment 1: Scope and Limitations of MFVI

Answer. We agree that the scope of our work should be stated more explicitly. In the revised manuscript, we have clarified in the contributions listed in the Introduction that our method is primarily developed and evaluated for BNNs under mean-field variational inference (MFVI), and have explicitly acknowledged this limitation in the Conclusion. We have also ensured that our performance claims are appropriately scoped to this setting.


Comment 2: Distinction from Laplace Approximation

Answer. We agree that the distinction between our formulation and Laplace approximation should be made more explicit.

1. Distinction from Laplace Approximation. Standard Laplace approximation (Immer et al., 2021; Daxberger et al., 2021) constructs a Gaussian posterior by expanding the negative log posterior around a MAP solution. At the MAP solution w∗\mathbf{w}^{\ast}, the gradients of the NLL and regularization term approximately cancel:

∇LNLL(w∗)+∇R(w∗)≈0.\nabla\mathcal{L}_{\mathrm{NLL}}(\mathbf{w}^{\ast})+\nabla\mathcal{R}(\mathbf{w}^{\ast})\approx 0.

Importantly, the vanishing gradient of the negative log posterior does not necessarily imply a vanishing NLL gradient, since it may be balanced by the regularization gradient. In contrast, our approach expands the NLL around the pretrained solution, retaining the potentially nonzero first-order term:

\approx \mathcal{L}_{\mathrm{NLL}}(\mathbf{w}^{\ast}) +\mathbf{g}^{\top}\boldsymbol{\delta} +\frac{1}{2}\boldsymbol{\delta}^{\top}\mathbf{H}\boldsymbol{\delta}.$$ Unlike Laplace approximation, this expansion is used to estimate parameter-wise pruning sensitivity for prior precision allocation, rather than to directly construct a Gaussian posterior. The variational posterior is subsequently learned through MFVI by optimizing the ELBO of the original nonlinear BNN. **2. GGN and Linearized Predictive Distribution.** We agree that a linearized predictive distribution is appropriate for maintaining consistency when the GGN is used to construct a Laplace posterior. However, the role of the GGN in our method is fundamentally different. Specifically, we use the diagonal GGN solely to approximate the NLL curvature when computing pruning sensitivity: $$s_i=\left|-g_iw_i^{\ast}+\frac{1}{2}G_{ii}(w_i^{\ast})^2\right|.$$ The resulting sensitivity scores determine the parameter-wise prior precisions, rather than directly defining the posterior covariance. The posterior means and variances are instead learned through MFVI by optimizing the ELBO of the original nonlinear BNN. Therefore, unlike Laplace approximation, our use of the GGN is limited to prior construction and does not require linearized prediction. Predictions are obtained through Bayesian model averaging using weight samples from the learned variational posterior and the original nonlinear network. We have clarified these distinctions in Section 2.2, emphasizing that our local quadratic expansion is applied to the NLL rather than the negative log posterior, and in Section 3.1, specifying that the GGN is used for pruning sensitivity estimation and prior construction rather than posterior approximation. --- ### Comment 3: Comparison with Pretrained Deterministic Networks **Answer.** We agree that pretrained deterministic networks remain competitive and can outperform their MFVI counterparts in certain settings. To make this comparison explicit, we have included the pretrained DNN as a reference baseline in Tables 1 and 2. We have also added a prior-only sampling evaluation in Section 5.3 to distinguish the predictive performance of the empirical prior itself from the additional benefits of MFVI. While the pretrained DNN remains competitive and outperforms MFVI baselines in certain experimental settings, our method achieves lower predictive NLL, better calibration, and stronger OOD detection in other evaluated settings. These observations highlight that Bayesian inference alone does not guarantee improved predictive performance over a strong deterministic solution, underscoring the importance of informative prior specification in BNNs.
AuthorsOpenReview ↗

Response to Reviewer K2ah (2)

Comments 4: Statistical Motivation for the Proposed Prior Precision Allocation

Answer.

Our method is motivated by selective shrinkage, which aims to allocate posterior flexibility according to the loss sensitivity of pretrained weights.

As illustrated by the pretrained DNN in Figure 1 (black curves), low-PSS parameters exhibit relatively flat loss landscapes, whereas high-PSS parameters are substantially more sensitive to perturbations. This observation motivates preserving the relatively stable characteristics of low-PSS parameters while allowing greater posterior adaptation of high-PSS parameters, where there may be more opportunity to improve predictive robustness.

From a statistical perspective, Eq. (5) shows that likelihood curvature and prior precision jointly determine the posterior mean and variance under the local quadratic approximation. Accordingly, we assign higher prior precision to low-PSS parameters to suppress unnecessary deviations from the pretrained solution, while assigning lower precision to high-PSS parameters to relax prior constraints and allow greater adaptation during VI. Thus, PSS guides the allocation of prior regularization rather than directly specifying the posterior variance.

Our empirical results support this motivation. As shown in Figure 1, VI with our proposed prior yields flatter predictive loss landscapes in both groups, with a more pronounced improvement in the high-PSS group across CNN and ViT architectures. Furthermore, the opposite precision allocation produces sharper predictive landscapes and worse predictive performance, supporting the importance of selectively allocating posterior flexibility according to pruning sensitivity.

Reviewer_K2ahOpenReview ↗

Response to the authors' rebuttal

Thank you for the detailed responses and for the revisions to the manuscript, which have strengthened the paper. However, some of my points remain partly unresolved.


Re: Comment 1 (Scope and Limitations of MFVI)

Scoping the contributions and conclusions to MFVI and to the reliance on pretrained networks has improved the clarity and transparency of the paper. Please also adapt the abstract accordingly.


Re: Comment 2 (Distinction from Laplace Approximation)

First, I would like to correct a typo in my original review. When discussing Immer et al. (2021) and Daxberger et al. (2021), the sentence should have read "[...] that Laplace approximation entails a quadratic approximation of the penalized NLL [...]". The authors interpreted my comment correctly (up to an additive constant independent of w\mathbf{w}, the penalized NLL corresponds to the negative log posterior density), and their response explains the differences from their approach, which I largely agree with.

However, I could not find the change to Section 3.1 mentioned in the response.

In Section 4.1, the authors write:

All methods share the same deterministic solution w∗\mathbf{w}^{\ast} as the prior mean, allowing us to compare different specifications of prior precision around a common pretrained solution. We consider two empirical Bayes (EB) variants as baselines that optimize prior precision by marginal likelihood maximization (Immer et al., 2021; Daxberger et al., 2021).

Immer et al. (2021) and Daxberger et al. (2021) use zero-centered priors rather than priors centered at w∗\mathbf{w}^{\ast}. For a w∗\mathbf{w}^{\ast}-centered prior, the w\mathbf{w}-dependent term of the log prior density and its gradient vanish at w∗\mathbf{w}^{\ast}. This encourages the precision divergence phenomenon described in Section 2.2. Please state clearly that EB-Global and EB-Diag are adaptations to the w∗\mathbf{w}^{\ast}-centered setting rather than implementations of the methods proposed in the cited papers.


Re: Comment 3 (Comparison with Pretrained Deterministic Networks)

The authors have included the pretrained DNNs as reference baselines in Tables 1 and 2. In their response, they acknowledge that "[...] Bayesian inference alone does not guarantee improved predictive performance over a strong deterministic solution [...]". The former Appendix A.7 acknowledged that the pretrained DNNs are competitive with, and in some settings outperform, their MFVI counterparts, but this observation was removed in the revision. Please include it in the main text.


Re: Comment 4 (Statistical Motivation for the Proposed Prior Precision Allocation)

It remains somewhat unclear to me how the data-driven shrinkage towards zero in the cited selective shrinkage literature motivates the PSS-based selection of precision parameters for w∗\mathbf{w}^{\ast}-centered priors, and how this relates to properties of the posterior predictive distribution. I do not see this as a major concern, however.


I maintain my rating of 3 (weak acceptance).

Thank you again for your careful consideration of my review.

AuthorsOpenReview ↗

Response to Reviewer K2ah’s Follow-up Comments

Re: Comment 1 (Scope and Limitations of MFVI)

Thank you for your suggestion. We have revised the Abstract (Lines 17–19) to explicitly specify that our approach is developed under MFVI using a pretrained deterministic network, consistent with the revised Introduction and Conclusion.

Re: Comment 2 (Distinction from Laplace Approximation)

We apologize for the confusion caused by our previous response regarding Section 3.1. We considered that the existing description sufficiently addressed the distinction discussed in our rebuttal, but our wording may have incorrectly suggested that additional text had been introduced.

Specifically, Lines 157–161 explain that pruning sensitivity is used as empirical information for prior construction, while Lines 169–175 specify that the diagonal GGN is employed to compute the pruning sensitivity score in Eq. (8). Sections 3.2–3.3 further describe how these scores are mapped to prior precisions, rather than directly defining the posterior covariance. Section 2.1 specifies that the variational posterior is learned through ELBO optimization.

We have also revised Section 4.1 (Lines 241–248) to clarify that EB-Global and EB-Diagonal adapt marginal likelihood maximization to our pretrained solution-centered prior setting, following the formulation in Section 2.2, rather than directly implementing the zero-centered prior formulations of Immer et al. (2021) and Daxberger et al. (2021).

Re: Comment 3 (Comparison with Pretrained Deterministic Networks)

Thank you for pointing this out. We have now explicitly acknowledged in the Conclusion (Lines 534–535) that pretrained deterministic networks remain competitive and outperform their MFVI counterparts in some settings, highlighting the importance of careful prior design.

Re: Comment 4 (Statistical Motivation for the Proposed Prior Precision Allocation)

We apologize that our previous explanation did not sufficiently clarify the statistical motivation behind our approach. While our analysis provides a statistical motivation for PSS-based prior precision allocation, further theoretical investigation may help clarify its relationship with posterior predictive properties.

To clarify the connection to selective shrinkage, our approach applies shrinkage to deviations from the pretrained solution, δi=wi−wi∗\delta_i = w_i - w_i^*, rather than to the original weights. Under this reparameterization, the Gaussian prior becomes zero-centered, p(δi)=N(0,λi−1)p(\delta_i) = \mathcal{N}(0,\lambda_i^{-1}). Eq. (5) then provides a local statistical interpretation of how prior precision controls posterior flexibility, while PSS serves as an empirical criterion for allocating this precision across parameters (Eqs. 8–10).

Although this allocation is not theoretically guaranteed to be optimal, its predictive relevance is supported by the posterior predictive decomposition (Table 3), the precision ordering ablations (Table 4), and the loss landscape analysis (Figure 1). We acknowledge that establishing a more rigorous theoretical connection remains an important direction for future work.

We sincerely thank the reviewer again for the careful reading and constructive feedback. Your suggestions have helped us further clarify the scope, methodology, and limitations of our work. We hope that these additional clarifications and revisions adequately address your remaining concerns, and we greatly appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.