acceptodds
Under review as a conference paper at ICLR 2027

Attenuate, Don’t Drop: A Sign Entropy Approach to Weight Regularization

Abstract

Modern neural networks often achieve high accuracy while producing poorly calibrated confidence estimates. Regularization techniques like Dropout and DropConnect improve generalization by randomly masking activations or weights. However, they do not address calibration error. To address this, we propose a stability-guided posterior attenuation mechanism for Bayesian Neural Networks (BNNs). We use Sign Entropy, an information-theoretic quantity, to identify unstable weights and perform annealed attenuation on them. Sign Entropy measures the posterior uncertainty of the sign of each Bayesian weight, i.e., whether the weight provides positive or negative evidence. During training, we softly attenuate high Sign Entropy weights by jointly scaling their posterior mean and standard deviation. We provide theoretical justification by showing that, in the single-feature binary classification case, Sign Entropy is directly related to the posterior entropy of the predicted class, and we extend it to multi-feature settings. We demonstrate our results on classification and regression tasks, using two well-known Deep Learning (DL) backbones on two datasets, and show that the proposed method generally preserves predictive accuracy while improving probabilistic fit, calibration error, epistemic uncertainty, and predictive interval quality.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
dropout, sign entropy, dropconnect, deep learning, calibration, uncertainity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jus6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a training-time regularizer for Bayesian last layers. It selects weights with high posterior sign entropy and attenuates their means and standard deviations together through an annealed gate. Experiments on classification and regression report gains on several probabilistic metrics.

Critical strengths

The method is easy to implement. On classification, it improves NLL, Brier score, and calibration error while largely preserving predictive accuracy. That is a reasonable first signal that posterior-guided attenuation is worth pursuing.

Critical weaknesses
  1. Limited novelty. For Gaussian posteriors, sign entropy is strictly decreasing in |μ|/σ, so it has the same weights as low-SNR selection. What remains is mainly the soft attenuation rule and its training schedule. The paper doesn’t do enough to show these add value over existing posterior-guided regularization, BELIEF and sparse variational dropout included.

  2. The theoretical motivation also doesn’t really establish the claimed calibration mechanism. The single-feature result is about the entropy of posterior-sampled hard predictions, and jointly scaling the mean and standard deviation leaves that unchanged. With multiple features, attenuation changes both the logit mean and variance, so lower variance doesn’t imply more stable predictions. Neither result shows that predicted confidence becomes better aligned with actual correctness.

  3. The experiments don’t isolate the proposed mechanism either. Dropout and DropConnect comparisons change several factors at once. Missing: matched random attenuation, uniform shrinkage, simple SNR-based gates, and a temperature-scaling baseline. The joint-versus-mean-only ablation doesn’t establish that sign-entropy selection is what helps. Lower epistemic uncertainty is also treated as better without testing whether it’s reliable, and there’s no reporting of variability across random seeds.

Questions for authors
  1. Once attenuation strength and schedule are matched, does sign-entropy selection actually beat random selection and simple SNR-based gates?

  2. Joint scaling leaves each weight’s sign entropy unchanged. So what evidence ties the attenuation to better calibration, rather than just less posterior variation?

  3. Do the gains hold across random seeds? And do they survive after temperature scaling is applied to the baselines?

5 comments
AuthorsOpenReview ↗

Official comments to Reviewer jus6 by authors

We thank the reviewer for their insightful comments. We request the reviewer to consider our responses below:

W1: We have clarified that the novelty in the abstract and made changes to Section 3 Proposed Approach to better reflect the proposed approach as a single Sign-Entropy-guided, soft, annealed posterior attenuation mechanism. Additionally as suggested by the reviewer we have compared our proposed approach with magnitude based and SNR based pruning and posterior-guided regularization, BELIEF and sparse variational dropout. The proposed approach outperforms all methods in NLL, Brier, and AECE except Sparse Variational Dropout. Hence we softened the claims regarding lower calibration error in all cases to most cases in the abstract. The detailed analysis can be found in subsection A1.10 titled "Comparison with post-hoc pruning and matched train-time sparsification baselines". We are pasting the results below for the convenience of the reviewer:

Table: Fashion-MNIST classification performance for ResNet18 at drop ratios ρ = 0.1 and ρ = 0.3. Proposed is compared with Plain BNN, post-hoc SNR and magnitude pruning, BELIEF-style matched-ρ hard elimination, and matched-ρ Sparse Variational Dropout (SparseVD). F1, NLL, Brier, and AECE are reported as mean ± standard deviation over 50 MC posterior draws. Epistemic U is computed per test image from 50 posterior draws and reported as mean ± standard deviation across 10,000 test images, in units of 10⁻³. Higher F1 and lower NLL, Brier, and AECE are preferred. Lower Epistemic U indicates reduced posterior predictive disagreement and is interpreted jointly with the other metrics. Best mean values within each drop-ratio setting are highlighted in bold. Plain BNN and Proposed were reevaluated with fresh MC draws, so their values may differ slightly from Table A4.

Drop ratioMethodF1 ↑NLL ↓Brier ↓AECE ↓Epistemic U (×10⁻³) ↓
ρ = 0.1Plain BNN0.9413 ± 0.00100.3490 ± 0.00560.1259 ± 0.00230.1796 ± 0.00463.2918 ± 0.9262
ρ = 0.1SNR Pruning0.9414 ± 0.00090.3489 ± 0.00620.1257 ± 0.00230.1796 ± 0.00513.1152 ± 0.8773
ρ = 0.1Magnitude Pruning0.9411 ± 0.00090.3473 ± 0.00590.1253 ± 0.00220.1778 ± 0.00513.2020 ± 0.9122
ρ = 0.1BELIEF-style matched-ρ0.9421 ± 0.00080.3533 ± 0.00490.1286 ± 0.00180.1834 ± 0.00403.2386 ± 0.8029
ρ = 0.1SparseVD matched-ρ0.9417 ± 0.00080.2552 ± 0.00610.0960 ± 0.00170.0866 ± 0.00574.9205 ± 2.1568
ρ = 0.1Proposed (α = 5, γ = 1)0.9411 ± 0.00030.2668 ± 0.00040.0987 ± 0.00010.0846 ± 0.00040.0530 ± 0.0309
ρ = 0.3Plain BNN0.9371 ± 0.00110.3395 ± 0.00700.1256 ± 0.00280.1615 ± 0.00573.4366 ± 1.1023
ρ = 0.3SNR Pruning0.9373 ± 0.00110.3428 ± 0.00530.1265 ± 0.00200.1652 ± 0.00432.9186 ± 0.9187
ρ = 0.3Magnitude Pruning0.9373 ± 0.00090.3424 ± 0.00550.1262 ± 0.00210.1648 ± 0.00472.8962 ± 0.8613
ρ = 0.3BELIEF-style matched-ρ0.9440 ± 0.00060.3435 ± 0.00530.1231 ± 0.00190.1795 ± 0.00452.5431 ± 0.4821
ρ = 0.3SparseVD matched-ρ0.9387 ± 0.00100.2608 ± 0.00740.1003 ± 0.00180.0835 ± 0.00704.5049 ± 2.6941
ρ = 0.3Proposed (α = 50, γ = 1)0.9400 ± 0.00020.3047 ± 0.00040.1083 ± 0.00020.1204 ± 0.00030.0541 ± 0.0305

In addition to these experiments we also performed statistical tests to support our claims. These can be found in subsections: A1.7, A1.8, A1.9, and A1.10. We have also modified the conclusion to add future work regarding baselines, and models.

W2: We would like to clarify that our method does not claim that joint attenuation reduces the Sign Entropy of an individual weight. The intended role of Sign Entropy is to identify directionally uncertain weights whose influence on the prediction should be reduced.

To avoid any ambiguity, we have now made this property explicit with blue color in the revised manuscript in **Section 3.4, “Joint Posterior Attenuation,” page 5.

As the proposed joint attenuation scales the posterior mean and posterior standard deviation by the same positive retain factor, their ratio remains unchanged. For a Gaussian posterior, the posterior sign probability, and therefore the instantaneous Sign Entropy, depends on this ratio. Consequently, joint attenuation preserves the instantaneous Sign Entropy of the selected weight but reduces its influence.

AuthorsOpenReview ↗

Official comments to Reviewer jus6 by authors

Additionally, we have clarified the distinction between posterior-sampled hard predictions and predictive confidence in the single-feature case. In Appendix A1.1, supplementary p.2, in the paragraph beginning “It is also useful to distinguish posterior class-sign uncertainty...”, we now explain that, for the same posterior realization, positive joint attenuation preserves the sign of the sampled logit. Therefore this leaves the sampled hard prediction unchanged. At the same time, it reduces the magnitude of the logit, which contracts the corresponding binary confidence toward 0.5. Thus, the single-feature result is not intended to claim that attenuation changes posterior class-sign entropy; rather, it shows that attenuation can reduce the confidence contribution of a directionally uncertain weight even when its sampled class sign does not change. We also state that this confidence contraction by itself does not guarantee improved calibration, because calibration depends on the relationship between predicted confidence and empirical correctness.

Furthermore, we agree with the reviewer that in the multi-feature setting a reduction in posterior logit variance alone does not imply more stable predictions or improved calibration. We have made this limitation explicit in Appendix A1.9.1, supplementary p.20, in the final paragraph of “Connecting Sign Entropy and Predictive Uncertainty.” Joint attenuation reduces the variance contribution of the selected weights, but it also changes the posterior mean logit. Since the nonlinear mapping from logits to probabilities depends on the resulting mean logit as well, lower logit variance does not necessarily imply lower predictive-probability variance or improved calibration. We therefore present this analysis as a conditional theoretical motivation for attenuating directionally uncertain weights rather than as a calibration guarantee.

W3: We agree with the reviewer's comment that the joint-versus-mean-only ablation does not isolate Sign-Entropy-based selection in isolation. However, this is not the purpose of that ablation as Sign-Entropy-based selection alone is not the claimed contribution of our work. The proposed method is a coupled train-time mechanism in which Sign Entropy determines which posterior weights are attenuated and their relative attenuation strength, while a soft gate, annealing schedule, and joint attenuation of the posterior mean and standard deviation determine how that intervention is applied during training. We have modified Overview of Section 3 Proposed Approach of the paper to remove any ambiguity regarding the novelty of the paper. The modified text from the paper is given below:

The proposed regularizer is a single Sign-Entropy-guided, soft, annealed posterior attenuation mechanism. For weight jj at epoch tt, the complete attenuation coefficient is

aj(t)=η(t)sigmoid⁡(α(Ej−τ))Mj.a_j(t) = \eta(t) \operatorname{sigmoid}(\alpha(E_j-\tau)) M_j.

Here, Ej=Hs(wj)E_j = H_s(w_j) is the Sign Entropy, τ\tau is the Sign-Entropy quantile threshold, MjM_j selects the high-Sign-Entropy weights, α\alpha controls the softness of the gate, and η(t)\eta(t) controls how progressively the attenuation is introduced over training.

Thus, Sign Entropy determines both which posterior weights are attenuated and their relative attenuation strength, while η(t)\eta(t) controls the temporal introduction of this Sign-Entropy-conditioned attenuation. The selection, soft gate, and annealing schedule are therefore coupled components of the same attenuation rule rather than independent regularization mechanisms.

SNR-based alternatives.
For the Gaussian posterior used in our model, Sign Entropy is a monotonic function of the posterior signal-to-noise ratio. Therefore, ranking weights by high Sign Entropy is equivalent to ranking them by low posterior SNR. We have strengthened this relationship explicit in Appendix A1.1, “Sign Entropy and its relation to posterior SNR.” Consequently, a train-time SNR score passed through the same soft gate, annealing schedule, and joint posterior attenuation would amount to a reparameterization of the same mechanism rather than an independent baseline. However, we compared the proposed approach with SNR based and magnitude based pruning and also with BELIEF and sparse variational dropout as suggested by the reviewer. We have detailed about the same in our response to W1.

Similarly, uniform shrinkage would constitute a synthetic component-level control for testing targeted attenuation against global contraction; we have not included this additional factorial control and consider it useful future analysis.

AuthorsOpenReview ↗

Official comments to Reviewer jus6 by authors

Temperature scaling.
We added the requested post-hoc calibration comparison in Appendix A1.9, “Post-calibration comparison under per-draw temperature scaling.” For each of the 50 MC realizations, a scalar temperature is fitted on the held-out validation set by minimizing NLL and then applied to the corresponding test logits from the same posterior realization. We are putting the results in table below for the convenience of the reviewer:

Table: Fashion-MNIST performance after per-draw temperature scaling for ResNet18 at drop ratios rho = 0.1 and rho = 0.3. Each Monte Carlo (MC) realization is calibrated independently on the validation set and evaluated on the test set. F1, NLL, Brier, and AECE are reported as mean ± standard deviation over 50 MC realizations; their standard deviations reflect variability across draws. Epistemic uncertainty (Epistemic U) is computed separately for each test image using the 50 temperature-scaled MC realizations and reported as mean ± standard deviation across 10,000 test images; its standard deviation reflects variation across images rather than draws. Higher F1 and lower NLL, Brier, and AECE are preferable. Epistemic U is reported in units of 10^-3. Best mean values within each drop-ratio setting are highlighted in bold.

MethodF1 ↑NLL ↓Brier ↓AECE ↓Epistemic U (×10^-3)
ResNet18, drop ratio rho = 0.1
Plain BNN0.9413 ± 0.00080.1813 ± 0.00170.0882 ± 0.00080.0160 ± 0.00122.14387 ± 3.65757
Dropout0.9398 ± 0.00100.1894 ± 0.00230.0924 ± 0.00100.0147 ± 0.00112.54539 ± 4.11782
DropConnect0.9396 ± 0.00390.2097 ± 0.01890.0950 ± 0.00700.0192 ± 0.003615.95392 ± 19.66647
Concrete Dropout0.9506 ± 0.00020.1623 ± 0.00020.0779 ± 0.00010.0175 ± 0.00030.05833 ± 0.07154
Proposed0.9412 ± 0.00020.2032 ± 0.00040.0929 ± 0.00010.0212 ± 0.00050.03512 ± 0.06052
ResNet18, drop ratio rho = 0.3
Plain BNN0.9373 ± 0.00090.1945 ± 0.00200.0951 ± 0.00090.0173 ± 0.00122.30247 ± 3.71217
Dropout0.9327 ± 0.00100.2097 ± 0.00270.1021 ± 0.00110.0133 ± 0.00113.08145 ± 4.72008
DropConnect0.9447 ± 0.00300.1895 ± 0.01330.0868 ± 0.00550.0169 ± 0.001610.95807 ± 15.74237
Concrete Dropout0.9488 ± 0.00030.1641 ± 0.00020.0783 ± 0.00010.0172 ± 0.00050.09982 ± 0.12174
Proposed0.9400 ± 0.00020.2034 ± 0.00060.0942 ± 0.00010.0231 ± 0.00040.03901 ± 0.07671

The results show that the pre-calibration advantage of the proposed method on conventional calibration metrics is reduced when every method is allowed an independent post-hoc calibration step. However, it should be noted that the hyperparameters alpha and gamma of the proposed method were selected based on predictive accuracy, rather than by minimizing a calibration metric. In contrast, temperature scaling is a dedicated post-hoc calibration procedure, and in our experiment its temperature is fitted on the held-out validation set by minimizing NLL. Nonetheless, we acknowledge this as a limitation of our current evaluation and hence modified the conclusion for investigating in this direction. We believe future work can select the hyperparameters alpha and gamma based on calibration error and accuracy rather than only accuracy. This would enable a more directly matched comparison with calibration-specific post-hoc methods such as temperature scaling.

Regarding lower epistemic uncertainty
We agree with the reviewer that claiming lower epistemic certainty maybe problematic as the model may be overconfident. However, in our paper though our theoretical justification and experimental results (with newly added statistical tests) we are claiming lower epistemic uncertainty in conjunction with by comparable accuracy (F1/MAE) and lower calibration error. We are not claiming that lower epistemic uncertainty in isolation is desirable. To better reflect this distinction and avoid an overly general claim, we have revised the statement in the abstract to explicitly qualify our findings:

"... improving predictive interval quality and reducing calibration error and epistemic uncertainty in most cases." Additionally we have made changes to the captions of all tables to accommodate the same.

Reviewer_D23BOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a regularizer for Bayesian neural networks that aims to improve calibration. Dropout and DropConnect suppress activations or weights at random. This method instead looks at the Bayesian last layer, finds the weights whose sign is most uncertain under the posterior (their "Sign Entropy"), and gradually shrinks their influence during training. The motivation is that, in the simplest case, a weight's sign uncertainty equals the uncertainty of the predicted class. Damping sign-unstable weights should therefore make predictions more stable and better calibrated. Experiments cover image classification (Fashion-MNIST) and age regression (UTKFace) with VGG16 and ResNet18 backbones. They report similar accuracy, with better calibration and uncertainty metrics than a plain Bayesian network and Dropout-style regularizers.

Critical strengths

The method is simple and cheap, with no learnable parameters, and the paper is open about the equivalence to SNR ranking. Both classification and regression are evaluated with proper scoring rules.

Critical weaknesses
  • The experiments conducted are too narrow in terms of both the dataset, the model and baseline methods. ResNe18 and VGG16 are ok to start with, but it would be good to showcase the performance on more state-of-the-art methods, since the methodology itself is somewhat empirical and incremental. Also, how does the proposed method perform on larger and more realistic datasets? Temperature scaling, deep ensembles (cited in Sec. 6) and last-layer Laplace are all missing as important baseline methods.
  • The reported results contradict each other, and none are replicated. For instance, In Table 1for VGG16 at ρ=0.1 gives the proposed method F1/NLL/AECE of 0.9247/0.3283/0.1235. Based on my understanding, this results should be the same as Table 3's μ+σ row, yet it gives 0.9213/0.3804/0.1627.

Overall, I think the empirical results is lacking in general. As such, I am inclined towards rejecting the paper for now.

Questions for authors
  • Can you address the discrepancies in the results? Is it caused by different runs/seeds? if so, can you report standard deviation?
2 comments
AuthorsOpenReview ↗

Official comments to Reviewer D23B by authors

We thank the reviewer for their insightful comments. We request the reviewer to kindly consider our responses below:

W1: We acknowledge that the the proposed method needs to be evaluated on other datasets and baseline. In this regard, we added the results for CIFAR-10 dataset in subsection A1.6 CLASSIFICATION RESULTS ON CIFAR-10 DATASET of appendix. We are including the table below for the convenience of the reviewer.

Table: CIFAR-10 classification results for ResNet18 at ρ = 0.3. F1, NLL, Brier, and AECE are averaged over 50 MC posterior draws; Epistemic U measures posterior predictive disagreement across draws. Higher F1 and lower NLL, Brier, and AECE are preferred. Best scores and lowest Epistemic U are highlighted in bold.

ModelF1 ↑NLL ↓Brier ↓AECE ↓Epistemic U ↓
Concrete Dropout0.94420.34270.11830.16860.00022
Plain BNN0.94900.25030.08800.08560.0020
Dropout0.94870.37630.12790.19940.0023
DropConnect0.95430.25390.08400.09560.0045
Proposed0.95140.18410.07550.01950.00014

Additionally, we also conducted experiments on SNR and magnitude based pruning and BELIEF and Sparse Variational Dropout and our methods outperforms all methods in NLL, BRIER, and AECE except Sparse Variational Dropout. Hence we softened the claims regarding lower calibration error in all cases to most cases in the abstract. The detailed analysis can be found in subsection A1.10 titled "Comparison with post-hoc pruning and matched train-time sparsification baselines". We are pasting the results below for the convenience of the reviewer:

Table: Fashion-MNIST classification performance for ResNet18 at drop ratios ρ = 0.1 and ρ = 0.3. Proposed is compared with Plain BNN, post-hoc SNR and magnitude pruning, BELIEF-style matched-ρ hard elimination, and matched-ρ Sparse Variational Dropout (SparseVD). F1, NLL, Brier, and AECE are reported as mean ± standard deviation over 50 MC posterior draws. Epistemic U is computed per test image from 50 posterior draws and reported as mean ± standard deviation across 10,000 test images, in units of 10⁻³. Higher F1 and lower NLL, Brier, and AECE are preferred. Lower Epistemic U indicates reduced posterior predictive disagreement and is interpreted jointly with the other metrics. Best mean values within each drop-ratio setting are highlighted in bold. Plain BNN and Proposed were reevaluated with fresh MC draws, so their values may differ slightly from Table A4.

Drop ratioMethodF1 ↑NLL ↓Brier ↓AECE ↓Epistemic U (×10⁻³) ↓
ρ = 0.1Plain BNN0.9413 ± 0.00100.3490 ± 0.00560.1259 ± 0.00230.1796 ± 0.00463.2918 ± 0.9262
ρ = 0.1SNR Pruning0.9414 ± 0.00090.3489 ± 0.00620.1257 ± 0.00230.1796 ± 0.00513.1152 ± 0.8773
ρ = 0.1Magnitude Pruning0.9411 ± 0.00090.3473 ± 0.00590.1253 ± 0.00220.1778 ± 0.00513.2020 ± 0.9122
ρ = 0.1BELIEF-style matched-ρ0.9421 ± 0.00080.3533 ± 0.00490.1286 ± 0.00180.1834 ± 0.00403.2386 ± 0.8029
ρ = 0.1SparseVD matched-ρ0.9417 ± 0.00080.2552 ± 0.00610.0960 ± 0.00170.0866 ± 0.00574.9205 ± 2.1568
ρ = 0.1Proposed (α = 5, γ = 1)0.9411 ± 0.00030.2668 ± 0.00040.0987 ± 0.00010.0846 ± 0.00040.0530 ± 0.0309
ρ = 0.3Plain BNN0.9371 ± 0.00110.3395 ± 0.00700.1256 ± 0.00280.1615 ± 0.00573.4366 ± 1.1023
ρ = 0.3SNR Pruning0.9373 ± 0.00110.3428 ± 0.00530.1265 ± 0.00200.1652 ± 0.00432.9186 ± 0.9187
ρ = 0.3Magnitude Pruning0.9373 ± 0.00090.3424 ± 0.00550.1262 ± 0.00210.1648 ± 0.00472.8962 ± 0.8613
ρ = 0.3BELIEF-style matched-ρ0.9440 ± 0.00060.3435 ± 0.00530.1231 ± 0.00190.1795 ± 0.00452.5431 ± 0.4821
ρ = 0.3SparseVD matched-ρ0.9387 ± 0.00100.2608 ± 0.00740.1003 ± 0.00180.0835 ± 0.00704.5049 ± 2.6941
ρ = 0.3Proposed (α = 50, γ = 1)0.9400 ± 0.00020.3047 ± 0.00040.1083 ± 0.00020.1204 ± 0.00030.0541 ± 0.0305

In addition to these experiments we also performed statistical tests to support our claims. These can be found in subsections: A1.7, A1.8, A1.9, and A1.10. We have also modified the conclusion to add future work regarding baselines, and models.

We agree that deep ensembles and last-layer Laplace would further broaden the comparison; these are not included in the current revision and have been added to the limitations/future-work discussion.

AuthorsOpenReview ↗

Official comments to Reviewer D23B by authors

W2: We can understand the confusion associated with the difference in scores for table 1 and table3. This was because of the different choices of alpha and gamma hyperparameters. We have now included the used hyperparameters along with the results in the table and the caption. Additionally we added a footnote: "All joint and μ\mu-only models were trained using the same fixed hyperparameter setting, (α,γ)=(20,1)(\alpha,\gamma)=(20,1). We intentionally used this common setting, rather than the architecture- and drop-ratio-specific hyperparameters selected for the main classification experiments, to avoid evaluating the μ\mu-only variant with hyperparameters that had been optimized specifically for joint attenuation." on page 9 regarding it.

Q1: As stated for W2 the difference in scores were not because of different seeds but because of different hyperparameters. We have updated the tables and also added explanations regarding the same. We have also added standard deviation and also performed statistical tests to support our claims. The details of the statistical tests can be found in A1.7, A1.8, A1.9, and A1.10. We have also performed experiments to understand the impact of different seeds on the observations of the experiments and we saw that across 3 random seeds the results and observations were consistent with those of table 1. The details of the multiple seeds experiment can be found in A1.11 INDEPENDENT-SEED ROBUSTNESS ANALYSIS of the supplementary. We are putting the table below for the convenience of the reviewer:

Table: Independent-seed robustness on Fashion-MNIST using ResNet18 at ρ=0.1\rho = 0.1, evaluated over three training seeds (42, 118, and 563). The Mean ± SD rows summarize across-seed variability. Epistemic U is the mean per-image uncertainty estimated from 50 MC posterior samples over 10,000 test images and is reported in units of 10−310^{-3}. Higher Macro-F1 and lower NLL, Brier, and AECE are preferred; lower Epistemic U indicates less posterior predictive disagreement. Best metric values and lowest Epistemic U within each seed and across-seed means are highlighted in bold.

MethodSeedMacro-F1 ↑NLL ↓Brier ↓AECE ↓Epistemic U (×10−3\times 10^{-3}) ↓
Plain BNN420.94330.34250.12230.17841.8673
Plain BNN1180.94160.34070.12260.17461.8314
Plain BNN5630.94060.33950.12420.17021.8119
Plain BNNMean ± SD0.9418 ± 0.00140.3409 ± 0.00150.1230 ± 0.00100.1744 ± 0.00411.8368 ± 0.0281
Dropout420.93870.35630.12910.17841.8492
Dropout1180.94110.35640.12790.18271.7913
Dropout5630.94110.36910.13330.19481.8204
DropoutMean ± SD0.9403 ± 0.00140.3606 ± 0.00740.1301 ± 0.00290.1853 ± 0.00851.8203 ± 0.0289
DropConnect420.94750.32400.11250.172226.0522
DropConnect1180.95060.31220.10850.167822.9332
DropConnect5630.94980.31270.10690.165323.2308
DropConnectMean ± SD0.9493 ± 0.00160.3163 ± 0.00670.1093 ± 0.00290.1684 ± 0.003524.0721 ± 1.7213
Concrete Dropout420.94860.32200.11250.17230.2246
Concrete Dropout1180.94920.31740.11010.16910.2152
Concrete Dropout5630.94710.31440.10930.16430.1958
Concrete DropoutMean ± SD0.9483 ± 0.00110.3179 ± 0.00380.1106 ± 0.00170.1685 ± 0.00400.2119 ± 0.0147
Proposed420.94040.23960.09250.04960.0370
Proposed1180.93940.25140.09580.07170.0463
Proposed5630.94170.26360.09640.07870.0463
ProposedMean ± SD0.9405 ± 0.00120.2515 ± 0.01200.0949 ± 0.00210.0667 ± 0.01520.0432 ± 0.0054
Reviewer_tZMmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a sign-entropy-guided posterior attenuation method for Bayesian neural networks. The method identifies final-layer weights whose Gaussian posteriors are uncertain about their sign and annealedly scales down both their posterior mean and standard deviation during training. Experiments on classification and regression show improved metrics relative to the selected baselines.

Critical strengths
  • The paper targets an important problem: Bayesian models may provide uncertainty estimates but still produce poorly calibrated predictive distributions.
  • The proposed mechanism is simple, computationally lightweight, and easy to integrate into a variational BNN.
  • The method is evaluated on both classification and regression, using several relevant metrics.
Critical weaknesses
  • The datasets are limited. There is only one dataset for classification and one for regression. The backbone is deterministic except for the final layer, and the Bayesian treatment is not evaluated in deeper or more advanced model architectures. This makes it difficult to assess the generalizability beyond relatively simple settings.
  • The baselines selected are too restricted. More uncertainty quantification methods can strengthen the comparison.
  • Claiming method with lower epistemic uncertainty is better is problematic. A method can be overconfident and always gives low epistemic uncertainty.
  • In regression, the proposed method has worse performance than existing ones in several metrics.
  • Single-seed experiments might be unstable. Multiple runs with statistical significance can further enhance the claims.
Questions for authors

Authors could refer to the aforementioned weaknesses for rebuttal.

4 comments
AuthorsOpenReview ↗

Official comments to Reviewer tZMm by authors

We thank the reviewer for their insightful comments. We request the reviewer to kindly look at our response below:

W1: We acknowledge that the number of datasets are somewhat limited and stated the same in the conclusion. Additionally, we ran the proposed method for the Resnet18 backbone variant with drop ratio=0.3 and setting \alpha=5 and \gamma=1 and the results were similar to fashion-mnist.

Table: CIFAR-10 classification results for ResNet18 at ρ = 0.3. F1, NLL, Brier, and AECE are averaged over 50 MC posterior draws; Epistemic U measures posterior predictive disagreement across draws. Higher F1 and lower NLL, Brier, and AECE are preferred. Best scores and lowest Epistemic U are highlighted in bold.

ModelF1 ↑NLL ↓Brier ↓AECE ↓Epistemic U ↓
Concrete Dropout0.94420.34270.11830.16860.00022
Plain BNN0.94900.25030.08800.08560.0020
Dropout0.94870.37630.12790.19940.0023
DropConnect0.95430.25390.08400.09560.0045
Proposed0.95140.18410.07550.01950.00014

W2: We added SNR and magnitude based pruning and BELIEF and Sparse Variational Dropout and our methods outperforms all methods in NLL, Brier and AECE except Sparse Variational Dropout. Hence we softened the claims regarding lower calibration error in all cases to most cases in the abstract. The detailed analysis can be found in subsection A1.10 titled "Comparison with post-hoc pruning and matched train-time sparsification baselines". We are pasting the results below for the convenience of the reviewer:

Table: Fashion-MNIST classification performance for ResNet18 at drop ratios ρ = 0.1 and ρ = 0.3. Proposed is compared with Plain BNN, post-hoc SNR and magnitude pruning, BELIEF-style matched-ρ hard elimination, and matched-ρ Sparse Variational Dropout (SparseVD). F1, NLL, Brier, and AECE are reported as mean ± standard deviation over 50 MC posterior draws. Epistemic U is computed per test image from 50 posterior draws and reported as mean ± standard deviation across 10,000 test images, in units of 10⁻³. Higher F1 and lower NLL, Brier, and AECE are preferred. Lower Epistemic U indicates reduced posterior predictive disagreement and is interpreted jointly with the other metrics. Best mean values within each drop-ratio setting are highlighted in bold.

Drop ratioMethodF1 ↑NLL ↓Brier ↓AECE ↓Epistemic U (×10⁻³) ↓
ρ = 0.1Plain BNN0.9413 ± 0.00100.3490 ± 0.00560.1259 ± 0.00230.1796 ± 0.00463.2918 ± 0.9262
ρ = 0.1SNR Pruning0.9414 ± 0.00090.3489 ± 0.00620.1257 ± 0.00230.1796 ± 0.00513.1152 ± 0.8773
ρ = 0.1Magnitude Pruning0.9411 ± 0.00090.3473 ± 0.00590.1253 ± 0.00220.1778 ± 0.00513.2020 ± 0.9122
ρ = 0.1BELIEF-style matched-ρ0.9421 ± 0.00080.3533 ± 0.00490.1286 ± 0.00180.1834 ± 0.00403.2386 ± 0.8029
ρ = 0.1SparseVD matched-ρ0.9417 ± 0.00080.2552 ± 0.00610.0960 ± 0.00170.0866 ± 0.00574.9205 ± 2.1568
ρ = 0.1Proposed (α = 5, γ = 1)0.9411 ± 0.00030.2668 ± 0.00040.0987 ± 0.00010.0846 ± 0.00040.0530 ± 0.0309
ρ = 0.3Plain BNN0.9371 ± 0.00110.3395 ± 0.00700.1256 ± 0.00280.1615 ± 0.00573.4366 ± 1.1023
ρ = 0.3SNR Pruning0.9373 ± 0.00110.3428 ± 0.00530.1265 ± 0.00200.1652 ± 0.00432.9186 ± 0.9187
ρ = 0.3Magnitude Pruning0.9373 ± 0.00090.3424 ± 0.00550.1262 ± 0.00210.1648 ± 0.00472.8962 ± 0.8613
ρ = 0.3BELIEF-style matched-ρ0.9440 ± 0.00060.3435 ± 0.00530.1231 ± 0.00190.1795 ± 0.00452.5431 ± 0.4821
ρ = 0.3SparseVD matched-ρ0.9387 ± 0.00100.2608 ± 0.00740.1003 ± 0.00180.0835 ± 0.00704.5049 ± 2.6941
ρ = 0.3Proposed (α = 50, γ = 1)0.9400 ± 0.00020.3047 ± 0.00040.1083 ± 0.00020.1204 ± 0.00030.0541 ± 0.0305

W3: We agree with the reviewer that claiming lower epistemic certainty maybe problematic as the model may be overconfident. However, in our paper though our theoretical justification and experimental results (with newly added statistical tests) we are claiming lower epistemic uncertainty accompanied by comparable accuracy (F1/MAE) and lower calibration error in most cases. We acknowledge that these improvements are not uniform across all experimental configurations. To better reflect this distinction and avoid an overly general claim, we have revised the statement in the abstract to explicitly qualify our findings:

"... improving predictive interval quality and reducing calibration error and epistemic uncertainty in most cases." Additionally we have made changes to the captions of all tables to accomodate the same.

AuthorsOpenReview ↗

Official comments to Reviewer tZMm by authors

W4: We acknowledge that our proposed method does not perform best in all cases for regression. We therefore performed analysis using Borda count to measure overall ranking across metrics. The metrics were treated as voters and we performed 10,000 paired bootstrap resamples of the test images to take into account the uncertainty associated with the ranks of the methods for each metric. Our results showed that the proposed method had the highest rank using the Borda count analysis (with and without MPIW90). The details of the analysis can be found in subsection A1.8.1 GLOBAL BOOTSTRAP BORDA ANALYSIS FOR REGRESSION of the supplementary. We are pasting the main tables of the analysis below for the convenience of the reviewer:

Table: Global Borda ranking for UTKFace age regression. The analysis covers four architecture–drop-ratio configurations, using five metrics per configuration (20 voters; maximum score 80) and excluding MPIW90 (16 voters; maximum score 64). Results summarize 10,000 paired test-image bootstrap repetitions, with identical resampled images across all methods and configurations. Bootstrap means are reported with standard deviations (SD), and 95% confidence intervals (CI) are percentile intervals. Pr(best) includes ties, while Pr(unique) denotes the frequency of sole winners.

20 voters: MAE, NLL, Brier, CE90, MPIW90

MethodObserved BordaBootstrap mean ± SD95% CIMean rankPr(best)Pr(unique)
Proposed72/8069.7162 ± 1.8896[66, 73]1.00021.00000.9997
Concrete Dropout55/8056.4697 ± 2.2846[52, 61]1.99990.00030.0000
Dropout41/8040.7482 ± 1.9250[37, 44]3.01230.00000.0000
Plain BNN32/8033.0660 ± 1.7892[30, 37]3.98780.00000.0000
DropConnect0/800.0000 ± 0.0000[0, 0]5.00000.00000.0000

16 voters: MAE, NLL, Brier, CE90 (excluding MPIW90)

MethodObserved BordaBootstrap mean ± SD95% CIMean rankPr(best)Pr(unique)
Proposed56/6453.9094 ± 1.8182[50, 57]1.08480.93340.8971
Concrete Dropout48/6449.1855 ± 2.1164[45, 53]1.91530.10290.0666
Dropout32/6431.9051 ± 1.8478[28, 35]3.01760.00000.0000
Plain BNN24/6425.0001 ± 1.6414[22, 28.5]3.98250.00000.0000
DropConnect0/640.0000 ± 0.0000[0, 0]5.00000.00000.0000

Table: Paired image-bootstrap Borda-score differences between Proposed and each baseline. Positive differences favor Proposed. Bootstrap SD describes the variability of paired score differences across 10,000 image-bootstrap repetitions, and 95% CI denotes the percentile confidence interval. Pr(>), Pr(=), and Pr(<) represent the proportions of repetitions where Proposed scores higher than, equal to, or lower than the baseline, respectively. These are not classical p-values.

20 voters: MAE, NLL, Brier, CE90, MPIW90

BaselineObserved ΔBMean ΔBBootstrap SD95% CIPr(>)Pr(=) / Pr(<)
Plain BNN+40+36.65022.9304[31, 42]1.00000.0000 / 0.0000
Dropout+31+28.96803.0005[23, 35]1.00000.0000 / 0.0000
DropConnect+72+69.71621.8896[66, 73]1.00000.0000 / 0.0000
Concrete Dropout+17+13.24653.5197[6, 20]0.99970.0003 / 0.0000

16 voters: MAE, NLL, Brier, CE90 (excluding MPIW90)

BaselineObserved ΔBMean ΔBBootstrap SD95% CIPr(>)Pr(=) / Pr(<)
Plain BNN+32+28.90932.6979[23, 34]1.00000.0000 / 0.0000
Dropout+24+22.00432.8835[16.9875, 28]1.00000.0000 / 0.0000
DropConnect+56+53.90941.8182[50, 57]1.00000.0000 / 0.0000
Concrete Dropout+8+4.72393.3948[-2, 11]0.89710.0363 / 0.0666

The proposed method achieves the highest observed Borda score in both analyses. Including MPIW90, its advantage over Concrete Dropout remains positive throughout the 95% bootstrap confidence interval while excluding MPIW90, the proposed method still outperforms Concrete Dropout in 89.71% of bootstrap repetitions. However, the confidence interval of the score difference includes zero in with analysis done without MPIW90, indicating a less robust ranking advantage.

AuthorsOpenReview ↗

Official comments to Reviewer tZMm by authors

W5: To understand the effect of mutiple seeds, we ran the ResNet18 backbone model with drop ratio=0.1 with seeds 42, 118 and 563. The train, val and test splits were kept exactly the same. Our results showed that the results are same with the results reported in Table 1 i.e., the proposed method maintained comparable accuracy while lower calibration error and epistemic uncertainty in comparison to Dropout, Dropconnect, Concrete dropout and pain methods. The details can be found in the subsection: A1.11 INDEPENDENT-SEED ROBUSTNESS ANALYSIS of the supplementary. We are putting the table below for the convenience of the reviewer:

Table: Independent-seed robustness on Fashion-MNIST using ResNet18 at ρ = 0.1, evaluated over three training seeds (42, 118, and 563). The Mean ± SD rows summarize across-seed variability. Epistemic U is the mean per-image uncertainty estimated from 50 MC posterior samples over 10,000 test images and is reported in units of 10⁻³. Higher Macro-F1 and lower NLL, Brier, and AECE are preferred; lower Epistemic U indicates less posterior predictive disagreement. Best metric values and lowest Epistemic U within each seed and across-seed means are highlighted in bold.

MethodSeedMacro-F1 ↑NLL ↓Brier ↓AECE ↓Epistemic U (×10⁻³) ↓
Plain BNN420.94330.34250.12230.17841.8673
Plain BNN1180.94160.34070.12260.17461.8314
Plain BNN5630.94060.33950.12420.17021.8119
Plain BNNMean ± SD0.9418 ± 0.00140.3409 ± 0.00150.1230 ± 0.00100.1744 ± 0.00411.8368 ± 0.0281
Dropout420.93870.35630.12910.17841.8492
Dropout1180.94110.35640.12790.18271.7913
Dropout5630.94110.36910.13330.19481.8204
DropoutMean ± SD0.9403 ± 0.00140.3606 ± 0.00740.1301 ± 0.00290.1853 ± 0.00851.8203 ± 0.0289
DropConnect420.94750.32400.11250.172226.0522
DropConnect1180.95060.31220.10850.167822.9332
DropConnect5630.94980.31270.10690.165323.2308
DropConnectMean ± SD0.9493 ± 0.00160.3163 ± 0.00670.1093 ± 0.00290.1684 ± 0.003524.0721 ± 1.7213
Concrete Dropout420.94860.32200.11250.17230.2246
Concrete Dropout1180.94920.31740.11010.16910.2152
Concrete Dropout5630.94710.31440.10930.16430.1958
Concrete DropoutMean ± SD0.9483 ± 0.00110.3179 ± 0.00380.1106 ± 0.00170.1685 ± 0.00400.2119 ± 0.0147
Proposed420.94040.23960.09250.04960.0370
Proposed1180.93940.25140.09580.07170.0463
Proposed5630.94170.26360.09640.07870.0463
ProposedMean ± SD0.9405 ± 0.00120.2515 ± 0.01200.0949 ± 0.00210.0667 ± 0.01520.0432 ± 0.0054

I have gone through the authors rebuttal, which has partially addressed the weaknesses. I still decide to keep my previous rating after reading everyone's discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thanks to all reviewers

We thank all reviewers for their insightful comments which helped us to improve the manuscript. We have highlighted the newly added text in blue in the revised version. For the reviewers' convenience, we have pasted some of the tables and results here in the comments section. However, we kindly request that the reviewers look at the main paper and the appendix for details of the changes made in response to their review comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.