- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a training-time regularizer for Bayesian last layers. It selects weights with high posterior sign entropy and attenuates their means and standard deviations together through an annealed gate. Experiments on classification and regression report gains on several probabilistic metrics.
Critical strengths
The method is easy to implement. On classification, it improves NLL, Brier score, and calibration error while largely preserving predictive accuracy. That is a reasonable first signal that posterior-guided attenuation is worth pursuing.
Critical weaknesses
-
Limited novelty. For Gaussian posteriors, sign entropy is strictly decreasing in |μ|/σ, so it has the same weights as low-SNR selection. What remains is mainly the soft attenuation rule and its training schedule. The paper doesn’t do enough to show these add value over existing posterior-guided regularization, BELIEF and sparse variational dropout included.
-
The theoretical motivation also doesn’t really establish the claimed calibration mechanism. The single-feature result is about the entropy of posterior-sampled hard predictions, and jointly scaling the mean and standard deviation leaves that unchanged. With multiple features, attenuation changes both the logit mean and variance, so lower variance doesn’t imply more stable predictions. Neither result shows that predicted confidence becomes better aligned with actual correctness.
-
The experiments don’t isolate the proposed mechanism either. Dropout and DropConnect comparisons change several factors at once. Missing: matched random attenuation, uniform shrinkage, simple SNR-based gates, and a temperature-scaling baseline. The joint-versus-mean-only ablation doesn’t establish that sign-entropy selection is what helps. Lower epistemic uncertainty is also treated as better without testing whether it’s reliable, and there’s no reporting of variability across random seeds.
Questions for authors
-
Once attenuation strength and schedule are matched, does sign-entropy selection actually beat random selection and simple SNR-based gates?
-
Joint scaling leaves each weight’s sign entropy unchanged. So what evidence ties the attenuation to better calibration, rather than just less posterior variation?
-
Do the gains hold across random seeds? And do they survive after temperature scaling is applied to the baselines?
5 comments
Official comments to Reviewer jus6 by authors
Additionally, we have clarified the distinction between posterior-sampled hard predictions and predictive confidence in the single-feature case. In Appendix A1.1, supplementary p.2, in the paragraph beginning “It is also useful to distinguish posterior class-sign uncertainty...”, we now explain that, for the same posterior realization, positive joint attenuation preserves the sign of the sampled logit. Therefore this leaves the sampled hard prediction unchanged. At the same time, it reduces the magnitude of the logit, which contracts the corresponding binary confidence toward 0.5. Thus, the single-feature result is not intended to claim that attenuation changes posterior class-sign entropy; rather, it shows that attenuation can reduce the confidence contribution of a directionally uncertain weight even when its sampled class sign does not change. We also state that this confidence contraction by itself does not guarantee improved calibration, because calibration depends on the relationship between predicted confidence and empirical correctness.
Furthermore, we agree with the reviewer that in the multi-feature setting a reduction in posterior logit variance alone does not imply more stable predictions or improved calibration. We have made this limitation explicit in Appendix A1.9.1, supplementary p.20, in the final paragraph of “Connecting Sign Entropy and Predictive Uncertainty.” Joint attenuation reduces the variance contribution of the selected weights, but it also changes the posterior mean logit. Since the nonlinear mapping from logits to probabilities depends on the resulting mean logit as well, lower logit variance does not necessarily imply lower predictive-probability variance or improved calibration. We therefore present this analysis as a conditional theoretical motivation for attenuating directionally uncertain weights rather than as a calibration guarantee.
W3: We agree with the reviewer's comment that the joint-versus-mean-only ablation does not isolate Sign-Entropy-based selection in isolation. However, this is not the purpose of that ablation as Sign-Entropy-based selection alone is not the claimed contribution of our work. The proposed method is a coupled train-time mechanism in which Sign Entropy determines which posterior weights are attenuated and their relative attenuation strength, while a soft gate, annealing schedule, and joint attenuation of the posterior mean and standard deviation determine how that intervention is applied during training. We have modified Overview of Section 3 Proposed Approach of the paper to remove any ambiguity regarding the novelty of the paper. The modified text from the paper is given below:
The proposed regularizer is a single Sign-Entropy-guided, soft, annealed posterior attenuation mechanism. For weight at epoch , the complete attenuation coefficient is
Here, is the Sign Entropy, is the Sign-Entropy quantile threshold, selects the high-Sign-Entropy weights, controls the softness of the gate, and controls how progressively the attenuation is introduced over training.
Thus, Sign Entropy determines both which posterior weights are attenuated and their relative attenuation strength, while controls the temporal introduction of this Sign-Entropy-conditioned attenuation. The selection, soft gate, and annealing schedule are therefore coupled components of the same attenuation rule rather than independent regularization mechanisms.
SNR-based alternatives.
For the Gaussian posterior used in our model, Sign Entropy is a monotonic function of the posterior signal-to-noise ratio. Therefore, ranking weights by high Sign Entropy is equivalent to ranking them by low posterior SNR. We have strengthened this relationship explicit in Appendix A1.1, “Sign Entropy and its relation to posterior SNR.” Consequently, a train-time SNR score passed through the same soft gate, annealing schedule, and joint posterior attenuation would amount to a reparameterization of the same mechanism rather than an independent baseline. However, we compared the proposed approach with SNR based and magnitude based pruning and also with BELIEF and sparse variational dropout as suggested by the reviewer. We have detailed about the same in our response to W1.
Similarly, uniform shrinkage would constitute a synthetic component-level control for testing targeted attenuation against global contraction; we have not included this additional factorial control and consider it useful future analysis.
Official comments to Reviewer jus6 by authors
Temperature scaling.
We added the requested post-hoc calibration comparison in Appendix A1.9, “Post-calibration comparison under per-draw temperature scaling.” For each of the 50 MC realizations, a scalar temperature is fitted on the held-out validation set by minimizing NLL and then applied to the corresponding test logits from the same posterior realization. We are putting the results in table below for the convenience of the reviewer:
Table: Fashion-MNIST performance after per-draw temperature scaling for ResNet18 at drop ratios rho = 0.1 and rho = 0.3. Each Monte Carlo (MC) realization is calibrated independently on the validation set and evaluated on the test set. F1, NLL, Brier, and AECE are reported as mean ± standard deviation over 50 MC realizations; their standard deviations reflect variability across draws. Epistemic uncertainty (Epistemic U) is computed separately for each test image using the 50 temperature-scaled MC realizations and reported as mean ± standard deviation across 10,000 test images; its standard deviation reflects variation across images rather than draws. Higher F1 and lower NLL, Brier, and AECE are preferable. Epistemic U is reported in units of 10^-3. Best mean values within each drop-ratio setting are highlighted in bold.
| Method | F1 ↑ | NLL ↓ | Brier ↓ | AECE ↓ | Epistemic U (×10^-3) |
|---|---|---|---|---|---|
| ResNet18, drop ratio rho = 0.1 | |||||
| Plain BNN | 0.9413 ± 0.0008 | 0.1813 ± 0.0017 | 0.0882 ± 0.0008 | 0.0160 ± 0.0012 | 2.14387 ± 3.65757 |
| Dropout | 0.9398 ± 0.0010 | 0.1894 ± 0.0023 | 0.0924 ± 0.0010 | 0.0147 ± 0.0011 | 2.54539 ± 4.11782 |
| DropConnect | 0.9396 ± 0.0039 | 0.2097 ± 0.0189 | 0.0950 ± 0.0070 | 0.0192 ± 0.0036 | 15.95392 ± 19.66647 |
| Concrete Dropout | 0.9506 ± 0.0002 | 0.1623 ± 0.0002 | 0.0779 ± 0.0001 | 0.0175 ± 0.0003 | 0.05833 ± 0.07154 |
| Proposed | 0.9412 ± 0.0002 | 0.2032 ± 0.0004 | 0.0929 ± 0.0001 | 0.0212 ± 0.0005 | 0.03512 ± 0.06052 |
| ResNet18, drop ratio rho = 0.3 | |||||
| Plain BNN | 0.9373 ± 0.0009 | 0.1945 ± 0.0020 | 0.0951 ± 0.0009 | 0.0173 ± 0.0012 | 2.30247 ± 3.71217 |
| Dropout | 0.9327 ± 0.0010 | 0.2097 ± 0.0027 | 0.1021 ± 0.0011 | 0.0133 ± 0.0011 | 3.08145 ± 4.72008 |
| DropConnect | 0.9447 ± 0.0030 | 0.1895 ± 0.0133 | 0.0868 ± 0.0055 | 0.0169 ± 0.0016 | 10.95807 ± 15.74237 |
| Concrete Dropout | 0.9488 ± 0.0003 | 0.1641 ± 0.0002 | 0.0783 ± 0.0001 | 0.0172 ± 0.0005 | 0.09982 ± 0.12174 |
| Proposed | 0.9400 ± 0.0002 | 0.2034 ± 0.0006 | 0.0942 ± 0.0001 | 0.0231 ± 0.0004 | 0.03901 ± 0.07671 |
The results show that the pre-calibration advantage of the proposed method on conventional calibration metrics is reduced when every method is allowed an independent post-hoc calibration step. However, it should be noted that the hyperparameters alpha and gamma of the proposed method were selected based on predictive accuracy, rather than by minimizing a calibration metric. In contrast, temperature scaling is a dedicated post-hoc calibration procedure, and in our experiment its temperature is fitted on the held-out validation set by minimizing NLL. Nonetheless, we acknowledge this as a limitation of our current evaluation and hence modified the conclusion for investigating in this direction. We believe future work can select the hyperparameters alpha and gamma based on calibration error and accuracy rather than only accuracy. This would enable a more directly matched comparison with calibration-specific post-hoc methods such as temperature scaling.
Regarding lower epistemic uncertainty
We agree with the reviewer that claiming lower epistemic certainty maybe problematic as the model may be overconfident. However, in our paper though our theoretical justification and experimental results (with newly added statistical tests) we are claiming lower epistemic uncertainty in conjunction with by comparable accuracy (F1/MAE) and lower calibration error. We are not claiming that lower epistemic uncertainty in isolation is desirable. To better reflect this distinction and avoid an overly general claim, we have revised the statement in the abstract to explicitly qualify our findings:
"... improving predictive interval quality and reducing calibration error and epistemic uncertainty in most cases." Additionally we have made changes to the captions of all tables to accommodate the same.
Official comments to Reviewer jus6 by authors
We thank the reviewer for their insightful comments. We request the reviewer to consider our responses below:
W1: We have clarified that the novelty in the abstract and made changes to Section 3 Proposed Approach to better reflect the proposed approach as a single Sign-Entropy-guided, soft, annealed posterior attenuation mechanism. Additionally as suggested by the reviewer we have compared our proposed approach with magnitude based and SNR based pruning and posterior-guided regularization, BELIEF and sparse variational dropout. The proposed approach outperforms all methods in NLL, Brier, and AECE except Sparse Variational Dropout. Hence we softened the claims regarding lower calibration error in all cases to most cases in the abstract. The detailed analysis can be found in subsection A1.10 titled "Comparison with post-hoc pruning and matched train-time sparsification baselines". We are pasting the results below for the convenience of the reviewer:
Table: Fashion-MNIST classification performance for ResNet18 at drop ratios ρ = 0.1 and ρ = 0.3. Proposed is compared with Plain BNN, post-hoc SNR and magnitude pruning, BELIEF-style matched-ρ hard elimination, and matched-ρ Sparse Variational Dropout (SparseVD). F1, NLL, Brier, and AECE are reported as mean ± standard deviation over 50 MC posterior draws. Epistemic U is computed per test image from 50 posterior draws and reported as mean ± standard deviation across 10,000 test images, in units of 10⁻³. Higher F1 and lower NLL, Brier, and AECE are preferred. Lower Epistemic U indicates reduced posterior predictive disagreement and is interpreted jointly with the other metrics. Best mean values within each drop-ratio setting are highlighted in bold. Plain BNN and Proposed were reevaluated with fresh MC draws, so their values may differ slightly from Table A4.
In addition to these experiments we also performed statistical tests to support our claims. These can be found in subsections: A1.7, A1.8, A1.9, and A1.10. We have also modified the conclusion to add future work regarding baselines, and models.
W2: We would like to clarify that our method does not claim that joint attenuation reduces the Sign Entropy of an individual weight. The intended role of Sign Entropy is to identify directionally uncertain weights whose influence on the prediction should be reduced.
To avoid any ambiguity, we have now made this property explicit with blue color in the revised manuscript in **Section 3.4, “Joint Posterior Attenuation,” page 5.
As the proposed joint attenuation scales the posterior mean and posterior standard deviation by the same positive retain factor, their ratio remains unchanged. For a Gaussian posterior, the posterior sign probability, and therefore the instantaneous Sign Entropy, depends on this ratio. Consequently, joint attenuation preserves the instantaneous Sign Entropy of the selected weight but reduces its influence.