Attenuate, Don’t Drop: A Sign Entropy Approach to Weight Regularization

Modern neural networks often achieve high accuracy while producing poorly calibrated confidence estimates. Regularization techniques like Dropout and DropConnect improve generalization by randomly masking activations or weights. However, they do not address calibration error. To address this, we propose a stability-guided posterior attenuation mechanism for Bayesian Neural Networks (BNNs). We use Sign Entropy, an information-theoretic quantity, to identify unstable weights and perform annealed attenuation on them. Sign Entropy measures the posterior uncertainty of the sign of each Bayesian weight, i.e., whether the weight provides positive or negative evidence. During training, we softly attenuate high Sign Entropy weights by jointly scaling their posterior mean and standard deviation. We provide theoretical justification by showing that, in the single-feature binary classification case, Sign Entropy is directly related to the posterior entropy of the predicted class, and we extend it to multi-feature settings. We demonstrate our results on classification and regression tasks, using two well-known Deep Learning (DL) backbones on two datasets, and show that the proposed method generally preserves predictive accuracy while improving probabilistic fit, calibration error, epistemic uncertainty, and predictive interval quality.