Stabilizing Variational Information Bottleneck Under Covariance Shift

In the Gaussian Information Bottleneck (IB) setting, the learned representation depends on input covariance statistics that can be distorted by finite-sample estimation error and distribution shift. We show that such perturbations can destabilize the Gaussian IB eigen-structure. Motivated by this sensitivity, we study how covariance shift distorts, and we propose two regularizers to improve robustness. Shrinkage-VIB encourages isotropic latent covariance via a Ledoit–Wolf-inspired penalty, while Sensitivity-VIB penalizes encoder Jacobian norm through a finite-difference proxy with a theory-motivated weight scaling. Experiments on MNIST, CIFAR-10, and STL-10 under two covariance-shift protocols show that Sensitivity-VIB is stronger under compound and severe shift, whereas Shrinkage-VIB provides consistent gains under milder geometric shift. Both methods substantially outperform Standard VIB and unregularized baselines.