acceptodds
Desk rejected at ICLR 2027

Stabilizing Variational Information Bottleneck Under Covariance Shift

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

In the Gaussian Information Bottleneck (IB) setting, the learned representation depends on input covariance statistics that can be distorted by finite-sample estimation error and distribution shift. We show that such perturbations can destabilize the Gaussian IB eigen-structure. Motivated by this sensitivity, we study how covariance shift distorts, and we propose two regularizers to improve robustness. Shrinkage-VIB encourages isotropic latent covariance via a Ledoit–Wolf-inspired penalty, while Sensitivity-VIB penalizes encoder Jacobian norm through a finite-difference proxy with a theory-motivated weight scaling. Experiments on MNIST, CIFAR-10, and STL-10 under two covariance-shift protocols show that Sensitivity-VIB is stronger under compound and severe shift, whereas Shrinkage-VIB provides consistent gains under milder geometric shift. Both methods substantially outperform Standard VIB and unregularized baselines.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Variational Information Bottleneck, Information Bottleneck, Covariance Shift, Gaussian Information Bottleneck

Related papers

Discussion (0)

Sign in to comment.