Precise Bayesian Neural Networks via Scale Invariance

Bayesian neural networks promise calibrated uncertainty, yet the posterior over their weights has been a difficult object: mean-field variational inference underfits it, it predicts well only when cooled below the Bayesian temperature, and sampling it exactly is out of reach at scale. We show that much of this difficulty comes from the parameterisation. Modern networks are normalised, so the length of a weight vector has no effect on the function: the posterior over length is exactly the prior, and every constraint the data impose falls on the direction. We therefore place the posterior on the sphere, as a distribution per unit with a single learned concentration under a uniform prior. In practice the method is adaptive noise: Gaussian noise added at each unit's pre-activation with a scale the network learns, one scalar per unit, whose KL divergence to the uniform prior has a closed form. Running Hamiltonian Monte Carlo on normalised networks, we find that this family fits: the true directional posterior is close to isotropic within a unit and independent across units, a geometry that layer normalisation and sparse activation create, and the fitted lands on it to within a few degrees, while a factorised Gaussian on the same network spends almost all of its KL on a weight length the data never determined. The learned concentration tracks the HMC posterior's as the dataset grows and follows the width scaling derived from the model; the model works at the untempered evidence lower bound, without the cold posteriors that weight-space methods typically need; and the learned uncertainty is a signal-to-noise ratio that reads as an angle, comparable across layers of any width.