Separating Posterior Geometry from Inference Mechanisms: A Unified Cayley-Orbit Posterior for Bayesian Neural Networks

Bayesian neural networks (BNNs) quantify epistemic uncertainty through posterior distributions over weights. Usually, weight posteriors in BNNs primarily control the scale and statistical dependence of perturbations in an unconstrained weight space, providing a tractable representation of weight uncertainty. However, such posterior parameterizations can suffer from restricted directional exploration and unstable weight structure during sampling. Moreover, posterior construction is often embedded within a particular inference mechanism, making useful posterior geometries difficult to transfer across BNN methods. To address these limitations, we introduce Unified Cayley-Orbit Posterior (UniCOP), a unified posterior constructor separated from the posterior inference mechanism that used to learn the posterior parameters. Our UniCOP constructs a posterior on a curved orbit by applying Cayley transform to a learned reference weight, preserving the reference weight’s structure. This geometry-aware construction better models the epistemic uncertainty through directional exploration while keeping structural properties, supporting predictive stability and robustness. Moreover, our constructor can therefore be employed with both explicit variational and optimizer-induced inference mechanisms. Our theoretical analysis characterizes how the Cayley-orbit construction preserves weight-matrix structure and establishes the geometric validity and versatility of UniCOP. Extensive experiments across multiple BNN methods also demonstrate consistent improvements of UniCOP on uncertainty estimation, robustness to input perturbations, and predictive generalization.