Tubular Low-Rank Laplace Approximations for Bayesian Neural Networks

Laplace approximations provide post-hoc uncertainty estimates, but a single local Gaussian around a maximum-a-posteriori (MAP) solution can poorly represent curved, anisotropic low-loss regions. We introduce Tubular Low-Rank Laplace (TRL), a single-checkpoint post-hoc posterior approximation that combines a discrete low-loss spine with transverse Gaussian uncertainty across the full network. Starting from one MAP checkpoint, TRL uses stochastic cross-entropy Hessian-vector products to select a low-rank positive-curvature subspace as a Fisher/GGN-motivated proxy, transports the basis along a predictor–corrector spine, and samples validation-scaled transverse perturbations around uniformly selected anchors. Predictions use the resulting nonlinear networks, avoiding full curvature matrices and repeated local decompositions. On CIFAR-100, TRL improves NLL and calibration over the evaluated last-layer and rank-matched all-weight Laplace baselines and SWAG-Diag. Across five independently trained checkpoints, mean top-label ECE is 0.013, versus 0.096 for MAP and 0.028 after temperature scaling. On ImageNet/ResNet-50, the MAP-centered TRL tube reduces ECE relative to the deterministic checkpoint in all ten evaluation splits, with small NLL and accuracy costs; temperature scaling achieves lower NLL and ECE. Ablations show that transverse uncertainty accounts for most gains around deeply trained CIFAR-100 checkpoints. In small-data fine-tuning, the spine contains functionally distinct anchors; uniform mixing increases predictive diversity while retaining mean NLL and Brier comparable to a validation-selected single anchor, without an additional anchor-selection search.