acceptodds
Under review as a conference paper at ICLR 2027

Tubular Low-Rank Laplace Approximations for Bayesian Neural Networks

Abstract

Laplace approximations provide post-hoc uncertainty estimates, but a single local Gaussian around a maximum-a-posteriori (MAP) solution can poorly represent curved, anisotropic low-loss regions. We introduce Tubular Low-Rank Laplace (TRL), a single-checkpoint post-hoc posterior approximation that combines a discrete low-loss spine with transverse Gaussian uncertainty across the full network. Starting from one MAP checkpoint, TRL uses stochastic cross-entropy Hessian-vector products to select a low-rank positive-curvature subspace as a Fisher/GGN-motivated proxy, transports the basis along a predictor–corrector spine, and samples validation-scaled transverse perturbations around uniformly selected anchors. Predictions use the resulting nonlinear networks, avoiding full curvature matrices and repeated local decompositions. On CIFAR-100, TRL improves NLL and calibration over the evaluated last-layer and rank-matched all-weight Laplace baselines and SWAG-Diag. Across five independently trained checkpoints, mean top-label ECE is 0.013, versus 0.096 for MAP and 0.028 after temperature scaling. On ImageNet/ResNet-50, the MAP-centered TRL tube reduces ECE relative to the deterministic checkpoint in all ten evaluation splits, with small NLL and accuracy costs; temperature scaling achieves lower NLL and ECE. Ablations show that transverse uncertainty accounts for most gains around deeply trained CIFAR-100 checkpoints. In small-data fine-tuning, the spine contains functionally distinct anchors; uniform mixing increases predictive diversity while retaining mean NLL and Brier comparable to a validation-selected single anchor, without an additional anchor-selection search.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian neural networks, Laplace approximation, uncertainty quantification, posterior approximation, calibration, low-rank inference, loss landscape geometry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JLavOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Tubular Low-Rank Laplace (TRL), a method for adding uncertainty estimates to a trained neural network. Starting from the MAP solution, it constructs a path through weight space and places a low-rank Gaussian distribution around each point on that path. Predictions average over samples from these distributions. The idea is to capture variation along a low-loss path as well as variation across it, rather than using a single Gaussian centered at the fitted weights.

The paper evaluates TRL on toy problems, image classification, distribution shifts, and fine-tuning experiments. It also studies how much the path, covariance scale, and choice of subspace contribute to the results. TRL improves calibration in several settings, although its advantage in predictive likelihood is less consistent, and most practical experiments show only a small benefit from using the full path.

Critical strengths
  1. The method is applicable to large networks, and the calibration improvements are supported by several experiments.

    TRL captures uncertainty in directions involving weights throughout the network. The CIFAR-100 results across independently trained checkpoints provide useful evidence that the calibration improvements are reproducible. The paper also includes stronger practical comparisons, such as temperature scaling and deep ensembles, which help put those gains in context.

  2. The ablations provide evidence about when the path matters.

    The sine regression experiment compares a distribution centered at the fitted weights with a mixture along the path while keeping the transverse sampling setup fixed. This is a useful test of the paper's central idea, and the reported improvement supports the value of longitudinal variation in that setting. The practical experiments also show that the path often adds little, which helps identify the limits of the method.

Critical weaknesses
  1. Some approximations in the covariance and centering need a clearer justification.

    The treatment of the prior in Equation (3) is my main concern about the formulation. The method uses different prior precisions for the backbone and the head. When this prior is restricted to the selected low-rank subspace, it generally produces a small, dense precision matrix. Equation (3) keeps only its diagonal.

    This matters because the result can depend on the particular basis used to describe the subspace. For example, rotating the basis within an eigenspace with equal curvature should leave the underlying subspace unchanged, but the diagonal approximation can change the resulting distribution over weights. I would like to see either a justification for discarding these terms or a comparison using the full projected prior matrix. At the ranks used here, that matrix is small.

    There is also a question about centering the Gaussian distributions along the path. A Gaussian obtained from a local quadratic approximation is centered at the expansion point only when the relevant gradient is zero. The path corrector uses the data loss without the prior gradient, so the paper has not established that its anchors satisfy this condition for the stated posterior. The authors acknowledge that TRL is a heuristic construction, but these choices still need to be connected more clearly to its Laplace interpretation.

  2. The experiments do not yet separate the benefit of tubular geometry from the benefit of restricting and shrinking uncertainty.

    In the toy comparisons, TRL differs from ELA and LLA in several ways at once. It uses a low-rank subspace and strongly reduces the transverse covariance: the reported scales multiply covariance by (0.00520.005^2) for sine regression and (0.0520.05^2) for two moons. Consequently, better predictions could partly reflect avoiding excessive posterior spread. A comparison with similarly tuned covariance scales and matched ranks would make the contribution of the tubular construction much clearer.

    The regression evaluation adds another complication. The reported NLL scores a Gaussian fitted to the predictive mean and variance. For TRL and nonlinear ELA, the sampled predictive distribution is generally a mixture of Gaussians. Those two distributions can assign very different likelihoods to the same observation. The reported mean-plus-or-minus-standard-deviation intervals have the same limitation: they need not be quantile intervals of the actual predictive mixture. I would therefore like to see the mixture likelihood and predictive quantiles evaluated directly.

    Finally, the baseline coverage is incomplete for the broader uncertainty-estimation claims. VaLLA, ELLA, and FMGP are particularly relevant alternatives to standard LLA. SNGP and mean-field variational inference would also help position the method, although their different training requirements should be reflected in the comparison. I do not expect every method to be included, but the paper needs a stronger comparison with relevant modern alternatives, or a narrower claim about what the current results establish.

References for the methods in the last paragraph:

[1] Luis A. Ortega, Simon Rodriguez Santana, and Daniel Hernández-Lobato. Variational Linearized Laplace Approximation for Bayesian Deep Learning. ICML, 2024.

[2] Zhijie Deng, Feng Zhou, and Jun Zhu. Accelerated Linearized Laplace Approximation for Bayesian Deep Learning. NeurIPS, 2022.

[3] Luis A. Ortega, Simón Rodríguez-Santana, and Daniel Hernández-Lobato. Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning. arXiv:2412.04177, 2024.

[4] Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness. NeurIPS, 2020.

[5] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight Uncertainty in Neural Networks. ICML, 2015.

Questions for authors

These are requests for clarification or checks that could change my assessment:

  1. How do the results change when Equation (3) uses the full projected prior precision? Have you checked sensitivity to rotations within eigenspaces with equal or nearly equal eigenvalues?

  2. Which posterior is the path intended to approximate? Are the posterior gradients perpendicular to the path small enough to justify centering each Gaussian at its anchor?

  3. Do the toy conclusions hold when ELA and LLA receive comparable covariance-scale tuning and rank restrictions, and when regression is evaluated using the actual predictive mixture likelihood and quantiles?

  4. The supplementary fresh-curvature control appears to use new eigenvectors without projecting them perpendicular to the path tangent. Can you clarify whether this constraint is enforced? Otherwise, this comparison changes both the age of the curvature estimate and the sampling directions.

  5. Can you provide a comparison with an appropriate modern alternative such as VaLLA, ELLA, or FMGP, or explain why the existing baselines are sufficient for the paper’s main claims?

Reviewer_P3fNOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method for approximating the posterior distribution over neural network weights, when given a single trained model. Instead of using a Gaussian approximation centered at the MAP solution, the authors construct a path through a low-loss region of parameter space, called the spine, and sample low-rank Gaussian perturbations around points along this path. The method is evaluated on synthetic tasks, CIFAR-100, and ImageNet, with a particular focus on predictive uncertainty and calibration.

Critical strengths

I think the main strength of the paper is the idea of approximating a curved low-loss region rather than relying on a single Gaussian. This is quite intuitive, and the method remains relatively practical since it only requires one trained checkpoint and avoids repeatedly computing full Hessians.

Critical weaknesses

Adding the spines seems to barely improves performance on CIFAR-100 benchmark (Most of the improvement comes from the Gaussian perturbations around the original trained network.)

Questions for authors

In which settings do you expect the full tubular construction to outperform a simpler MAP-centered low-rank Laplace approximation? Is there a practical way to determine beforehand whether the additional complexity will lead to a substantial improvement/in which situations would you expect that?

Reviewer_HHhXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a variant of the Laplace Approximation that aims to capture the curved geometry of low-loss regions in neural network parameter space. They start from the MAP solution and identify a low-rank subspace of high-curvature directions using Hessian-vector products and Lanczos. Then they construct a path orthogonal to these directions in parameter space, and transport the subspace along this path; this is referred to as the spine. Posterior samples are generated by selecting points along the spine and adding low-rank Gaussian perturbations, followed by averaging predictions from the resulting nonlinear networks. The method is tested on CIFAR-100, against Laplace approximation variants and temperature scaling.

Critical strengths
  1. The paper is well-written and easy to follow
  2. The general idea of combining low-rank Laplace approximations with a low-loss path through parameter space makes sense.
  3. It does a good job of discussing limitations such as lack of reparameterization invariance and important experimental details such as batchnorm recalibration.
Critical weaknesses
  1. Scalability: I have some concerns about whether the proposed method is scalable to large neural networks. Computing the leading Hessian eigenvectors using Lanczos requires many Hessian-vector products, each involving expensive differentiation through the network. Storing the low-rank basis requires storing memory proportional to the number of model parameters times the retained rank. Storing separate bases along the spine further multiplies this cost. For large models, these memory requirements are prohibitive.
  2. Contribution of the spine: The ablation studies(Table 3) show that most of the improvements come from low-rank Gaussian sampling around the MAP, with the spine contributing very little to predictive performance. This raises questions about whether the central methodological novelty is necessary. The authors should address this, as this is a potentially major weakness.
  3. Low-loss paths: The spine is constructed by moving along a random direction orthogonal to the leading Hessian eigenvectors, followed by gradient corrections. Why should this procedure reliably produce a low-loss path? Although the authors empirically examine loss along the path, there is no clear theoretical justification or guarantee that the constructed trajectory remains in a low-loss region. It is unclear why the trailing eigenvalues of the Hessian give us directions of low loss path, the low loss could just be due to the gradient correction which bring us back to the MAP.
  4. Connection to the Bayesian posterior: It is unclear how the proposed approximation relates to the actual Bayesian posterior. In particular, the spine is constructed heuristically, its points receive uniform probability regardless of posterior density, and the Gaussian perturbation scale is selected using validation performance. The authors should provide stronger justification that this procedure approximates posterior mass, rather than simply constructing an ensemble of networks that improves calibration.
Questions for authors

See weaknesses (specially 2 and 3)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.