- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose Tubular Low-Rank Laplace (TRL), a method for adding uncertainty estimates to a trained neural network. Starting from the MAP solution, it constructs a path through weight space and places a low-rank Gaussian distribution around each point on that path. Predictions average over samples from these distributions. The idea is to capture variation along a low-loss path as well as variation across it, rather than using a single Gaussian centered at the fitted weights.
The paper evaluates TRL on toy problems, image classification, distribution shifts, and fine-tuning experiments. It also studies how much the path, covariance scale, and choice of subspace contribute to the results. TRL improves calibration in several settings, although its advantage in predictive likelihood is less consistent, and most practical experiments show only a small benefit from using the full path.
Critical strengths
-
The method is applicable to large networks, and the calibration improvements are supported by several experiments.
TRL captures uncertainty in directions involving weights throughout the network. The CIFAR-100 results across independently trained checkpoints provide useful evidence that the calibration improvements are reproducible. The paper also includes stronger practical comparisons, such as temperature scaling and deep ensembles, which help put those gains in context.
-
The ablations provide evidence about when the path matters.
The sine regression experiment compares a distribution centered at the fitted weights with a mixture along the path while keeping the transverse sampling setup fixed. This is a useful test of the paper's central idea, and the reported improvement supports the value of longitudinal variation in that setting. The practical experiments also show that the path often adds little, which helps identify the limits of the method.
Critical weaknesses
-
Some approximations in the covariance and centering need a clearer justification.
The treatment of the prior in Equation (3) is my main concern about the formulation. The method uses different prior precisions for the backbone and the head. When this prior is restricted to the selected low-rank subspace, it generally produces a small, dense precision matrix. Equation (3) keeps only its diagonal.
This matters because the result can depend on the particular basis used to describe the subspace. For example, rotating the basis within an eigenspace with equal curvature should leave the underlying subspace unchanged, but the diagonal approximation can change the resulting distribution over weights. I would like to see either a justification for discarding these terms or a comparison using the full projected prior matrix. At the ranks used here, that matrix is small.
There is also a question about centering the Gaussian distributions along the path. A Gaussian obtained from a local quadratic approximation is centered at the expansion point only when the relevant gradient is zero. The path corrector uses the data loss without the prior gradient, so the paper has not established that its anchors satisfy this condition for the stated posterior. The authors acknowledge that TRL is a heuristic construction, but these choices still need to be connected more clearly to its Laplace interpretation.
-
The experiments do not yet separate the benefit of tubular geometry from the benefit of restricting and shrinking uncertainty.
In the toy comparisons, TRL differs from ELA and LLA in several ways at once. It uses a low-rank subspace and strongly reduces the transverse covariance: the reported scales multiply covariance by () for sine regression and () for two moons. Consequently, better predictions could partly reflect avoiding excessive posterior spread. A comparison with similarly tuned covariance scales and matched ranks would make the contribution of the tubular construction much clearer.
The regression evaluation adds another complication. The reported NLL scores a Gaussian fitted to the predictive mean and variance. For TRL and nonlinear ELA, the sampled predictive distribution is generally a mixture of Gaussians. Those two distributions can assign very different likelihoods to the same observation. The reported mean-plus-or-minus-standard-deviation intervals have the same limitation: they need not be quantile intervals of the actual predictive mixture. I would therefore like to see the mixture likelihood and predictive quantiles evaluated directly.
Finally, the baseline coverage is incomplete for the broader uncertainty-estimation claims. VaLLA, ELLA, and FMGP are particularly relevant alternatives to standard LLA. SNGP and mean-field variational inference would also help position the method, although their different training requirements should be reflected in the comparison. I do not expect every method to be included, but the paper needs a stronger comparison with relevant modern alternatives, or a narrower claim about what the current results establish.
References for the methods in the last paragraph:
[1] Luis A. Ortega, Simon Rodriguez Santana, and Daniel Hernández-Lobato. Variational Linearized Laplace Approximation for Bayesian Deep Learning. ICML, 2024.
[2] Zhijie Deng, Feng Zhou, and Jun Zhu. Accelerated Linearized Laplace Approximation for Bayesian Deep Learning. NeurIPS, 2022.
[3] Luis A. Ortega, Simón Rodríguez-Santana, and Daniel Hernández-Lobato. Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning. arXiv:2412.04177, 2024.
[4] Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness. NeurIPS, 2020.
[5] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight Uncertainty in Neural Networks. ICML, 2015.
Questions for authors
These are requests for clarification or checks that could change my assessment:
-
How do the results change when Equation (3) uses the full projected prior precision? Have you checked sensitivity to rotations within eigenspaces with equal or nearly equal eigenvalues?
-
Which posterior is the path intended to approximate? Are the posterior gradients perpendicular to the path small enough to justify centering each Gaussian at its anchor?
-
Do the toy conclusions hold when ELA and LLA receive comparable covariance-scale tuning and rank restrictions, and when regression is evaluated using the actual predictive mixture likelihood and quantiles?
-
The supplementary fresh-curvature control appears to use new eigenvectors without projecting them perpendicular to the path tangent. Can you clarify whether this constraint is enforced? Otherwise, this comparison changes both the age of the curvature estimate and the sampling directions.
-
Can you provide a comparison with an appropriate modern alternative such as VaLLA, ELLA, or FMGP, or explain why the existing baselines are sufficient for the paper’s main claims?