- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies two problems with current Laplace approximations for uncertainty estimation. First, calibrating the prior precision on large neural networks can be time-consuming. Second, building a posterior for large neural networks can also be very expensive. To tackle these issues, the authors introduce SigmaTransfer, a method to set the prior of the large network's posterior via a width-normalized version based on a smaller (lower width) proxy network. The paper shows that the prior cannot naively be transferred to the big network because the prior kernel can blow up as width grows. With SigmaTransfer building on top of a muP-trained model, it converges to a fixed limit, so the posterior converges too. Empirically, the paper backs this by demonstrating that SigmaTransfer yields calibrated posteriors in a range of evaluation settings.
Critical strengths
-
The method is simple and well motivated, and the theory behind it is sound. SigmaTransfer combines two established methods, muP and the Laplace approximation. By only changing the prior precision, it adds no hyperparameters. The theoretical result is that the prior affects the posterior only through the prior kernel. This lays out why simple direct prior transfer fails and why muP training alone is not enough. The method is easy to adopt in existing Laplace pipelines with muP training and rests on principled theory.
-
The evaluation demonstrates that SigmaTransfer produces calibrated posteriors with only little degradation in quality compared to a prior calibrated on the target model directly.
Critical weaknesses
-
The cost-benefit seems to be narrower than it is presented. For the Sigma-Transfer to be beneficial in practice, we would need to be in the exact situation of a proxy-target model muP-trained pair of models, which limits the use case. If muP training is used, the cost-benefit of sigma transfer is proxy calibration + building the target posterior vs. target calibration. For the public 1B ->7B models, the paper sweeps by an eigendecomposition of the curvature, which then lets us evaluate candidate prior precisions cheaply. For the target model, we only have to factorize the posterior precision once, for example via a Cholesky factorization. While this is cheaper than the eigendecomposition for the sweep, this is only by a constant factor. So, as we see it, the cost saving is on the order of: eigendecomposition of the curvature of the target model against Cholesky of the target posterior precision + eigendecomposition of the curvature of the proxy. The paper, in contrast, presents the sweep cost of the proxy vs. the sweep cost of the target as the saving, which is not the practical saving. Moreover, the largest LLM speedups (260× and 330× on Yahoo and DBpedia) occur exactly where the 7B eigenproblem exceeds the GPU solver and runs on CPU. These factors therefore largely reflect a hardware fallback rather than the method itself, while the cost of the target’s own factorization is not reported.
-
The type of Laplace approximation changes a lot between experiments (last-layer full, last-layer diagonal, last-layer KFAC, all-parameter diagonal). This makes it hard to judge whether a result reflects the method or the approximation.
Questions for authors
-
We ask the authors to provide a detailed breakdown of the cost of sigma transfer, including building the target posterior vs. target-only calibration (in Table 15/3).
-
Could the authors add SP+metric in Figures 5, 6, 12 and 13, and Tables 8 and 9, as it is missing there? In Figures 10 and 11, for example, we see that SP+metric is the second-best method.
Thank you for finding σTransfer clean and well motivated, with solid theory, thorough validation and sufficient ablations. All changes are in the revised PDF.
Yes, decision transfer can hinder performance. However, decision transfer is meant for settings where the large network's Laplace posterior is computationally intractable. The large network's own uncertainty may be better, but it cannot be computed. Recall that in decision transfer, the small network's uncertainty is used to make decisions for the large network, such as which points to label or which inputs to reject. Under σTransfer, the small network's uncertainty approximates the large network's, both provably and empirically, enabling uncertainty-based decisions that would otherwise not be possible.
Our evidence for this is both theoretical and empirical. Theoretically, we prove that as both networks grow wider, their posteriors converge to the same limit, and so do the decisions based on them (Theorems 2 and 4, Appendix K). Empirically, the two networks largely agree. In active learning on MNIST, their rankings of which points to label have correlation 0.75 under σTransfer, against 0.07 under standard Laplace (Figure 3). In Figure 1(b), the width-128 and width-4096 posteriors nearly coincide under σTransfer but not under standard Laplace (KL divergence 0.21 vs. 19.53).
Regarding your point:
Such tasks are not lacking. Every decision-transfer experiment in the paper compares the small network's decisions with the large network's own (Tables 3, 11 and 13), and in each, the larger model's uncertainty is indeed better. The clearest case is abstention on language models (Appendix G, Table 13), now also in the main text (Section 6.3). A width-2048 language model must refuse prompts from other datasets. With its own posterior, with the prior precision and threshold selected on the large network itself, it rejects 98.0% of these prompts; with a width-128 network's decisions, it rejects 87.6%. In active learning, the gap is small. MNIST test accuracy is 0.926 with the large network's own choices and 0.923 with the small network's (random labels: 0.888; Table 11).
So yes, decisions transferred from the small network can be worse than the large network's own, but where the large network's posterior is intractable they are the only Laplace-based decisions available. When that posterior is affordable, transferring only the prior precision λ lets the large network use its own posterior instead, at a cost of 0.001 AUROC in MNIST out-of-distribution detection (Table 8).
Yes. σTransfer is evaluated across many Laplace approximations, and last-layer Laplace is only one of them. We test it with full-covariance Laplace over all parameters (PenDigits and Letter; Table 3, Appendix C.5), diagonal Laplace over all parameters (MNIST, width 128 to 4096, Tables 8 and 9, Figure 6), and full, diagonal and KFAC Laplace over the last layer (Tables 1, 2, 14 and 15). With all-parameter diagonal Laplace, for example, the large network reaches the same out-of-distribution detection AUROC with the small network's prior precision as with its own (0.950), while standard Laplace loses 0.072 (Table 8). Our theory covers these settings too. The guarantees hold for full-covariance Laplace over all parameters or over the last layer (Theorems 1–4, Corollary 2), and extend to diagonal and KFAC Laplace under an additional condition (Proposition 2, Appendix K.4).
Our billion-parameter experiments do use last-layer Laplace. This is because traditional full-covariance Laplace over billions of parameters is intractable (it costs O(P3) for P parameters), and even the last-layer sweep is expensive at that scale. On DBpedia, selecting λ on the public 1B model instead of the 7B model is 3.9× faster end to end, including building the 7B posterior (Table 19). We have stated this scope in Section 6.4.
So yes, σTransfer works in settings more flexible than last-layer Laplace, and at billion-parameter scale, where last-layer Laplace is the tractable choice, it still makes the precision sweep 2.7–3.9× faster end to end (Table 19).