acceptodds
Under review as a conference paper at ICLR 2027

SigmaTransfer: Uncertainty Transfer from Small to Large Networks under muP

Abstract

Reliable predictive uncertainty in Laplace approximations depends critically on the prior precision, yet selecting it requires a posterior sweep that is prohibitively expensive for neural networks with billions of parameters. Under the Maximal Update Parametrization, we derive a rescaling of the prior covariance that stabilizes its induced prior kernel as model width grows. This leads to SigmaTransfer: we select the precision on a smaller model and zero-shot transfer it to the much larger model, i.e., without searching for the precision on the larger model at all. Under explicit conditions, we show convergence of the prior kernel, posterior covariance, selected precision, and posterior-derived decisions, and verify SigmaTransfer across regression, image classification, and Transformer readouts. For example, measured precision-sweep speedups reach when transferring from width 128 to 4096 on MNIST, at a target-NLL degradation of ; transferring from a public 1B to 7B model gives , with a degradation of NLL. The same posterior stability also enables transfer of acquisition, OOD-detection, and abstention decisions without constructing a target posterior.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Laplace approximation, Bayesian deep learning, uncertainty quantification, maximal update parametrization, muP, hyperparameter transfer, prior precision, width scaling, active learning, out-of-distribution detection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_J1zfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies two problems with current Laplace approximations for uncertainty estimation. First, calibrating the prior precision σ\sigma on large neural networks can be time-consuming. Second, building a posterior for large neural networks can also be very expensive. To tackle these issues, the authors introduce SigmaTransfer, a method to set the prior of the large network's posterior via a width-normalized version based on a smaller (lower width) proxy network. The paper shows that the prior cannot naively be transferred to the big network because the prior kernel can blow up as width grows. With SigmaTransfer building on top of a muP-trained model, it converges to a fixed limit, so the posterior converges too. Empirically, the paper backs this by demonstrating that SigmaTransfer yields calibrated posteriors in a range of evaluation settings.

Critical strengths
  1. The method is simple and well motivated, and the theory behind it is sound. SigmaTransfer combines two established methods, muP and the Laplace approximation. By only changing the prior precision, it adds no hyperparameters. The theoretical result is that the prior affects the posterior only through the prior kernel. This lays out why simple direct prior transfer fails and why muP training alone is not enough. The method is easy to adopt in existing Laplace pipelines with muP training and rests on principled theory.

  2. The evaluation demonstrates that SigmaTransfer produces calibrated posteriors with only little degradation in quality compared to a prior calibrated on the target model directly.

Critical weaknesses
  1. The cost-benefit seems to be narrower than it is presented. For the Sigma-Transfer to be beneficial in practice, we would need to be in the exact situation of a proxy-target model muP-trained pair of models, which limits the use case. If muP training is used, the cost-benefit of sigma transfer is proxy calibration + building the target posterior vs. target calibration. For the public 1B ->7B models, the paper sweeps by an eigendecomposition of the curvature, which then lets us evaluate candidate prior precisions cheaply. For the target model, we only have to factorize the posterior precision once, for example via a Cholesky factorization. While this is cheaper than the eigendecomposition for the sweep, this is only by a constant factor. So, as we see it, the cost saving is on the order of: eigendecomposition of the curvature of the target model against Cholesky of the target posterior precision + eigendecomposition of the curvature of the proxy. The paper, in contrast, presents the sweep cost of the proxy vs. the sweep cost of the target as the saving, which is not the practical saving. Moreover, the largest LLM speedups (260× and 330× on Yahoo and DBpedia) occur exactly where the 7B eigenproblem exceeds the GPU solver and runs on CPU. These factors therefore largely reflect a hardware fallback rather than the method itself, while the cost of the target’s own factorization is not reported.

  2. The type of Laplace approximation changes a lot between experiments (last-layer full, last-layer diagonal, last-layer KFAC, all-parameter diagonal). This makes it hard to judge whether a result reflects the method or the approximation.

Questions for authors
  1. We ask the authors to provide a detailed breakdown of the cost of sigma transfer, including building the target posterior vs. target-only calibration (in Table 15/3).

  2. Could the authors add SP+metric in Figures 5, 6, 12 and 13, and Tables 8 and 9, as it is missing there? In Figures 10 and 11, for example, we see that SP+metric is the second-best method.

Reviewer_VkTVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes σ\sigmatransfer, i.e. selecting precision on a small model and transfer it to a much larger model. σ\sigmatransfer does this by normalizing the prior geometry, scaling each layer's prior by a facor fixed by the width and Maximal Update Parametrization's coordinate scaling. This is a practical method that is supported both theoretically and empirically.

Critical strengths
  • This paper proposes a clean and well-motivated method.
  • This paper provides solid theoretical grounding, and thorough empirical validation.
  • The proposed method shows strong empirical performance, with sufficient ablation experiments are provided.
Critical weaknesses
  • W1. On the decision transfer claim and results. Width-stability is built into σ\sigmaTransfer, so the decision agreement is expected, what's lacking is a task where a (swept) larger model’s uncertainty is actually better than the independently swept small model. Does the decision transfer hinder the performance of a σ\sigmatransfer network there?
  • W2. The motivation mentions performing posterior sweeps on billion parameter networks, but it seems like empirical results are only shown on last-layer Laplace.
Questions for authors
  1. Please clarify W1.
  2. Have you tried sigma-transfer in settings more flexible than last-layer Laplace?
2 comments

Thank you for finding σ\sigmaTransfer clean and well motivated, with solid theory, thorough validation and sufficient ablations. All changes are in the revised PDF.


W1 / Q1. On the decision transfer claim and results. Width-stability is built into σ\sigmaTransfer, so the decision agreement is expected, what's lacking is a task where a (swept) larger model's uncertainty is actually better than the independently swept small model. Does the decision transfer hinder the performance of a σ\sigmatransfer network there?

Yes, decision transfer can hinder performance. However, decision transfer is meant for settings where the large network's Laplace posterior is computationally intractable. The large network's own uncertainty may be better, but it cannot be computed. Recall that in decision transfer, the small network's uncertainty is used to make decisions for the large network, such as which points to label or which inputs to reject. Under σ\sigmaTransfer, the small network's uncertainty approximates the large network's, both provably and empirically, enabling uncertainty-based decisions that would otherwise not be possible.

Our evidence for this is both theoretical and empirical. Theoretically, we prove that as both networks grow wider, their posteriors converge to the same limit, and so do the decisions based on them (Theorems 2 and 4, Appendix K). Empirically, the two networks largely agree. In active learning on MNIST, their rankings of which points to label have correlation 0.75 under σ\sigmaTransfer, against 0.07 under standard Laplace (Figure 3). In Figure 1(b), the width-128 and width-4096 posteriors nearly coincide under σ\sigmaTransfer but not under standard Laplace (KL divergence 0.21 vs. 19.53).

Regarding your point:

"what's lacking is a task where a (swept) larger model's uncertainty is actually better than the independently swept small model"

Such tasks are not lacking. Every decision-transfer experiment in the paper compares the small network's decisions with the large network's own (Tables 3, 11 and 13), and in each, the larger model's uncertainty is indeed better. The clearest case is abstention on language models (Appendix G, Table 13), now also in the main text (Section 6.3). A width-2048 language model must refuse prompts from other datasets. With its own posterior, with the prior precision and threshold selected on the large network itself, it rejects 98.0% of these prompts; with a width-128 network's decisions, it rejects 87.6%. In active learning, the gap is small. MNIST test accuracy is 0.926 with the large network's own choices and 0.923 with the small network's (random labels: 0.888; Table 11).

So yes, decisions transferred from the small network can be worse than the large network's own, but where the large network's posterior is intractable they are the only Laplace-based decisions available. When that posterior is affordable, transferring only the prior precision λ\lambda lets the large network use its own posterior instead, at a cost of 0.001 AUROC in MNIST out-of-distribution detection (Table 8).


W2 / Q2. The motivation mentions performing posterior sweeps on billion parameter networks, but it seems like empirical results are only shown on last-layer Laplace. Have you tried sigma-transfer in settings more flexible than last-layer Laplace?

Yes. σ\sigmaTransfer is evaluated across many Laplace approximations, and last-layer Laplace is only one of them. We test it with full-covariance Laplace over all parameters (PenDigits and Letter; Table 3, Appendix C.5), diagonal Laplace over all parameters (MNIST, width 128 to 4096, Tables 8 and 9, Figure 6), and full, diagonal and KFAC Laplace over the last layer (Tables 1, 2, 14 and 15). With all-parameter diagonal Laplace, for example, the large network reaches the same out-of-distribution detection AUROC with the small network's prior precision as with its own (0.950), while standard Laplace loses 0.072 (Table 8). Our theory covers these settings too. The guarantees hold for full-covariance Laplace over all parameters or over the last layer (Theorems 1–4, Corollary 2), and extend to diagonal and KFAC Laplace under an additional condition (Proposition 2, Appendix K.4).

Our billion-parameter experiments do use last-layer Laplace. This is because traditional full-covariance Laplace over billions of parameters is intractable (it costs O(P3)O(P^3) for PP parameters), and even the last-layer sweep is expensive at that scale. On DBpedia, selecting λ\lambda on the public 1B model instead of the 7B model is 3.9× faster end to end, including building the 7B posterior (Table 19). We have stated this scope in Section 6.4.

So yes, σ\sigmaTransfer works in settings more flexible than last-layer Laplace, and at billion-parameter scale, where last-layer Laplace is the tractable choice, it still makes the precision sweep 2.7–3.9× faster end to end (Table 19).

Thank you for the clarifications. I have no outstanding concerns and maintain my original positive view of the paper.

Reviewer_W1CrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the zero-shot transfer of prior precision and uncertainty-derived decisions from narrow proxy networks to wide target networks under the Maximal Update Parametrization (μP\mu\text{P}) by rescaling prior covariances to stabilize the linearized Laplace prior kernel.

Critical strengths
  1. The paper establishes a creative, principled connection between the asymptotic limits of neural network training via Tensor Programs and Bayesian uncertainty quantification in linearized Laplace approximations.
  2. The paper distinguishes two highly practical regimes of uncertainty transfer: Prior-Precision Transfer and Decision Transfer.
  3. The paper is well-written.
Critical weaknesses
  1. Corollary 5 proves the consistency of plug-in Laplace evidence-score transfer assuming the prior-weighted parameter norm qnq_n converges to a finite limit q∞<∞q_\infty < \infty. However, as noted in Appendix K.5, standard zero-mean priors across hidden layers of width nn lead to n−1∥W^n∥F2=Θ(n)n^{-1}\|\hat{W}_n\|_F^2 = \Theta(n), which makes qnq_n diverge as width grows. Because of this, the theoretical guarantee only holds for normalized last-layer readouts, rather than full deep all-parameter posteriors.

  2. In Section 6.3 (and Appendices G–I), the Transformer backbones are kept completely frozen, and the Laplace approximation is only applied to a linear classification head. On top of that, the head features are manually normalized by their mean squared norm c2c^2 before fitting. In practice, this reduces the task to Bayesian linear regression on pre-normalized features and avoids the harder problem of transferring posterior uncertainty across deep attention layers.

  3. In Appendix C.4, the simple heuristic baseline (μP+empirical\mu\text{P} + \text{empirical}), which just rescales prior variance using empirical feature norms without deriving architectural scaling factors, performs nearly the same as σ\sigmaTransfer. For instance, on MNIST LL-diagonal and Fashion-MNIST LL-full (Table 6), it matches the predictive transfer gap of σ\sigmaTransfer. It is therefore unclear how much practical gain the formal μP\mu\text{P} prior scaling adds over straightforward data-driven variance matching.

  4. For sequential active learning on Fashion-MNIST (Figures 7 and 9), proxy-guided EPIG acquisition shows a strong rank correlation (ρˉ=0.73\bar{\rho} = 0.73) with the target scores, yet it fails to beat random acquisition—all methods end up with negative ΔAULC\Delta\text{AULC}. This suggests that a high ranking correlation between proxy and target does not necessarily translate into better downstream performance, especially under cold-start or low signal-to-noise conditions.

  5. The Transformer experiments use only one pretraining seed per width, so it is hard to tell how sensitive the reported precision stability is to pretraining randomness. Also, the public uu-μP\mu\text{P} experiment transfers from a 1B model (16 layers) to a 7B model (32 layers). Mixing width scaling with depth scaling goes beyond the paper's fixed-depth theoretical assumptions.

  6. Both the abstract and Table 3 highlight speedups of up to "~5000$\times$" on MNIST. However, this number only compares the grid search time on the proxy model against the grid search time on the target model (7560 s/1.5 s7560\text{ s} / 1.5\text{ s}). For Regime A deployment, the target model still needs to compute its full posterior covariance at the transferred precision, so the actual end-to-end speedup is bounded by the 61-point grid size.

Questions for authors

see weaknesses

4 comments

Thank you for finding the link between Tensor Programs and Laplace uncertainty creative and principled, the two transfer regimes practical, and the paper well written. In the revised PDF, the abstract and Table 3 now report end-to-end speedups that include building the large network's posterior (W6).


W1. Consequently, evidence-based precision transfer is theoretically substantiated only for normalized last-layer readouts, not for deep all-parameter posteriors.

Yes, this is correct, and the paper states it in Section 5 and Appendix K.5. However, it concerns only one of the three ways we select the prior λ\lambda, and it does not affect the other two.

The first is the Laplace evidence, the case you refer to. Our theory covers it only for normalized last-layer readouts, where its condition can hold (Corollary 5), but empirically it also transfers over all parameters, with target-NLL changes of −0.007 on PenDigits and +0.078 on Letter (Table 3).

The second is validation NLL. Our theory covers this search for posteriors over all parameters (Theorems 1–3), and we test it in Table 2, so this criterion is supported both theoretically and empirically.

The third is a task metric, such as out-of-distribution detection AUROC. With all-parameter diagonal Laplace, the large network reaches the same AUROC with the small network's prior λ\lambda as with its own (0.950; Table 8).

So the limitation applies only to evidence-based selection over all parameters. For normalized last-layer readouts, evidence-based transfer is supported by our theory, and the other two ways of selecting the prior λ\lambda, validation NLL and task metrics, are unaffected. We will list a proof for the all-parameter case as future work in Appendix A.


W2. In the Transformer experiments (Section 6.3, Appendices G, H, and I), the backbone models are kept completely frozen, and Laplace approximations are applied solely to a linear classification head. Moreover, head features are manually normalized by their mean squared norm [...] prior to fitting. This simplifies the problem to Bayesian linear regression over pre-normalized representations, bypassing the core challenge of transferring posterior uncertainty over deep attention representations.

Yes, our Transformer experiments place a posterior only on the last layer, as the paper states (Appendix G, Section 6.4). However, even in this setting, σ\sigmaTransfer's width scaling of the prior is what makes the prior λ\lambda transfer across width.

With everything else identical, adding σ\sigmaTransfer's width scaling cuts the loss from transferring the prior λ\lambda twentyfold. μ\muP + isotropic and σ\sigmaTransfer use the same frozen backbones, the same normalized features and the same fitted last layers, and differ only in this scaling (Appendix G). Without it, the selected log⁡2λ\log_2\lambda drifts from 5.5 at width 128 to 10.2 at width 2048, and using the small model's prior λ\lambda raises the large model's test NLL by 0.028 on average. With it, the increase is 0.0014 (Figure 13, Table 14).

Restricting the posterior to the last layer is also standard practice for pretrained networks, including Transformers [1]. For attention weights, the core challenge you describe, the σ\sigmaTransfer prior extends using the μ\muP scalings for Transformers [2], but proving and testing transfer there remains future work (Appendix A).

So yes, our Transformer posteriors cover only the last layer, but even there the prior matters, as the controlled comparison in Figure 13 shows.

References

[1] Daxberger, Erik, et al. "Laplace Redux – Effortless Bayesian Deep Learning." Advances in Neural Information Processing Systems 34 (2021): 20089-20103.

[2] Yang, Ge, et al. "Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer." Advances in Neural Information Processing Systems 34 (2021): 17084-17097.

W3. In the extended comparisons in Appendix C.4, the heuristic control baseline [...], which simply normalizes prior variance by empirical feature norms without deriving architectural factors, performs on par with σ\sigmaTransfer. [...] questioning whether the formal [...] prior scaling provides tangible benefits over basic data-driven variance matching.

Only partly. μ\muP + empirical performs on par with σ\sigmaTransfer in 2 of the 9 settings, the two you name, and is worse in the other 7 (Tables 5 and 6). It also selects a prior λ\lambda further from the large network's own choice in all nine.

σ\sigmaTransfer is also principled where the control is a heuristic. μ\muP + empirical matches only the average size of the prior kernel across widths, which does not guarantee that the kernel converges (Appendix C.4). σ\sigmaTransfer derives its scales from the architecture and comes with a convergence guarantee (Theorem 1).

The control works only because it is built on μ\muP. The same rescaling on SP backbones (SP + empirical) is much worse in most settings (Tables 5 and 6). It is therefore a data-based estimate of the width scaling that σ\sigmaTransfer derives exactly.

So σ\sigmaTransfer is never worse than the heuristic and transfers better in 7 of 9 settings. It also sets its scales from the architecture alone, without using any data, and, unlike the heuristic, comes with a guarantee.


W4. In sequential active learning on Fashion-MNIST (Figures 7 and 9), proxy-guided EPIG acquisition achieves a high score correlation [...] but completely fails to outperform random acquisition, yielding negative Δ\DeltaAULC values across all methods. High proxy-target ranking correlation does not guarantee downstream utility when learning trajectories suffer from cold-start or low signal-to-noise dynamics.

On Fashion-MNIST, EPIG does not beat random for any method, including the large network choosing with its own posterior, and the paper states this (Figures 7 and 9, Section 6.2). However, what σ\sigmaTransfer claims, and what this experiment tests, is that the small network's uncertainty matches the large network's. Whether either beats random depends on the acquisition function, and uncertainty-based acquisition is known to struggle to beat random at small labelling budgets [3].

On this test, σ\sigmaTransfer succeeds on both datasets. The small and large networks' rankings of which points to label have correlation 0.75 on MNIST and 0.73 on Fashion-MNIST under σ\sigmaTransfer, against 0.07 and 0.36 for standard Laplace (Figures 3 and 7). On MNIST, where EPIG does beat random, the small network's choices beat random as well (+0.018 AULC, Figure 8).

So the experiment shows what our theory predicts, that the small network's uncertainty-based decisions match the large network's, and we claim downstream utility only where the acquisition function itself is useful.

References

[3] Hacohen, Guy, Avihu Dekel, and Daphna Weinshall. "Active learning on a budget: Opposite strategies suit high and low budgets." arXiv preprint arXiv:2202.02794 (2022).

W5. The Transformer experiments rely on a single pretraining seed per architecture width, leaving claims of precision stability susceptible to pretraining seed variance. Furthermore, the public [...] evaluation transfers from 1B (16 layers) to 7B (32 layers), confounding width scaling with depth scaling—a regime explicitly outside the paper's fixed-depth theoretical foundation.

The public 1B and 7B models were included to show that, in practice, adding layers does not break transfer.

Recall that we prove the prior λ\lambda, and the decisions based on it, transfer as the network gets wider, meaning each layer gets more neurons. We do not prove this when layers are added. Our own language models stay within the proof, since they all have 8 layers (Section 6.3), while the public 7B model has twice as many layers as the 1B model.

Transfer still works for this pair. On 7 of 10 tasks, the 1B model picks the same prior λ\lambda as the 7B model, and the 7B model's test NLL goes up by less than 10−410^{-4} on average (Table 15). Other work has already extended μ\muP so that training hyperparameters carry over when layers are added [4, 5]. We expect the same for the prior λ\lambda and leave the proof to future work (Appendix A).

Regarding this:

The Transformer experiments rely on a single pretraining seed per architecture width, leaving claims of precision stability susceptible to pretraining seed variance

Seed noise is unlikely to explain the result, because it holds for two independently trained families of language models. In our own models, the small model's prior λ\lambda is within 1 bit of the large model's on all ten tasks, while standard Laplace is off by 1.5 to 3.5 bits (Table 14). The public u-μ\muP models, trained by others on different data, show the same (Table 15), as do our MLP experiments with 10 seeds (Table 2). Training several seeds per size is also rarely done for language models because of the cost, and public models are rarely released with more than one, so studies such as Pythia train one model per size [6].

In short, the result does not depend on one seed, and σ\sigmaTransfer works in practice when layers are added, even though our proof only covers adding neurons.

References

[4] Bordelon, Blake, et al. "Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit." International Conference on Learning Representations. Vol. 2024. 2024.

[5] Dey, Nolan, et al. "Don't be lazy: CompleteP enables compute-efficient deep transformers." Advances in Neural Information Processing Systems 38 (2026): 137707-137739.

[6] Biderman, Stella, et al. "Pythia: A suite for analyzing large language models across training and scaling." International conference on machine learning. PMLR, 2023.


W6. The abstract and Table 3 emphasize speedups reaching "~5000×" on MNIST, but this ratio compares only proxy-side grid search time against target-side grid search time [...]. In Regime A deployment, the target model must still construct its full posterior covariance at the transferred precision, capping the true end-to-end acceleration by the 61-point grid size.

We have added exactly this comparison to the revised paper (Table 3, and in full in Appendix L, Table 19) and reproduce it below. The column "Proxy search + target posterior" gives the end-to-end speedup you describe, and σ\sigmaTransfer is faster in all eight settings (1.9–108.6×). We have also replaced the search-only ratios in the abstract, Sections 6.2 and 6.4, Appendix C.5 and the conclusion with these end-to-end figures. The former ~5000× figure, for MNIST with last-layer full Laplace, becomes 49.9× end to end.

Speedup over selecting the prior λ\lambda on the target alone (×), from Table 19(b). Each column divides the target-only time by the time of the steps named in its header.

SettingApproximationProxy search onlyProxy search + target posteriorProxy training + proxy search + target posterior
MNIST, 128 → 4096Last-layer full5,04049.945.4
MNIST, 128 → 4096All-parameter diagonal152.0108.620.5
Our LLMs, 128 → 2048Last-layer full7.11.9N/A
AG News, 1B → 7BLast-layer full7.52.7N/A
Yahoo, 1B → 7BLast-layer full7.53.6N/A
DBpedia, 1B → 7BLast-layer full7.73.9N/A
PenDigits, 32 → 128All-parameter full22.04.11.5
Letter, 32 → 128All-parameter full32.14.31.7

We hope these answers address your soundness concerns, and we are happy to clarify any point further.

The authors' response seems to resolve most of my concerns, so I decide to keep the original score and vote for weak acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.