acceptodds
Under review as a conference paper at ICLR 2027

Precise Bayesian Neural Networks via Scale Invariance

Abstract

Bayesian neural networks promise calibrated uncertainty, yet the posterior over their weights has been a difficult object: mean-field variational inference underfits it, it predicts well only when cooled below the Bayesian temperature, and sampling it exactly is out of reach at scale. We show that much of this difficulty comes from the parameterisation. Modern networks are normalised, so the length of a weight vector has no effect on the function: the posterior over length is exactly the prior, and every constraint the data impose falls on the direction. We therefore place the posterior on the sphere, as a distribution per unit with a single learned concentration under a uniform prior. In practice the method is adaptive noise: Gaussian noise added at each unit's pre-activation with a scale the network learns, one scalar per unit, whose KL divergence to the uniform prior has a closed form. Running Hamiltonian Monte Carlo on normalised networks, we find that this family fits: the true directional posterior is close to isotropic within a unit and independent across units, a geometry that layer normalisation and sparse activation create, and the fitted lands on it to within a few degrees, while a factorised Gaussian on the same network spends almost all of its KL on a weight length the data never determined. The learned concentration tracks the HMC posterior's as the dataset grows and follows the width scaling derived from the model; the model works at the untempered evidence lower bound, without the cold posteriors that weight-space methods typically need; and the learned uncertainty is a signal-to-noise ratio that reads as an angle, comparable across layers of any width.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian neural networks, variational inference, uncertainty quantification, normalization, von Mises-Fisher

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uHH1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes the concentration-adapted perturbation (CAP), a variational method that models weight directions with non Mises-Fisher distributions. It implements through learned Gaussian pre-activation noise and an approximate KL penalty. The authors compare posterior concentration with HMC and evaluate calibration and scaling behavior on synthetic and Fahsinon-MNIST datasets.

Critical strengths
  • The paper is well motivated and well-written with excellent illustrations and organization.
  • The HMC comparisons well evaluate the posterior approximation. The controlled normalization and sparsity experiments also help to identify when within-unit isotropy and independence across units are reasonable.
  • The method requires only one concentration parameter per unit. The width-scaling experiment provide evidence that this parameter adapts systematically to changes in network size.
Critical weaknesses
  • Section 2 treats layer normalization as making each unit independently scale invariant. However, standard layer normalization does not generally satisfy this assumption as scaling one pre-activation changes its value relative to the others and changes their normalization. Therefore, Proposition 1 does not justify independent radial factorization for every normalization method listed.
  • The variance approximation underlying Equation 2 does not generally have the claimed high-concentraion behavior. For a unit input aligned with the mean direction, x=μx=\mu the vMF variance is asymptotically ((D−1)/(2κ2))((D-1)/(2\kappa^2)) at fixed dimension. The proposed approximation behaves as (1/κ)(1 / \kappa). Both vanish, but their leading orders are different. This affects the interpretation of the implemented noise as directional posterior uncertainty.
Questions for authors
  • Which tested layers satisfy independent per-unit scale invariance?
  • Under what assumptions on dimension and input direction is the variance approximation accurate? Does correcting its directional dependence change the posterior and calibration results?
1 comment

Thank you for the review and for two technical points that improved the paper. We are glad that you found the paper well motivated, and that the HMC comparisons and the controlled experiments on normalisation and sparsity were useful.

"…standard layer normalization does not generally satisfy this assumption…" Indeed, layer normalisation does not normalise each unit, as weight and batch normalisation do, and we have made this precise in the paper (Section 2 and Appendix A). Our results are unaffected, because none of our networks relies on layer normalisation for the invariance.

"Which tested layers satisfy independent per-unit scale invariance?" All of them. We weight-normalise every trained layer with a fixed gain and follow it with batch normalisation without affine parameters (Appendix K), which gives the invariance for each unit. In the teacher-student networks the weight vectors have unit norm, so each unit is also normalised. We also apply layer normalisation to the pre-activations of some of these networks, to measure its effect on the posterior (Section 4.1, Appendix I).

"For a unit input aligned with the mean direction … their leading orders are different." Our closed form ‖x‖²/(κ + D) assumes (µᵀx)² ≈ ‖x‖²/D, an input with no preferred alignment to the mean direction, which is the typical case in high dimension. Appendix A of the original manuscript stated this assumption, and we now state it in the main text as well (Section 3). For an input aligned with µ the leading order is different, and we derive it in Appendix B ("For an input along µ the variance is instead C∥‖x‖², which for large κ is ‖x‖²(D − 1)/(2κ²)"), in agreement with your expression.

"Under what assumptions on dimension and input direction is the variance approximation accurate?" It is accurate for inputs whose normalised pre-activation z is of order one, and the more so the larger the fan-in D. The exact variance is C∥(µᵀx)² + C⊥(‖x‖² − (µᵀx)²) and (µᵀx)² = z²‖x‖²/D, so the variance differs from the unaligned case (z² = 1) by a relative amount of at most |z² − 1|/D. This is below 4% for |z| ≤ 3 at fan-ins of 256 and above. The aligned input corresponds to z² = D. In the unaligned case the closed form is within 20% of the exact variance at every κ (4% at κ/D = 10).

"Does correcting its directional dependence change the posterior and calibration results?" We tested this by retraining with the exact dependence, in two ways. We first replaced the Gaussian noise by a vMF sampler of the weights, which has the full dependence on the direction of the input, and the ECE stays at 3.1% (Table 6). We then kept the Gaussian noise and used the exact variance in place of the closed form, and the learned noise scale is 2% smaller while the accuracy, ECE and NLL change by at most one standard error (Table 3). The results therefore do not change.

Reviewer_H3hJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a family of approximate posteriors for performing inference in neural network architectures whose weights exhibit scale invariance. They constrain the network weights to lie on a hyperspherical shell, and the variational family is a von Mises-Fisher distribution. There is a mean direction parameter for each weight, and a concentration (spread) parameter shared across all weights corresponding to the same neuron. This is a reduction in the number of variational parameters relative to existing approaches for hyperspherical variational inference, and the authors also devise a way to compute the ELBO without the need to evaluate Bessel functions, which is also novel. They evaluate their approximate posterior against similar baselines and full-batch HMC for small network architectures on toy, Fashion-MNIST, CIFAR-10, and UCI regression settings.

Critical strengths

S1 The figures, particularly 1 and 2, are very information-dense and useful for visualising the concepts covered in the paper. They have great potential (they are currently far too small, see below weaknesses).

S2 The authors make a valiant effort to evaluate the quality of their approximate posterior family against the "true" posterior (samples from full-batch HMC). This is relatively rare in BNN papers, but it is a good practice.

Critical weaknesses

W1: Writing Clarity. By far the most serious weakness of the paper is that it is barely intelligible. The authors need to take improving their paper's clarity with absolute seriousness. Unfortunately, I do not see a way to sufficiently improve the paper's clarity without significant refactors being performed. My recommendation would be for the authors to adopt a more traditional paper structure that involves an introduction, background, method, related work, experiments, and discussion sections, and to rewrite most parts of the paper from scratch and with less LLM usage.

For the AC's benefit, the following is an excerpt from the manuscript that is representative of the paper's writing clarity: "The KL divergence to the uniform prior has a closed-form approximation in the standard deviation σ\sigma of additive Gaussian noise at the unit’s pre-activation, with no Bessel functions, so the layer is adaptive noise: Gaussian noise at each pre-activation with a scale the network learns, and a KL term that sets it. We call it a concentration-adapted perturbation (CAP).".

For the authors' benefit, here are some general-purpose pieces of advice that I believe their re-write would benefit from:

  1. Use fewer colons. In many cases they can be replaced with em dashes, the phrase "such as", or something else to keep the sentence structure diversity high. No abstract needs three colons, and no page of well

  2. All variables, acronyms, and novel technical terms need to be defined somewhere. This paper has an extremely high incidence of terms that are used but never introduced, which leaves readers guessing.

  3. The manuscript should not be anthropomorphised with phrases such as "Section 3 tests [a hypothesis]". It is the authors who do the testing, not a section of a manuscript. A rephrasing of this particular example that would be more appropriate for scientific writing would be something like "In Section 3 we test [a hypothesis]".

  4. The authors should be more explicit about when they are discussing the directional component of a BNN's true posterior, and when they are discussing their von Mises-Fisher approximate posterior for BNNs. For what I consider to be a perfect example of separating discussion of the true BNN posterior and of an approximate posterior inspired by the true one, see Section 2 of this lovely paper

  5. Every plot needs to be made larger. Plot font sizes should be similar to that of the main text, and readers should not need to zoom in on a plot to be able to see the details of it. The figures are one of the strengths of this paper (particularly Figures 1 and 2), so the authors should display them in a size they deserve.

  6. More time should be spent describing the student-teacher setup used in the experiments. Currently, the presence of student and teacher networks is referred to but never explained to readers.

  7. The authors should be more explicit about related prior work on the topic of approximate inference for hyperspherical weights.

W2: Propositions. Proposition 1 is crippled by a missing variable definition$\textemdash$what is θ\theta? I am guessing it represents the weights' direction vector w∣w∣\frac{w}{\\|w\\|}, but the authors should write this explicitly, or indeed whatever else it represents if my guess is incorrect. The proofs for both Proposition 1 and Proposition 2 are lacking in rigour (and the proof for Proposition 1 doesn't even exist at all). Both proofs should be provided in the appendix and should consist of detailed derivation with easy-to-follow algebraic manipulation and the occasional supporting statement for a particular line. The current approach of verbal exposition is insufficient.

W3: Lack of Error Bars. None of the reported results come with any form of reported standard error, so it is impossible for readers to assess the statistical significance of any results.

W4: Limited Novelty. In light of Farquhar et al. (2020) "Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning", Oh et al. (2020) "Radial and Directional Posteriors for Bayesian Neural Networks", and Goshal and Tucker (2021) "Hyperspherical Weight Uncertainty in Neural Networks" this paper's contributions are limited; as far as I can tell, their formulation allows for a reduced number of variational parameters, and the avoidance of computing Bessel functions. These improvements over the baselines should be emphasised as the core contribution. Compared with the idea of approximate inference over weights restricted to sit on a hyperspherical shell, which is what the paper's lack of clarity implies the contribution is, the actual contributions are much less significant.

W5: Title. The title is unrelated to the method. Why should this family of approximate posterior be referred to as a "precise" one? It is far from the first approximate posterior to use precision parameters, and in the paper itself these parameters are referred to as "concentration" parameters rather than "precision" ones. While a bad title is not such a major issue in itself, it points towards the larger issue with this paper, which is that it is incoherent (possibly due to excessive LLM use).

Questions for authors

Since there is no other suitable field for me to insert the following, I use this box as a general purpose "further review comments" box. The following suggestions are intended to assist the authors when conducting their re-write. The AC does not need to read this section.

Minor Corrections / Editorial Suggestions

C1 On lines 46/47, "or a variational learning only recently made to train them" does not make sense. Changing the phrasing to something like "or bespoke variational learning strategies" would be more readable.

C2 On line 49, "We start from..." is an odd way to begin this paragraph because the authors have yet to specify what their goal is. If they wish to keep things general, this phrase would be improved by writing "We start by investigating...". Otherwise, the authors should be explicit about their goal of designing a new approximate posterior, which then motivates "starting" somewhere in the eyes of the reader. An easy way to achieve this would be "Motivated by the need to design ever better approximate posteriors for BNNs, we start by investigating...".

C3 On line 56, "Section 4 shows what it does to the fit" reads ambiguously; Section 4 of the current work or of Farquhar et al. (2024)? I would alter this sentence to something like "and we demonstrate the negative impact this has on approximate posterior quality in Section 4". "Fit" is also a rather ambiguous term in the context of BNNs; do the authors mean fit-to-data (e.g. expected log-likelihood term in VI, or predictive mean RMSE etc) or degree to which the approximate posterior matches the true posterior?

C4 On line 67, "is the uniform prior" would read better as "corresponds to the uniform prior".

C5 On lines 81 and 82, the sentence starting with "The network's output..." reads poorly. There is no need to compare the case for weights ww with weights cwcw for any c>0c>0 since the former is already a special case of the latter where c=1c=1. It would be simpler to write something like "Since the network output is then invariant to a scaling of the weights by multiplicative scaling factor c>0c>0, the likelihood also exhibits scale invariance such that ddcp(D∣cw)=0\frac{\mathrm{d}}{\mathrm{d}c}p(\mathcal{D}\mathrel{|}cw) = 0".

C6 On line 95, the authors write "almost all of the Gaussian’s KL is spent on the radius". What does it mean to "spend" KL divergence? This language needs to be formalised.

C7 On line 102, "the network's weight" should be removed. Also, the authors should clarify what they mean by "per unit or per layer"; what determines which is the case? Furthermore, the authors never introduce the meaning of CD(κ)C_D(\kappa); line 102 would be the perfect place to do that.

C8 On line 104, what does I⋅(κ)I_{\cdot}(\kappa) represent? The authors should introduce this notation before using it.

C9 On line 105, "in both limits" needs to be clarified; in both limits of what?

C10 On line 105, the authors need to define σu\sigma_u before using it.

C11 On lines 105/106, the authors introduce σeff\sigma_\text{eff} as the "noise relative to the signal". Why don't they refer to it as the noise-to-signal ratio (NSR), the coefficient of variation, or any other common term?

C12 On lines 106 and 136, the authors write "the layer computes [an equation] and learns log⁡σeff\log \sigma_\text{eff}. How does the layer itself learn the noise parameter? If the authors instead mean that the noise parameter is learned/optimised during training, they should write something to that effect instead.

C13 On line 137, I suggest the authors swap out "absorbs" in favour of "eliminates the effect of".

C14 The acronym "VD" is never introduced, so readers have no way to understand what the authors mean by it. This should be fixed.

C15 In Figure 1's caption, what do the authors mean CAP's posterior is a cap on the unit circle? The authors should remove that ambiguous phrase in favour of something more precise, and hopefully something that makes it clearer to readers that the red density curve in the leftmost plot is intended to come out-of-the-page rather than veering off the unit circle within the plane of the page (which is how I first interpreted it and I expect other readers might too). Also, the unconventional phrase "spends mass" should be swapped out in favour of "places mass".

C16 In Figure 2's caption and line 139, the authors refer to "the family", but they never describe what family they mean. If they mean something like the von Mises-Fisher family of approximate posteriors for neuron weights, they should write that somewhere.

C17 Everything in Figure 2 is far too small. Axis labels, legend labels, tick labels, and so on, are all practically illegible. The authors should fix this.

C18 On line 141, "The dictionary between noise and angle" is a meaningless phrase. Perhaps the authors mean something like "The relationship between noise and angle"? The rest of this paragraph is also difficult to understand; the authors should work to improve the clarity.

C19 The authors need to explain where equation 3 comes from. Have they derived it or adopted it from prior work?

C20 On line 158, the authors need to formalise their definition and exposition of the objective function. They never introduce the acronym NLL (even though many readers might guess it refers to negative log-likelihood). The authors should justify where this objective function comes from, as it appears similar but different to the tempered ELBO (which would have an expected log-likelihood term instead of an NLL term). Furthermore, while readers might guess that the KL terms are additive across layers, and therefore that the subscript ⋅l\cdot_l indexes network layers, the authors need to mention this somewhere explicitly!

C21 The first sentence of section 3 is clunky and difficult to understand. The authors should explain what is meant by the "tangent covariance of a von Mises-Fisher". Furthermore, "; across units qq factorises" should be replaced with something like ", and the posterior qq factorises between units" (to make it clear that this is a separate point to the within-unit covariance structure".

C22 The second sentence of section 3 is also both clunky and difficult to understand. Which two assumptions are the authors referring to? What is meant by a "scalable" BNN? What is meant by "the posterior is too hard to sample to test them"\textemdash\textemdashI am not aware of any BNN architecture whose posterior can be sampled, except for a single hidden-layer architecture?

C23 The third sentence of Section 3 is difficult to understand. What do the authors mean by "On the directions of a small normalised network HMC converges"?

C24 In the paragraph below Proposition 2, what do the authors mean that the inequality in Proposition 2 "is" Cauchy-Schwarz? On lines 190-191, what "predictions" are the authors referring to? What do they mean by an "erf teacher" and "ReLU teacher"? What do they mean by "The error of a single concentration per unit is thus a known function of the posterior’s shape"? This whole paragraph needs to be re-written in clearer terms.

C25 On line 198, the authors need to introduce the acronyms AM and HM somewhere before using them.

C26 On line 199, the authors should replace "crossed" with something like "investigated".

C27 On line 203, it makes no sense to "affect different assumptions". Something can affect the validity of an assumption, but not an assumption itself.

C28 The plots in Figure 3 need to be larger. Also, rr needs to be defined before it is used.

C29 On lines 231-232, the authors claim that normalised ReLU networks develop sparse activations as they train (in general). The authors need to support this with suitable a argumentation, demonstration, or citation.

C30 Section 4 is more clear than Section 3, but there is still insufficient explanation of the student-teacher setup. This should be addressed.

C31 Figure 5's results should be displayed in some kind of figure (table or plot). The current method of reporting adds to the paper's immense lack of clarity.

C32 On line 323, the authors state the AM/HM ratio vanishes. By this, do they mean it converges to 1.0? If so, they should make that clear.

C33 On line 346, what does an "arm" refer to? The authors should clarify this.

2 comments
AuthorsOpenReview ↗

Response 1/2

Thank you for the detailed review and for the many concrete suggestions on the writing, which guided our revision. We are glad that you value the figures (S1) and the evaluation against the true posterior (S2).

W1. "…rewrite most parts of the paper…" We rewrote the paper following this recommendation, with the structure you propose. Following your seven points, we removed the colons that introduced clauses, defined the variables and acronyms at first use and made the authors the subject of our sentences. We also separated the true posterior from the approximation (Section 4), enlarged the figures and their fonts, introduced the teacher-student setup before using it (Section 4.1, Appendix I) and added the prior work on hyperspherical weights to Section 5 (see W4).

W2. "Both proofs should be provided in the appendix." We added a proof of Proposition 1 (Appendix A) and rewrote the proof of Proposition 2 as a step-by-step derivation from three stated assumptions (Appendix E). You read θ correctly as the direction w/‖w‖, which the proposition now defines explicitly as ŵ, together with the length r = ‖w‖. Because the proof of Proposition 2 rests on a local approximation of the posterior, we also test the proposition against HMC on single units, and the measured overconfidence agrees with the predicted one (Table 7).

W3. We now report the mean and standard error over four seeds in Table 2 and in the appendix tables, and we give the fitted exponents with their errors (0.476 ± 0.006 for the width law). We also give the convergence diagnostics of the HMC chains of Table 1 in Appendix K.

W4. "…this paper's contributions are limited; as far as I can tell, their formulation allows for a reduced number of variational parameters, and the avoidance of computing Bessel functions." Indeed, these three papers already place a posterior on the sphere. We have rewritten the contributions more clearly in the revised Introduction. Our contributions with respect to these papers are the following. First, we have no weight magnitude to model. Farquhar et al. model it with a radial distribution and Oh et al. with a posterior over the norms. Second, we address how close the approximation is to the true posterior. The three papers propose a family for a given network and evaluate it by predictive performance, whereas we design the network, with normalisation and sparse activation, so that its true posterior is close to our family, and we verify this against HMC. Third, our model is best calibrated without tempering, whereas in our experiments the radial posterior of Farquhar et al. is the best method when tempered but loses its calibration at β = 1 (Table 2). Fourth, the three papers sample the weights, whereas we express the posterior as Gaussian noise on the units, leading to a practical adaptive noise method. Fifth, our match to a simple true posterior allows for the derivation and testing of how the learned noise scales with width, fan-in and data (Section 4.3). We compare our method with the three papers in Section 5.

W5. We have changed the title to Untempered Bayesian Neural Networks via Scale Invariance. The word "Precise" was meant for matching the posterior with precision, and it could indeed be confused with a precision variable in this context. "Untempered" encapsulates the paper better. Because we design the model so that its approximation matches the true posterior, it can be trained on the Bayesian objective without tempering.

AuthorsOpenReview ↗

Response 2/2

C1 to C33. We fixed each of these points in the revised manuscript. Several of them ask a question, which we answer here.

C19. "The authors need to explain where equation 3 comes from. Have they derived it or adopted it from prior work?" We derived this closed-form KL ourselves, as an interpolation between the two limits of the exact KL (Appendix B of the original manuscript). The main text now says so ("we construct a simpler closed-form approximation", Section 3), and Appendix C gives the derivation in full.

C20. "The authors should justify where this objective function comes from, as it appears similar but different to the tempered ELBO…" It is the negative tempered ELBO, up to our two approximations. NLL is a single-sample estimate of the expected log-likelihood, as usual in stochastic variational inference, which is why it appears in place of the expectation. We now define each term in Section 3, including the sum of the KL over layers.

C22 and C23. Full-batch HMC has been run on deep networks, including ResNets on CIFAR-10 (Izmailov et al., 2021), but at a cost that rules it out as a routine check. We sample the full posterior only for the small teacher–student networks. On Fashion-MNIST we sample the directions of one stochastic layer with the rest of the network fixed, starting from the trained weights (Section 4.2, Appendix K).

C29. "…the authors claim that normalised ReLU networks develop sparse activations as they train (in general). The authors need to support this with suitable a argumentation, demonstration, or citation." Li et al. (2023) show that sparse activations emerge during the training of ReLU networks, in transformers and in MLPs, and we now cite them. In our own networks we control the sparsity of the units by adjusting the bias.

Reviewer_k2L5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposea a new variational inference method for Bayesian neural networks based on scale invariance. The authors argue that removing redundant radial degrees of freedom simplifies posterior inference and propose a factorized von Mises–Fisher approximation over weight directions, implemented through adaptive Gaussian preactivation noise.

Critical strengths

The geometric motivation is very interesting, and exploiting scale invariance to eliminate redundant parameters is a promising approach.

The empirical results suggest that their method can improve uncertainty calibration compared with several existing variational methods.

Critical weaknesses
  1. Misleading claims regarding posterior tractability.

My main issue with the paper is that the abstract and the intro overclaim the actual results. The prose in the abstract for example seem to suggest that their scale-invariance-based method has lifted an important barrier for the tractability of the posterior of Bayesian neural nets, a longstanding challenge of BNN inference. It turns out that the contribution is much narrower than advertised.

Proposition 1 merely establishes that, under exact scale invariance and a factorized prior, the radial posterior equals its prior. While mathematically correct, this is an elementary consequence of Bayes' rule and does not address the fundamental intractability of the remaining directional posterior. In particular, the directional posterior remains generally nonlinear, potentially multimodal, and correlated across units. Neither its normalizing constant nor exact posterior expectations become tractable through the proposed reparameterization.

The authors subsequently introduce a factorized von Mises–Fisher approximation, but its suitability does not follow from scale invariance.

Although the authors do not explicitly claim to solve exact posterior inference, the abstract and discussion strongly suggest that removing radial redundancy resolves a substantial part of the general inference problem. I find this framing misleading and insufficiently supported by the theoretical results. The paper establishes a useful reduction in parameter redundancy, not a general resolution of posterior intractability, and should frame the contribution as such. The authors should tone down their prose, and clearly articulate the extent of the contribution.

  1. Insufficient theoretical justification of proposed approximation.

The practical implementation does not sample exactly from the proposed von Mises–Fisher posterior. Instead, it approximates preactivations using Gaussian noise and employs an approximate KL divergence.

The paper does not establish a rigorous connection between the resulting training objective and the exact ELBO of the stated spherical Bayesian model, nor does it provide theoretical guarantees controlling the combined approximation error.

Consequently, it remains unclear to what extent the proposed method should be interpreted as a faithful Bayesian posterior approximation.

Questions for authors

What theoretical or empirical evidence supports the claim that parameterization accounts for "much of the difficulty of Bayesian neural-network inference"?

Can the authors establish a precise relationship between the Gaussian preactivation approximation and the ELBO associated with the proposed von Mises–Fisher posterior?

What evidence suggests that the directional posterior of a general multilayer Bayesian ReLU network is approximately factorized and isotropic?

2 comments
AuthorsOpenReview ↗

Response 1/2

Thank you for the review and for your interest in the geometric motivation.

"The authors should tone down their prose, and clearly articulate the extent of the contribution." We rewrote the abstract and the introduction with this in mind, and the contribution now reads "Instead of making the approximation richer, we design a variational model that makes the true posterior simple enough for a simple approximation to match it." On your two specific points, Proposition 1 is indeed elementary, and we now present it as a known fact (Poirier, 1998) whose role is to define the posterior on the sphere. It does not imply that a vMF is suitable, as you say, since the directional posterior of a general network can be anisotropic and correlated across units. This is why we design the network, and we show with HMC that normalisation and sparse activation bring its true posterior close to the isotropy and independence that the vMF assumes (Section 3 of the original manuscript, now Section 4.1).

"The paper does not establish a rigorous connection between the resulting training objective and the exact ELBO." Indeed, although the original manuscript quantified each of the two approximations against its exact counterpart, it did not state explicitly how our objective relates to the exact ELBO. In Section 3 we now say that, at β = 1, the objective is the negative ELBO of the vMF model with two substitutions, Gaussian noise for the vMF sample and a closed form for the exact KL. We also ran two new experiments to measure the combined error of these substitutions, one that evaluates the exact ELBO on a trained network and one that retrains the network with the exact model. The first two points below were in Appendices A and B of the original manuscript, and the last two are the new experiments.

  • The Gaussian noise has the exact mean of the pre-activation under the vMF. Its variance is within 20% of the exact one for inputs with no preferred alignment to the mean direction (4% at κ/D = 10).
  • The ratio of the closed-form KL to the exact KL depends only on κ/D. It tends to one for small and for large κ/D and stays between 1 and 1.22.
  • On a trained network our objective is 0.7% below the exact negative ELBO. This is the net of a data term 0.017 nats lower and a KL 0.015 nats higher (Table 5).
  • We implemented the exact model, with a vMF sampler and the exact KL, and retrained it. Accuracy and NLL change by at most about 1.3 standard errors. The ECE is 3.3 ± 0.3%, against 3.1 ± 0.2% for our objective (Table 6).

Training with the exact vMF model thus gives the same accuracy and calibration within error, which is why we regard CAP as a faithful approximation of this model. A bound on the gap between the two objectives would not by itself show that q is close to the posterior. MFVI maximises its own ELBO exactly and still reaches only a quarter of the true concentration (Table 1), which is why we measure closeness to the posterior directly against HMC.

AuthorsOpenReview ↗

Response 2/2

"What theoretical or empirical evidence supports the claim that parameterization accounts for 'much of the difficulty of Bayesian neural-network inference'?" The evidence is our measurement of how the parameterisation affects MFVI (Section 4.2). The parameterisation determines what the approximation has to match. In weight space this includes the length of the weights, although the likelihood does not depend on it. For MFVI the radial share of the KL rises from 50% to 98% as the fan-in grows 32-fold and its error on the directions grows from 12° to 49°, whereas the error of CAP, defined on the sphere, stays between 9° and 10° (Fig. 8). The parameterisation is thus the first step towards a posterior that a simple approximation can match, and the revised abstract states the difficulty in these terms ("the posterior over their weights is so complex that variational approximations fit it poorly").

"Can the authors establish a precise relationship between the Gaussian preactivation approximation and the ELBO associated with the proposed von Mises–Fisher posterior?" Yes, it is the relationship given above. The Gaussian approximation affects only the data term of the ELBO, where it replaces the vMF sample of the weights. On a trained network it lowers this term by 0.017 nats, and because the error of the closed-form KL has the opposite sign, the objective as a whole is within 0.7% of the exact negative ELBO (Table 5).

"What evidence suggests that the directional posterior of a general multilayer Bayesian ReLU network is approximately factorized and isotropic?" In the networks we trained, these properties were driven by normalisation and sparsity. In our teacher-student networks, layer normalisation makes the posterior of each unit nearly isotropic (the AM/HM ratio falls from 1.95 to 1.03), and sparse activation makes the units nearly independent (their correlation falls three- to sixfold, Table 12). We then checked with HMC that both properties hold in a trained Fashion-MNIST layer, whose units are almost uncorrelated (0.030) and whose learned concentration is close to the true one (within 5%, Table 1). In general networks, we expect that similar mechanisms may be at work, as most modern architectures are normalised and sparse activations emerge when ReLU networks are trained (Li et al., 2023). We agree that understanding anisotropy and how it might be mitigated in general networks is a promising line for future work (Section 7).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reading and their detailed suggestions. We are glad that the motivation was found "very interesting" (k2L5) and the paper "well motivated" (uHH1). We also appreciate that the comparison with the true posterior was recognised as "a good practice" that is "relatively rare in BNN papers" (H3hJ).

We kept the main argument and the results of the paper and updated them with new results that address the reviews. The paper proposes a Bayesian layer in which the posterior over the direction of each unit's weights has a single concentration parameter and acts on the unit as Gaussian noise with a learned scale. We design the network, with normalisation and sparse activation, so that the true posterior is close to this simple family. By comparison with HMC we show that the learned noise then follows the uncertainty of the true posterior, and the model is best calibrated without tempering.

We followed Reviewer H3hJ's recommendation and rewrote the presentation, which is why the revision took us until now. We believe that the paper has become much clearer. We adopted the structure that the reviewer proposed and made sure to define the notation and setups before their use. In the abstract and the introduction we now state the extent of the contribution explicitly, as Reviewer k2L5 asked.

We have changed the title to Untempered Bayesian Neural Networks via Scale Invariance in response to Reviewer H3hJ. The word "Precise" was meant for matching the posterior with precision, and it could indeed be confused with a precision variable in this context. "Untempered" encapsulates the paper better. Because we design the model so that its approximation matches the true posterior, it can be trained on the Bayesian objective without tempering.

Reviewers k2L5 and uHH1 both asked about the accuracy of the two approximations in our objective. The original manuscript compared each with its exact counterpart, and the revised one adds their combined effect, for which we retrained the networks with the exact vMF model. The accuracy and the calibration are unchanged within error, so the approximations do not affect our conclusions. We also completed the proofs of both propositions and now report the mean and standard error over multiple seeds, as Reviewer H3hJ requested. We also corrected the HMC reference in Table 1. The submission sampled the posterior given 20,000 of the training images rather than all 60,000. With the full data, the concentration CAP learns is within 5% of the true one, and variational dropout and MFVI reach half and a quarter of it.

Our answers to each reviewer follow, with references to the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.