acceptodds
Withdrawn from ICLR 2027

Separating Posterior Geometry from Inference Mechanisms: A Unified Cayley-Orbit Posterior for Bayesian Neural Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Bayesian neural networks (BNNs) quantify epistemic uncertainty through posterior distributions over weights. Usually, weight posteriors in BNNs primarily control the scale and statistical dependence of perturbations in an unconstrained weight space, providing a tractable representation of weight uncertainty. However, such posterior parameterizations can suffer from restricted directional exploration and unstable weight structure during sampling. Moreover, posterior construction is often embedded within a particular inference mechanism, making useful posterior geometries difficult to transfer across BNN methods. To address these limitations, we introduce Unified Cayley-Orbit Posterior (UniCOP), a unified posterior constructor separated from the posterior inference mechanism that used to learn the posterior parameters. Our UniCOP constructs a posterior on a curved orbit by applying Cayley transform to a learned reference weight, preserving the reference weight’s structure. This geometry-aware construction better models the epistemic uncertainty through directional exploration while keeping structural properties, supporting predictive stability and robustness. Moreover, our constructor can therefore be employed with both explicit variational and optimizer-induced inference mechanisms. Our theoretical analysis characterizes how the Cayley-orbit construction preserves weight-matrix structure and establishes the geometric validity and versatility of UniCOP. Extensive experiments across multiple BNN methods also demonstrate consistent improvements of UniCOP on uncertainty estimation, robustness to input perturbations, and predictive generalization.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian Neural Networks, Unified Posterior Geometry, Uncertainty estimation, Cayley-Orbit Posterior

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_T9EZOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

UniCOP separates posterior sample construction from the inference mechanism. A Gaussian latent matrix is skew-symmetrized and transformed into a Cayley rotation of a learned, non-sampled reference weight. Samples preserve the reference row Gram matrix, singular values, and Frobenius norm. The paper develops a conditional latent-space ELBO, geometric characterization, and function-space uncertainty analysis, then evaluates several BNN inference methods and feature-posterior transfer to CLAP4CLIP.

Critical strengths
  1. The probability-space formulation appropriately handles orbit-supported distributions. The latent ELBO and KL contraction handle singular weight distributions correctly. The exact structural invariants and local tangent coverage establish the geometry of the resulting support. The ELBO is appropriately conditional on the learned, non-sampled reference. [pp. 4–5; Appendix A.1.1–A.1.2, pp. 13–15]

  2. Practical evidence extends beyond the disputed mechanistic control. Independently trained IVON+UniCOP improves all four reported mean metrics across CIFAR-100, Tiny-ImageNet, and ImageNet-100, including accuracy gains of 0.36, 0.48, and 1.12 percentage points. On held-out CLAP4CLIP class orders, future-class MI AUROC improves by 1.73 points, with a 0.54-point final-accuracy reduction. [Table 8, p. 22; Table 10, p. 24]

Critical weaknesses
  1. The Euclidean control does not generally match local perturbation energy. Equations (59)–(60) use a dimension-only scale, whereas the Cayley Jacobian depends on the latent center and its expected energy depends on the covariance. Writing fW∘(R)=W∘Q(R)⊤f_{W^\circ}(R)=W^\circ Q(R)^\top for the constructor, including skew extraction, take d=2d=2, reference row W∘=(1,0)W^\circ=(1,0), skew center with M12=−M21=aM_{12}=-M_{21}=a and M11=M22=0M_{11}=M_{22}=0, and iid standard-normal perturbation entries in HH,

E∥DfW∘(M)[H]∥F2=2/(1+a2)2,\mathbb E\|Df_{W^\circ}(M)[H]\|_F^2=2/(1+a^2)^2,

while the Euclidean derivative has expected squared norm 22. At a=1a=1, the comparator therefore has fourfold local energy. The comparison also decodes a Cayley-trained posterior through an alternative map, including directions discarded by skew extraction. It therefore does not isolate geometry at matched perturbation energy. [Eq. (33), pp. 14–15; Eqs. (59)–(60), Table 7, p. 20]

  1. The practical cost–benefit comparison remains underspecified. The BNN layers, convolution grouping, blocks, and reference/latent update rules are insufficiently specified. A dense dd-dimensional Cayley solve requires O(d3)O(d^3) arithmetic and O(d2)O(d^2) latent storage; blocking changes both cost and accessible geometry. Appendix A.4 specifies CLAP4CLIP blocking, while also disclosing unmatched posterior parameter counts. Without analogous BNN details and measured overhead, the accuracy/calibration/OOD trade-offs cannot be assessed against resource-matched alternatives. [p. 4; Appendices A.2 and A.4–A.6, pp. 18–24]
Questions for authors
  1. What are the actual Jacobian-energy ratios at the trained checkpoints, and do Table 7’s ranking gains survive correct energy matching? A corrected control is needed to attribute the difference to posterior geometry.

  2. Please specify the BNN layer/block implementation, parameter counts, reference/latent updates, and measured training/inference time and memory.

  3. Under matched parameter and sampling budgets, how do the OOD gains compare with accuracy and calibration costs, particularly in CLAP4CLIP? Corrected attribution and a favorable, documented resource–quality trade-off would support a rating of 3.

Reviewer_AUEwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a method to sample weights in Bayesian neural networks. Instead of adding Gaussian noise directly to the weights, it learns a fixed reference weight matrix and a Gaussian distribution over rotations (Q which is a random rotation matrix built from a Gaussian sample.) and each sample rotates the reference weights. Training uses a standard variational objective on the rotation parameters while treating the reference weights deterministic. Providing improvements over the plain parameter sampling

Critical strengths
  • The idea of restricting posterior samples to rotations of a learned reference weight is simple, differentiable.
  • The construction is compatible with several inference methods
  • The evaluation covers a broad range of tasks and the inclusion of the CLIP feature experiments was very well received
Critical weaknesses

Presentation and notation

  • The method is hard to follow because basic quantities are never clearly defined. There is no background section setting out the standard BNN components (likelihood, prior, posterior over weights)
  • Shapes of the main objects are never stated in the method section (dimensions of the weights only appearing in Theorem 2). As a result, the reader is left with no impression the size of the latent space.
  • No dimensions are given for any real layer, including how convolutional layers are handled
  • The notation is inconsistent or redundant: W₀ (a Gaussian sample) and W° (the reference weight)
  • Several new terms (“posterior constructor,” “structure-preserving control,” “geometric representation efficiency”) with no clear meaning in the literature

Missing related work (compromises novelty claims)

  • The paper omits a closely related line of work: (Izmailov et al., 2020), which also places a simple distribution over a small set of latent coordinates and maps it into the full weight space. UniCOP follows the same pattern: a Gaussian over latent rotation parameters is mapped through a nonlinear transformation to weights on a curved set around a reference weight. These differences should be discussed explicitly, and subspace inference should be included as a baseline, since it directly tests whether the gains come from the specific rotation geometry or simply from restricting where posterior uncertainty can live.

    • Izmailov, Pavel, et al. "Subspace inference for Bayesian deep learning."(UAI 2020)
  • Constructing a BNN posterior by pushing a Gaussian through a transformation is well established, most directly in posterior refinement with normalizing flows (Kristiadi et al., NeurIPS 2022), which also targets the shape of Gaussian posteriors to improve predictive performance.

    • Kristiadi, Agustinus, et al. "Posterior refinement improves sample efficiency in Bayesian neural networks." (NeurIPS 2022):

Missing strong UQ baselines

  • The paper claims state-of-the-art performance (Section 6.1), yet omits the standard and strongest uncertainty-quantification baselines: deep ensembles are neither cited nor compared, Laplace approximations are cited but not evaluated, and recent efficient alternatives such as Variational Bayesian Last Layers (Harrison et al., 2024) are not considered
    • Harrison et al. "Variational Bayesian last layers." (ICLR 2024)

Minor weaknesses

  • No highlighting of best results. The tables do not mark the best (or statistically tied) result per metric. The method doesn’t need to win everywhere, but bolding the best results would make the comparisons much easier to read.

  • Figure 2 is hard to interpret. The IVON and IVON+UniCOP curves are visually almost indistinguishable

  • Table 10 belongs in the main paper. The Tiny-ImageNet and ImageNet-100 results are the only evidence that the method generalizes beyond CIFAR

  • Theory is largely standard. They describe the construction but don’t explain why it should improve uncertainty or accuracy.

Questions for authors

Why must the reference weight W° be learned jointly from scratch, and how is it initialized? Could UniCOP instead be applied post hoc, fixing W° to a pretrained MAP solution (or a SWAG mean) and learning only the rotation posterior (M, S)? Such a variant would be much cheaper, would enable direct comparison with post hoc methods such as Laplace and flow-based posterior refinement, and would help separate the effect of the rotation posterior from that of jointly retraining the weights.

Reviewer_Qa6hOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a new family of distributions, to use to approximate posteriors of neural networks. The new approximate posterior is motivated by samples preserving the Gram matrix, Frobenius norm, and singular values.

Critical strengths

A new approximate posterior with interesting structure was introduced.

Critical weaknesses

The approximate posterior is introduced with no justification is given as to preserving the mentioned properties is good. The paper is written as though it is obvious that preserving these properties is good. The paper introduces new concepts that seem tautological (e.g. "structure-preserving control", "geometric representation efficiency"), and the theorems just prove the definition of these properties, from which the method was derived. For theorems to be actually useful, they would need to prove why these properties are good, or what consequences of these properties are.

The justification for the theorems is also extremely poor, e.g. the two-sentence justification of Theorem 1. There is no definition of what is even meant by "validity" of the variational objective. In fact, I would argue that the variational bound is not particularly valid, as there is no Bayesian treatment of most of the parameters that are optimized: W◦. This is not even acknowledged! Surely if enough care is taken to discuss the validity of the variational bound, this should be discussed, as this is the main departure from a full Bayesian treatment!

Some existing concepts are reintroduced under another name. E.g. line 150 is just the reparameterization trick.

The writing is obtuse and unclear, with many concepts being introduced before (or even without) definition.

  • What is meant with "geometric control" (line 43), and "directional exploration". These terms are not defined, and not standard in the field.
  • "Fail to preserve geometric properties of weights" (line 47): What is meant with this? Does the Louizos & Welling paper make this claim, or is it an example of a paper that fails to preserve geometric properties? (I know this paper well).
  • "Specifically in a rotation orbit [...]" (line 48.5) This may be true for a Gaussian, although this is not mentioned explicitly. Without reference to a distribution, this sentence is vague.
  • What is a "posterior constructor"? Is this an existing concept? If not, why is this introduced without definition?

I find this paper too poorly written to discern if it has any value.

Questions for authors
  • What justification does the paper provide on why preserving the weight's Gram matrix, Frobenius norm, and singular values is good? Where can I find this in the paper? Perhaps the paper should have acknowledged from the start that this is an idea that is later empirically verified?
  • Is there a Bayesian treatment of W◦? What does this imply for the validity of the variational approximation?
  • How is the "unified posterior construction" not just the reparameterisation trick? Would it not be better to acknowledge the similarity and use the same name, and cite the original paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.