acceptodds
Under review as a conference paper at ICLR 2027

A Reassessment of Global Weight-Space Sampling via Deterministic Flat Minima

Abstract

Deep Ensembles remain the empirical upper bound for Uncertainty Quantification, successfully capturing global epistemic diversity across disjoint loss valleys. To circumvent their prohibitive computational overhead, modern frameworks frequently rely on local weight-space sampling (e.g., SWAG, Laplace Approximations) to approximate an ensemble within a single basin. In this paper, we analyze the scaling issues that arise when global weight-space sampling is applied to overparameterized architectures, formalizing this as the weight-space variance dilemma. We highlight a geometric tension: strictly bounding stochastic weight perturbations within a certified optimal-transport safety radius leads to dimensionality dilution, where noise merges with the high-dimensional null space, leaving predictions functionally identical to the base checkpoint. Consequently, practical single-basin samplers are forced to inject unconstrained variance to achieve ensemble diversity. Such macroscopic noise can disrupt internal feature manifolds, necessitating computationally expensive deployment heuristics such as data-dependent batch normalization recalibration. Based on recent theoretical insights that flat minima can implicitly regularize predictive entropy, extensive evaluations on ResNet-50, ViT-B/16, and YOLOv8x demonstrate that much of the out-of-distribution logit smoothing attributed to global sampling can be natively embedded in the deterministic base optimizer. While unable to model true multi-modal posteriors, a single deterministic Sharpness-Aware Minimization (SAM) checkpoint serves as a highly efficient, zero-overhead baseline that frequently matches the local calibration of global ensembles. Furthermore, we show that while uniform Euclidean bounds stabilize homogeneous classifiers, complex dense prediction tasks require scale-aware Mahalanobis-Wasserstein spaces (ASAM) to prevent structural regression collapse.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty Quantification, Optimal Transport, Sharpness-Aware Minimization, Weight-Space Sampling, Out-of-Distribution (OOD) Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WMH3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work compares sharpness aware minimizer (SAM) to bayesian modeling methods to compare their accuracy and uncertainty quantification on classification and object detection tasks. The authors argued that bayesian methods that are global and perform full-parameter weight-space sampling are ill-posed because they either hit dimensionality dilution or functional disruption, and showed empirically that SAM and Adaptive SAM (ASAM) models are comparable to bayesian models on calibration and OOD detection.

Critical strengths
  1. This paper does a good job motivating single-basin method. The authors presented their experiment protocol and results clearly.
  2. The extensive experiment results are included in the appendix is appreciated.
Critical weaknesses

The novelty of this paper needs clarification. It is known that current bayesian methods can't sufficiently disentangle epistemic from aleatoric uncertainty (Mucsányi et al. 2024), which leads to suboptimal performance in OOD detection. It is also known that SAM and its variants can do well on calibration and OOD detection (e.g. Tan et al. 2026).

  • Tan et al. 2026, TOWARDS UNDERSTANDING THE CALIBRATION BENEFITS OF SHARPNESS-AWARE MINIMIZATION, ICLR
  • Mucsányi et al. 2024, Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks, NeurIPS
Questions for authors
  1. The stated weight-space variance dilemma argues that bayesian methods are impractical because the injected noise is either too small to matter or too big that it will disrupt the learned parameters. However, is SAM and ASAM more theoretically principled for uncertainty tasks?

  2. From the presented experiment results, SAM and ASAM are often still outperformed by deep ensembles, especially when looking at the full table across OOD shifts. Arguably one benefit of using SAM instead of deep ensembles would be the computation saving, I am curious to see how SAM would compare to other types of ensembles that are more computationally efficient? e.g. TreeNet (Lee et al. 2015, Kellner et al. 2024) and Asymmetric Duos (Zhou et al. 2025).

  • Kellner et al. 2024. "Uncertainty quantification by direct propagation of shallow ensembles." Machine Learning: Science and Technology
  • Lee et al. 2015, "Why m heads are better than one: Training a diverse ensemble of deep networks."
  • Zhou er al 2025. "Asymmetric duos: Sidekicks improve uncertainty." NeurIPS
Reviewer_g7zrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks the question whether, to get good uncertainty estimates, one needs to do ensembling or sampling in weight space, or whether using a single network trained with sharpness-aware minimization (SAM or ASAM) already gets most of the benefit. Concretely, they compare single SAM and ASAM models with, e.g., Deep Ensembles, SWAG and MC dropout variants on multiple benchmarks. Most significantly, they find that the single SAM and ASAM models often beat the sampling methods in terms of calibration.

Critical strengths

Generally, I believe the paper's question is relevant.

Furthermore, in terms of the methodology, I appreciate that there are a large number of tests (i.e., multiple different architectures) and that they are also comprehensively reported throughout the paper.

Critical weaknesses

One aspect that is missing to me is an additional comparison with a model that is fine-tuned in the same way but without SAM, or with temperature scaling, since it is unclear how much of the calibration benefit comes from SAM vs. just the fine-tuning itself or simple recalibration.

Questions for authors

I would be curious to hear more from the authors about how their work compares to and what significant insights it adds on top of the work of Tan et al. (2026) beyond just confirming their results?

Reviewer_8ZoEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether deterministic flat-minimum training can provide a competitive alternative to full-parameter, single-basin sampling for predictive calibration and robustness. It proposes a “weight-space variance dilemma”: arguing that a fixed global perturbation budget yields negligible functional diversity in high dimensions, whereas larger perturbations disrupt learned representations. It then evaluates deterministic SAM/ASAM against SWAG, deep ensembles, and activation-sampling baselines on ResNet-50, ViT-B/16, and YOLOv8x, using ImageNet-1k/C/O and COCO/COCO-O. Synthetic noise and recalibration ablations investigate inference cost and perturbation sensitivity. The reported results suggest that SAM/ASAM can be useful low-inference-cost baselines, while acknowledging advantages of sampling on some shifts and the absence of multimodal posterior modeling.

Critical strengths
  1. The deployment question is consequential. A single checkpoint and one forward pass can substantially reduce inference and model-storage costs. Table 1 reports competitive classification results and ASAM detection mAP of 0.5433 versus 0.5400 for the five-model ensemble. Establishing when such a baseline suffices would be valuable, even without a new optimizer or a Bayesian interpretation.
  2. The evaluation spans classification and dense prediction, and the paper reports useful counterexamples to its own broad narrative: SWAG performs better on some stylized COCO-O domains, while Euclidean SAM performs poorly on YOLOv8x. The full corruption/domain tables are helpful for examining these task-dependent tradeoffs.
Critical weaknesses
  1. The central variance dilemma is not established (Sections 3.1–3.4; Appendix A.2). Vanishing per-coordinate perturbation variance does not imply vanishing predictive variance without assumptions on the prediction Jacobian; likewise, low loss curvature does not imply unchanged predictions. The isotropic noise ablation therefore cannot establish inevitable failure for structured SWAG/Laplace distributions. Although the transport boundary is described as heuristic, the paper also treats its first-order derivation as an exact equivalence and safety certificate. Neither this argument nor the single-detector comparison establishes the claimed necessity of ASAM geometry.
  2. The uncertainty evaluation leaves key claims unsupported (Tables 7–13). ImageNet-O NLL/Brier targets are unspecified, and higher OOD entropy alone does not establish ID/OOD discrimination. Table 12 reports entropy of 9.903 for 1,000 classes, exceeding the natural-log maximum of 6.908 used by the supplied code; the units or implementation require reconciliation. Bounding-box standard deviation is also undefined for deterministic checkpoints. Missing deterministic SGD/AdamW controls, run-to-run variability, and radius sensitivity further limit attribution of the reported gains and SAM collapse to geometry.
Questions for authors
  1. What assumptions on the prediction Jacobian, loss curvature, and parameter scaling make the dimension-only dilution argument valid? Please distinguish a first-order W₁ surrogate, a W₂ moment budget, and a pointwise perturbation guarantee, and state which conclusions apply beyond the isotropic diagnostic sampler.
  2. How are ImageNet-O targets and the NLL/Brier metrics defined? Can you provide ID-versus-OOD AUROC/AUPR or FPR95 with a specified score and ID set, to establish anomaly discrimination rather than only increased average OOD entropy?
  3. What entropy definition produces Table 12’s values above ln(1000), given the natural-log implementation? How is bounding-box standard deviation computed for a deterministic checkpoint, including the repeated observations and matching rule? Please also reconcile Figure 1a’s ASAM-above-ensemble Pixelate ranking with Table 7, which reports the reverse ranking at every severity.
  4. Are the same pretrained starting checkpoints, tuning budgets, and evaluation rules used across methods? Please provide the plain deterministic SGD/AdamW baseline, independent-run variability for the main comparisons, and a SAM/ASAM radius sensitivity check on YOLOv8x. These would distinguish a robust geometric finding from optimizer selection or an unfavorable SAM configuration. The 0.0033 detection mAP advantage should be interpreted with this variability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.