acceptodds
Under review as a conference paper at ICLR 2027

From Optimization to Sampling: MUON-Inspired Langevin Dynamics for Bayesian Learning

Abstract

Matrix-aware optimizer such as MUON exploits the spectral geometry of neural-network weight matrices, but their update directions cannot be inserted directly into Langevin dynamics without generally changing the invariant distribution. We ask instead where MUON's matrix geometry can enter Bayesian Neural Network sampling while retaining a clear probabilistic interpretation. We identify two complementary routes. Proximal-MUON Langevin Dynamics (PMLD) transfers the geometry to the target by introducing a spectral prior, yielding a smooth posterior with explicit spectral localization and robustness guarantees. Annealed PMLD (APMLD) takes the transient route: it uses a finite MUON phase to construct a randomized warm start and then switches to target-preserving Langevin sampling. Experiments on Bayesian classifiers and diffusion models reflect these distinct roles: PMLD improves robustness through spectral posterior structure, while APMLD improves finite-time posterior-predictive performance without altering the eventual sampling target. Together, the two methods illustrate a simple design principle for transferring optimizer geometry into Bayesian sampling: change the target, or change only the transient.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian neural networks, posterior sampling, Langevin dynamics, MUON, matrix-aware optimization

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_6xpgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Two algorithms, Proximal MUON Langevin dynamic (PMLD) and Annealed PMLD (APMLD) are introduced to incorporate the geometry induced by the MUON optimizer into the Langevin dynamics for improving Bayesian learning tasks. For the first algorithm, PMLD, the authors design a version of the Unadjusted Langevin algorithm (ULA) that targets a "modified" posterior measure involving the original objective and the Moreau-Yosida envelope of the indicator function of a spectral-norm ball. In this way, the spectral constraint is encoded through a smooth prior. The aim of the second algorithm, APMLD, is to propose "a more informative way" to initialize the sampling to improve the finite-time behaviour. In particular the algorithm runs MUON on the smoothed potential during an initial phase, then selects the iterate associated with the smallest (nuclear) norm of the gradient of the smoothed potential, applies a Gaussian perturbation, and then run standard ULA to the chosen target. A convergence analysis is provided under strong convexity and smoothness assumptions separating the equilibrium spectral localization from the (finite-time) sampling error of ULA. Numerical experiments on Bayesian classifiers show the advantage of PMLD in terms of robustness comparing with ULA on Bayesian diffusion models and Bayesian classifiers. Moreover, APMLD shows good early posterior-predictive performance.

Critical strengths
  1. To the best of my knowledge, this work is one of the first attempts in the literature to study the MUON's spectral geometry within the framework of the Langevin dynamics, which is a highly non-trivial and important research problem. I would add another perspective of the importance of this work beyond the motivation for Bayesian learning stated in the manuscript by the authors. Stochastic MUON with fixed mini-batches may fail to converge to a global minimizer even for smooth and strongly convex objectives, see, e.g., [1]. Moreover, non-convergence examples for convex Lipschitz functions have been appeared recently in the literature, see, e.g., [2]. Therefore, it is crucial to investigate how Muon's spectral geometry can be incorporated into stochastic frameworks whose long-time behaviour can be rigorously characterized with stochastic analysis tools. Such direction may provide alternative approaches to studying MUON-inspired algorithms and overcome some of the limitations of the current optimization guarantees in the literature. While the proposed PMLD algorithm is designed for Bayesian learning tasks (and it is in essence a standard ULA applied to a modified target) and for this reason, it does not directly address the issue of the convergence of stochastic MUON, I believe that the idea of the authors behind PMLD could provide an important stepping stone for further studies of MUON-"inspired" Langevin algorithms, which will be of benefit not only to the sampling community, but in particular to the optimization community.

References:

[1] On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations, Thang Do, Steffen Dereich, and Arnulf Jentzen

[2] Muon Does Not Converge on Convex Lipschitz Functions, Tetiana Parshakova, Ahmed Khaled, Michael Crawshaw, Guillaume Garrigos, Robert M. Gower

  1. The authors manage to show that, outside the prescribed spectral region, the posterior associated with the PMLD target (composed of the original objective and the Moreau-Yosida envelope) has exponential decay in the squared spectral violation. This is important because it allows to quantify the contribution of the spectral prior and then later in the finite-time analysis separate the equilibrium spectral localization (MUON part) from the finite-time sampling error (ULA part). This contribution may be useful for the subsequent analysis of convergence of stochastic MUON suggested in point 1 above.
Critical weaknesses
  1. (Critical Weakness): Limited discussion of the comparison with prior work on ULA/MALA/proximal gradient algorithms in the convergence analysis, including the comparison provided in Table 1 (page 2). More clarification is needed for positioning the proposed algorithms. More discussion about the "representative rate" is needed, particularly regarding the contribution of the modified target in PMLD and the differences between the target distributions considered in prior work. See question 1 below.

  2. The convergence analysis for ULA is provided under very strong assumptions (strong convexity and smoothness). These assumptions are presented as necessary at page 4: "while the finite-time ULA guarantees additionally require the global convexity and smoothness assumptions stated below." The literature on the convergence analysis of ULA is quite well-established beyond strong convexity and smoothness, see, e.g., "From stability of Langevin diffusion to convergence of proximal MCMC for non-log-concave sampling" NeurIPS 2025, and "The Performance Of The Unadjusted Langevin Algorithm Without Smoothness Assumptions" TMLR, 2025 and references therein for the results in the nonconvex and nonsmooth settings. This is a limitation of the current convergence analysis, which does not cover the nonconvex settings under which the experiments are conducted. The authors state this limitation in the text, but still it remains unclear how the theoretical finite-time guarantees can explain the behavior of the experiments shown in the numerical experiments.

Questions for authors
  1. In Table 1 and in the text below the table, the comparison of the "representative rate" is very limited, particularly regarding the different target distributions (and assumptions) considered in prior works. Some of the parameters in Table 1 are not specified.

(a) For the RGO sampler result stated in the table, what other terms are contained in O~\widetilde{O}? Can you explan why the dependence in the dimension in the PMLD and APMLD convergence analysis is worse comparing to the result of Liu and Chewi, (2026) for the RGO sampler (taking into account the different oracle assumptiosn)?

(b) What is the price in terms of the dimension and accuracy that we have to pay when using PMLD and APMLD instead of Proximal Langevin dynamics and Proximal Muon Langevin dynamics under Assumption 4.1? For PMLD how does the dependence on the parameter γ\gamma (Moreou-Yosida parameter) has an effect on this comparison?

(c) In addition, can you please clarify this sentence (Page 2, lines 97-98):

"Table 1 records representative guarantees together with their initialization requirements. " Where the initialization requirements are stated in Table 1?

  1. In proximal Muon Langevin dynamics, how can we choose the spectral cap RR? Do you need to be tuned? Is it problem specific as in Section 5.1.1? Is there a way that I can select it based on the values of γ\gamma or if the inverse temperature/temperature parameter is set to be different from one in ULA (i.e. 2β−1\sqrt{2 \beta^{-1}} or 2τ\sqrt{2 \tau} with β−1=τ=1\beta^{-1}=\tau =1) can it be related to it? Or do we need to tune separately RR and γ\gamma?

  2. Section 5.2 "Bayesian Diffusion Models" compares the performance of PMLD/APMLD with SGLD rather than ULA. Can you please comment whether the convergence analysis for the PMLD can be extended to PMLD with the stochastic gradient version of PMLD? The theory of non-asymptotic convergence analysis of SGLD is quite now well-established for results beyond convexity and standard smoothness assumption.

  3. (Suggestion) It would be helpful to summarize what are the main difficulties in proving convergence analysis for PMLD and APMLD comparing to ULA, as the proof in Appendix F seems to follow standard arguments for the ULA convergence analysis.

  4. (Suggestions to improve clarity) It would be helpful to provide a more detailed derivation for the steps summarized in the proof of Lemma F.2 in lines 890-897 at page 17.

  5. (Suggestions) I would suggest including in Proposition 4.3 the Big-O expression of the bounds for the discretization error and the initialization error and mixing terms provided in Theorem F.1. As it is stated right now, the finite-time sampling bound in Proposition 4.3 is standard for ULA and is not very informative about the specific properties of the PMLD and APMLD. Lemma F.2, Theorem F.1: It would be helpful to state and highlight the restriction on the step size η\eta provided in these two statements in the main text (Section 4) since LgL_g depends on γ\gamma for PMLD algorithm.

5 comments

(1/2)

We sincerely thank the reviewer for the detailed technical comments and helpful references! We agree that the convergence comparison and the presentation of our theoretical results can be substantially clarified.

Q1(a). Their Theorem 3.1 establishes a TV-accuracy complexity of O~(κdlog⁡4(1/ε))\widetilde{\mathcal O}(\kappa\sqrt{d}\log^4(1/\varepsilon)), where κ=max⁡(1,β/α)\kappa=\max(1,\beta/\alpha), and O~\widetilde{\mathcal O} suppresses additional logarithmic factors in dd and κ\kappa. Their algorithm uses an augmented-space proximal sampler with Metropolis–Hastings corrections, access to the mode, and a restricted Gaussian oracle (RGO) for gg. For our spectral-ball indicator, this oracle requires sampling a Gaussian restricted to the spectral-norm ball, which is different from evaluating its projection, and might not have implementation in a closed form . Their sharper dimension and accuracy dependence reflects both the oracle access and the sampling mechanism. Our methods instead retain explicit Euler–Langevin updates and their associated discretization error. We will clarify these differences rather than suggest a direct comparison of computational costs.

Q1(b). Under Assumption 4.1, PMLD retains the polynomial dimension and accuracy dependence O~(κγ2d/ε2)\widetilde{\mathcal O}(\kappa_\gamma^2 d/\varepsilon^2) of ULA, with κγ=(L+γ−1)/M\kappa_\gamma=(L+\gamma^{-1})/M. Holding initialization constants fixed, the condition-number factor increases relative to ordinary ULA by $\frac{\kappa_\gamma^2}{\kappa^2} =\left(1+\frac{1}{L\gamma}\right)^2.$ Thus, smaller γ\gamma strengthens spectral localization but worsens the conservative smoothness constant and restricts the sampling step size.

Q1(c). Thank you for identifying this oversight. We will add initialization and oracle requirements to Table 1. However, it is worth pointing out that our objective is not to improve upon the convergence rates of state-of-the-art samplers, but rather to provide a representative overview of related methods and position our proposed algorithms within the broader sampling literature.

Q2. These parameters have distinct roles: RR specifies the desired spectral scale, whereas γ\gamma controls the softness of the constraint. In Section 5.1.1, we use the layer-specific choice Rℓ=0.5∣Wℓ,MAP∣2R_\ell=0.5\\|W_{\ell,\mathrm{MAP}}\\|_2 and γ=10−4\gamma=10^{-4}. Indeed, they can be tuned separately. Here our γ\gamma is the scale on prior only, while a temperature β\beta or τ\tau governs the whole posterior; they are compatible since the scales are multiplicative. For example, we can have

πγ,τ(∣W∣op>R+t)≤exp⁡[−t2/(2γτ)]πτ(B(R)),\pi^{\gamma,\tau}(\\|W\\|_{\text{op}} > R+t) \leq \frac{\exp[-t^2/(2\gamma\tau)]}{\pi^{\tau}(B(R))},

where πτ∝e−U/τ\pi^{\tau}\propto e^{-U/\tau} is the tempered base distribution.

Q3. Since stochastic PMLD is SGLD applied to UγU_\gamma, existing SGLD analyses can be utilized when their assumptions hold. More concretely, under Assumption 4.1 and conditionally unbiased stochastic gradients with conditional variance bounded by σG2\sigma_G^2, synchronous coupling of full-gradient PMLD (XkX_k) and stochastic PMLD (YkY_k), starting from the same initialization, gives

\leq \sigma_G\sqrt{\frac{\eta(M+L_\gamma)}{2ML_\gamma}},$$ for $0<\eta\leq 2/(M+L_\gamma)$. This provides an explicit additional stochastic-gradient error in Wasserstein distance. We also appreciate the references on nonconvex and nonsmooth Langevin analysis. Global strong convexity and smoothness are assumptions of our particular analysis, not necessary conditions for ULA convergence generally. For example, Vempala and Wibisono (2019) establish ULA convergence under smoothness and a log-Sobolev inequality without global convexity. Since PMLD and the post-switch phase of APMLD reduce to ULA, such results provide alternative sampling guarantees under their respective assumptions. We will clarify this broader applicability while acknowledging that our current quantitative guarantees do not directly cover the experimental neural-network posteriors.

(2/2)

Q4. We agree that Appendix F follows standard ULA arguments. The method-specific contributions concern spectral target localization for PMLD and the conversion of MUON's optimization progress into quantitative initialization bounds for APMLD. Our common sampling analysis deliberately separates these contributions from the subsequent Langevin dynamics, rather than introducing a new ULA convergence mechanism.

Q5–Q6. Thank you for this suggestion. The paragraph provides a technical justification for applying the standard Girsanov-based discretization argument for Langevin Monte Carlo (see, e.g., Section 4.4 of Chewi's Log-Concave Sampling). The localization step ensures that the relative entropy inequality is valid under our finite-second-moment assumption, without requiring additional exponential integrability of the unstopped drift difference. We will expand the derivation by explicitly defining the stopping time, stating the Girsanov entropy identity for the stopped path laws, and justifying the passage to the unstopped laws.

The explicit discretization and initialization-dependent bounds, together with the step size restriction, are already established in Lemma F.2 and Theorem F.1. Section 4.3 also discusses the trade-off between spectral localization and the increased smoothness constant induced by smaller Moreau parameters (lines 340–344).

We agree, however, that incorporating these quantitative estimates directly into Proposition 4.3 would make the method-specific implications more transparent. We will therefore include the explicit discretization bound from Theorem F.1 and highlight the step size restriction in the main text, particularly its dependence on the Moreau parameter for PMLD. This will better illustrate how PMLD affects the target smoothness and discretization error, whereas APMLD affects the initialization-dependent terms.

Reviewer_6xpgOpenReview ↗

Reply to Authors

Thank you very much for the detailed rebuttal.

Q2 in (1/2): I have small a follow-up question about this point. In practice, can you identify a rule for choosing RR and/or γ\gamma? Have you observed any behaviour/pattern for choosing the values for these parameters, e.g. through a sensitivity analysis in Experiment 5.1.1? If not possible, did you get any insights into how to choose RR and γ\gamma from the derivation of Proposition 4.1?

Q1(b) in (1/2): OK, thank you for clarifying this. I suggest to report explicitly the value of κγ\kappa_\gamma in the Table 1 if possible to highlight this point about the trade-off to the readers since the aim is to position the proposed algorithms within the broader sampling literature (as point it out in the answer to Q1 (c)).

Q1(a), Q1(c), and Q3 in (1/2): OK.

Q4- Q6 in (2/2): I thank the authors for taking into account my suggestions.

Reviewer_QNwYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MUON optimizer enjoys demonstrated success in training large-scale neural networks by utilizing the spectral geometry of the weights. The paper studies how to extend MUON to performing Bayesian sampling, proposing two alternative solutions, Proxmal-MUON Langevin Dynamics (PMLD) and Annealed PMLD (APMLD). PMLD modifies the Bayesian prior placed on neural network weights to a special Moreau prior, so as to recover the desirable properties of MUON during sampling. APMLD instead uses MUON for warm-start, where ordinary Langevin sampling is performed after running some steps of MUON optimization. Empirical evidence shows that PMLD and APMLD to better results than ULA using Gaussian prior and combining Adam with ULA when applicable.

Critical strengths

The paper studies an interesting problem, namely how to use the geometric notions of MUON to improve Bayesian inference methods for Bayesian neural networks. The problem is timely, and reasonable solutions for it would indeed be interesting.

Critical weaknesses

I am hesitant regarding both PMLD and APMLD. For PMLD, if I am understanding correctly the proposed solution essentially replaces the Gaussian prior with a different spectral Moreau prior and runs ordinary Langevin sampling. However, while a Gaussian prior is arguably the most obvious and common choice, it is by no mean the only choice. There are work studying different priors that may be heavy-tailed ones [1] as well as work proposing functional priors [2]. I am sure that are more literature on this. For APMLD, it is perhaps not too surprising that one can first use an optimizer to optimize the parameters before running actual sampling; for instance, the code for [1] on GitHub already supports such option. A bit minor, but I also find the names slightly confusing: while they are quite different algorithms in practice, APMLD sounds like simply the annealed version of PMLD.

[1] Bayesian Neural Network Priors Revisited, Fortuin et al., ICLR 2022

[2] All You Need is a Good Functional Prior for Bayesian Deep Learning, Tran et al., JMLR 2022

Questions for authors

Please refer to Critical Weaknesses above.

  1. Can the authors comment on PMLD and APMLD, especially clarify the novelty and new contributions upon what was previously known?

  2. Some previous works look into the approach of accounting for the geometry using a position-dependent Riemannian metric [1] [2] [3] [4]. Given that the authors explicitly noted geometry, it might be interesting to discuss this alternative line of work.

[1] Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks, Li et al., AAAI-2016

[2] Stochastic gradient Langevin dynamics with adaptive drifts, Kim et al., Journal of Statistical Computation and Simulation, 2022

[3] Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics, Yu et al., TMLR 2023

[4] On the Convergence of Locally Adaptive and Scalable Diffusion-Based Sampling Methods for Deep Bayesian Neural Network Posteriors, Rensmeyer and Niggemann, AISTATS 2025

3 comments

We sincerely thank the reviewer for these thoughtful comments and relevant references! They help us better position our work within the broader literature on Bayesian deep learning.

Q1. Novelty of PMLD and APMLD. We are familiar with Fortuin et al. (2022), which was one of the works we consulted when developing this paper. We appreciate the suggestion to connect our work more explicitly with alternative prior specifications, including heavy-tailed and functional priors.

Our starting point is a somewhat different question: given the recent success of MUON in training large neural networks, can its matrix-geometric insights be translated into Bayesian sampling? Our objective is not to propose a universally superior prior or sampler, but to investigate how this connection can inform new sampling methodologies.

For PMLD, we agree that departing from Gaussian priors is not itself novel. Our motivation comes from MUON's interpretation as spectral-norm-constrained steepest descent. PMLD seeks to mimic this spectral control at the posterior level through a Moreau-smoothed spectral constraint. Proposition 4.1 formalizes the resulting posterior spectral localization. We view this as complementary to prior design based on empirical weight distributions or functional properties, as studied by Fortuin et al. and Tran et al.

For APMLD, we agree that optimizer-based initialization is established practice. Our contribution is the investigation of MUON specifically as a matrix-aware initialization strategy, together with a finite-time analysis characterizing how such initialization influences subsequent Langevin sampling. Our Adam+ULA comparisons also aim to distinguish MUON-specific benefits from generic warm-start benefits.

We also appreciate the comment on naming! We agree that APMLD may misleadingly suggest an annealed version of PMLD, whereas the two represent distinct design routes. We will reconsider the terminology to make this distinction clearer.

Q2. Riemannian and adaptive Langevin methods. Thank you for highlighting this important line of research! We indeed considered geometry-aware Langevin dynamics when developing our methods. Position-dependent preconditioning offers a promising alternative for incorporating matrix geometry into sampling. Actually, SGRLD is among the first few frameworks of sampler designs we had tried out.

However, directly translating MUON's spectral geometry presents an additional challenge. As shown by Peyré (2026, Appendix H.5), the spectral Wasserstein gradient flow of relative entropy is generally nonlocal, with its effective preconditioning depending on global statistics of the evolving distribution. Consequently, unlike classical Wasserstein entropy flows, it does not generally admit a conventional autonomous single-particle Langevin representation.

This does not rule out local Riemannian samplers inspired by MUON, but it suggests that faithfully realizing its spectral Wasserstein geometry may require law-dependent dynamics, introducing additional computational and analytical challenges in large-scale Bayesian deep learning.

Our work therefore investigates two complementary routes that avoid these complications: incorporating spectral geometry through the target distribution (PMLD) or through initialization (APMLD). We appreciate the suggested references and will expand our discussion of their connections to our work, including the recent MUON-inspired underdamped Langevin construction of Makras and Sabanis (2026).

I thank the authors for their response. However, while the problem is well-motivated, I still do not find the proposed methods novel enough for a venue like ICLR.

Some other findings while revisiting the paper: Bayesian diffusion models seem to be the only experiments where PMLD and APMLD are compared against each other, but for these experiments each method only used 5 or 10 posterior samples, which I imagine could be quite crude for such nontrivial and high dimensional posteriors.

We thank the reviewer for the feedback and for taking the time to revisit our work.

We acknowledge the limitations of the relatively small number of retained posterior samples in our diffusion experiments. While these experiments were intended primarily to demonstrate the practical applicability of our methods to challenging generative modeling tasks, we agree that a larger number of posterior samples would allow for a more comprehensive evaluation of their sampling behavior.

We appreciate the reviewer's constructive comments and suggestions, which will be valuable in guiding our future research.

Reviewer_xhGfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper tackles how to exploit spectral geometry, an underlying concept of the MUON optimizer, in Bayesian posterior sampling. It suggests two algorithms: PMLD and APMLD. PMLD is proximal-MUON Langevin dynamics, and it replaces the typical Gaussian prior with a spectral-norm prior. On the other hand, APMLD is annealed proximal-MUON Langevin dynamics, where the initial sample (for warmup) is determined by a converged optimum of the MUON optimizer while using the same posterior as in conventional Langevin dynamics. Both are tested by sampling a portion of the parameters of image classifiers and image diffusion models, as in large-scale Bayesian deep learning. They showed better performance than the baseline, unadjusted langevin dynamics, in terms of NLL, accuracy, ECE (image classification) and FID (image generation).

Critical strengths
  1. The paper showed some theoretical explanation how spectral geometry can affect Bayesian deep learning.
  2. The paper evaluated performance benefits of the spectral geometry in different models and different metrics.
Critical weaknesses
  1. Novelty is limited. PMLD is Langevin dynamics + spectral norm regularization. APMLD is MUON (for warmup) + Langevin dynamics. Both are just combinations of two existing methods. The spectral norm stabilizes the Lipschitz constants of a neural network, whose connection with sample diversity is not addressed. Proposition 4.1 explains posterior spectral localization, but I do not know why this helps diversity or efficiency of sampling. Proposition 4.2 explains a benefit of MUON warmup in bounding the distance to the posterior, but I think we can get a similar upper bound even with ULA warmup under the strong convexity and smoothness assumptions (which do not even align with the loss surface). They addressed benefits in robustness of the prediction, but it is less relevant to the challenges in Bayesian inference. Rather, it is relevant to the NN prediction performance of Bayesian deep learning. I recommend explaining (1) why the posterior spectral localization is beneficial for posterior sampling and (2) why the MUON initialization is better than other optimizers in terms of Bayesian inference.

  2. The experiments do not fully reflect the benefits in Bayesian deep learning. Although the paper includes some uncertainty metrics such as NLL and ECE, they are limited in their ability to express the performance of Bayesian inference. I recommend including NLL and ECE on out-of-distribution data.

Questions for authors
  1. It seems PMLD and APMLD have independent benefits. Combining them into a single algorithm can be better than individuals?
  2. Why did you use ULA as a main posterior sampler instead of SGMCMC, which is more practical?
3 comments

We thank the reviewer for their constructive comments.

Novelty and motivation. Our contribution is to investigate two principled routes for translating MUON's matrix geometry from optimization to Bayesian sampling: modifying the target distribution (PMLD) and improving initialization (APMLD).

For PMLD, our motivation goes beyond conventional spectral regularization of neural networks. MUON admits a steepest-descent interpretation under a spectral-norm constraint. PMLD is designed to mimic this spectral control in the sampling setting by introducing a spectral prior and constructing the corresponding Langevin dynamics. Proposition 4.1 formalizes how this construction controls the spectral properties of posterior samples, rather than merely those of a point estimate. We agree that spectral localization does not necessarily imply improved sample diversity or sampling efficiency. Instead, PMLD provides a principled way to incorporate MUON-inspired spectral geometry into posterior design, with predictive robustness as one demonstrable benefit.

For APMLD, we agree that other initialization strategies may yield similar finite-time bounds, including additional ULA iterations. Our motivation for MUON stems from its effectiveness in matrix-parameterized neural networks and its distinct optimization geometry. Our comparison with Adam+ULA specifically investigates whether MUON provides benefits beyond generic optimization-based initialization. While our theory does not establish MUON's universal superiority, it characterizes how improved initialization can reduce transient sampling error, and our experiments investigate whether MUON realizes this advantage.

Experimental evaluation. We agree that evaluating NLL and ECE under distribution shifts would provide a more comprehensive assessment of Bayesian predictive uncertainty. OOD uncertainty evaluation is an important complement to our current experiments on predictive performance, calibration, robustness, and generative modeling, and we intend to investigate it in future work.

Q1. Combining PMLD and APMLD. Indeed, we have explored this idea, and the APMLD framework actually allows such a combination: MUON initialization followed by PMLD sampling, which is reflected in the flexibility of poetential function specification. Our current presentation deliberately separates the two routes to distinguish their respective mechanisms. We agree that combining them is promising and plan to include a systematic experimental evaluation in future work.

Q2. ULA versus SGMCMC. We use ULA as the most direct example to isolate the effects of our proposed methods and simplify theoretical analysis. Importantly, our large-scale diffusion experiments already employ SGLD, a widely used SGMCMC method. Our empirical evaluation therefore already investigates the applicability of MUON-based initialization to practical, scalable stochastic-gradient sampling.

Thank you for your clarification. However, I still have some concerns unresolved.

I understand that PMLD is intended as a posterior design contribution rather than a sampling efficiency improvement. However, PMLD essentially applies standard ULA with a spectral prior. Proposition 4.1 establishes spectral localization, which is an expected consequence of this prior. My main question is why introducing a spectral prior constitutes an improvement in Bayesian deep learning. The observed benefits seem to arise primarily from regularization improving individual model predictions, rather than from better Bayesian inference.

The authors acknowledge that Proposition 4.2 does not establish MUON's superiority over other initialization strategies. The benefit of better initialization is already well understood in Langevin dynamics. Although APMLD improves predictive performance over Adam+ULA, this does not necessarily indicate better posterior approximation. More direct evidence of improved sampling quality would strengthen the contribution.

I believe OOD uncertainty evaluation should be included in the current paper rather than deferred to future work. Since the proposed methods aim to improve Bayesian deep learning, demonstrating improved uncertainty quantification under distribution shifts is particularly important. Although OOD metrics do not directly measure posterior approximation quality, they would provide valuable evidence of the practical benefits of the proposed methods for Bayesian inference.

We thank the reviewer for their follow-up comments and for further clarifying their concerns.

We understand the distinction between improvements in predictive performance and those in Bayesian inference. For PMLD, our intention is to incorporate spectral structure into the posterior through prior design, rather than to claim a more accurate approximation of the original, unregularized posterior. We recognize, however, that our current empirical results do not fully establish the extent to which these approaches improve posterior exploration or uncertainty quantification.

We also appreciate the suggestion regarding OOD uncertainty evaluation. We agree that such experiments would provide valuable additional evidence of the practical utility of our methods, particularly in distinguishing improvements in predictive performance from improvements in uncertainty quantification.

We appreciate the reviewer's thoughtful assessment and will take these concerns into account as we continue exploring this research direction.

Reviewer_LMo3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks how MUON’s matrix/spectral geometry can be transferred from optimization to Bayesian posterior sampling. The central observation is that one cannot simply replace the gradient drift in Langevin dynamics with a MUON/orthogonalized direction while keeping isotropic Brownian noise, because this generally changes the posterior distribution. The authors propose two work arounds, PMLD and APMLD. PMLD modifies the target distribution by introducing a smooth spectral-norm prior. Sampling is then ordinary Langevin/ULA from this modified posterior. APMLD keeps the desired posterior fixed but modifies the initialization. it first runs a finite number of MUON optimization steps, chooses an iterate with small nuclear-norm gradient, adds Gaussian noise, and then switches to ordinary Langevin sampling.

Critical strengths
  • They show that an explicitly chosen spectral prior produces posterior spectral localization and associated Lipschitz/robustness bounds. This is a nice result.
  • The experimental results are interesting.
  • The core hypothesis that MUON may be unusually good at generating finite-horizon warm starts for Langevin-type samplers because of how it redistributes residual error across curvature directions is potentially interesting.
Critical weaknesses

The primary weakness of this paper is that there is a very large theory-experiment gap where the experiments are seemingly in a very different regime than the theory. The theory for APMLD and finite-time sampling makes some strong simplifying assumptions. The ones of note are:

  • globally log-concave / strongly convex posteriors
  • globally smooth potentials
  • exact polar-factor MUON
  • full gradients
  • ordinary isotropic Langevin noise

The experiments instead use:

  • highly nonconvex neural networks
  • approximate Newton–Schulz MUON
  • finite practical step sizes
  • and, in the diffusion experiments, SGLD rather than the full-gradient ULA analyzed theoretically.

While some simplifying assumptions are okay and some theory-experiment gap can also be acceptable, there must be some reason to assume the gap is not that large but in this case the gap is rather large between the two. This does not invalidate the experiments themselves but makes it much less clear how the theory actually explains the observations.

Questions for authors

How should the theory be interpreted for the diffusion experiments, where the baseline is SGLD rather than the full-gradient ULA analyzed in the paper?

Why is MUON the right warm-start optimizer, rather than just one of many reasonable choices?

Can the robustness result be evaluated in the norm actually used in the experiments?

1 comment

We thank the reviewer for the thoughtful comments. We acknowledge the theory–experiment gap and would like to clarify the scope and implications of our theoretical contributions.

Theory–experiment gap. Our work primarily aims to develop a principled methodology for translating MUON's matrix-geometric insights into Bayesian deep learning, rather than to resolve the longstanding gap between log-concave sampling theory and nonconvex deep learning. Our theoretical analysis uses tractable settings to isolate the mechanisms underlying our proposed methods.

Specifically, our PMLD analysis establishes how spectral regularization shapes the posterior and controls spectral violations, without requiring strong convexity for the posterior localization result. For APMLD, our analysis characterizes how MUON initialization influences the finite-time error of subsequent Langevin sampling. The objective is not to establish theoretically superior convergence rates over existing samplers, but to provide a mathematical foundation for our constructions and identify mechanisms that may benefit sampling. Our experiments investigate whether these benefits persist in realistic deep learning settings.

Q1. SGLD and practical implementations. We agree that our full-gradient ULA guarantees do not directly apply to the diffusion experiments using SGLD. Nevertheless, the central principle of APMLD—improving initialization while retaining the subsequent Langevin sampling mechanism—extends naturally to stochastic-gradient implementations. Similarly, approximate Newton–Schulz MUON affects the initialization rather than the subsequent sampling kernel. We acknowledge that accounting for stochastic gradient variance and numerical approximation errors is necessary for extending our quantitative guarantees to these implementations, and we leave this to future work.

Q2. Why MUON? MUON is not the only viable warm-start optimizer. Our motivation stems from its demonstrated effectiveness in large neural networks and its exploitation of matrix geometry through singular-value transformations. Our comparison with Adam-based warm initialization is intended to distinguish MUON-specific benefits from those of optimization-based initialization more generally. We do not claim universal superiority, but seek to establish MUON as a promising matrix-aware approach to sampling initialization.

Q3. Robustness norm. Our spectral analysis naturally yields an ℓ2\ell_2 certificate. An ℓ2\ell_2 certified radius rr implies an ℓ∞\ell_\infty certified radius r/dr/\sqrt{d} for dd-dimensional inputs. We acknowledge that this conversion can be conservative, and that evaluating certified robustness directly in the experimental norm would strengthen the connection between theory and experiments.

Comments on the paper

2 comments

Dear Reviewers, please check the authors' rebuttal and join the discussion to clarify any open questions before the discussion phase ends on Nov 18 AoE. Thank you!

Dear reviewers, the author–reviewer discussion deadline is approaching: November 18 (AoE).

LMo3, you raised a gap between the APMLD theory and the experiments, particularly full-gradient ULA versus SGLD in the diffusion experiments. You also asked why MUON was chosen for the warm start and whether the robustness result can be evaluated in the norm used in the experiments.

Do the authors’ responses adequately address these concerns? If any remain, please let us know before the discussion deadline.

xhGf, thank you for following up. Please feel free to share any additional comments before the deadline.

QNwY, you asked how PMLD’s prior differs from alternatives such as heavy-tailed and functional priors, and what distinguishes APMLD from existing optimizer warm starts. You also asked how the approach relates to geometry-aware samplers with position-dependent Riemannian metrics.

Do the authors’ responses adequately address these concerns? If any remain, please let us know before the discussion deadline.

6xpg, thank you for following up. Please feel free to share any additional comments before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.