acceptodds
Withdrawn from ICLR 2027

When Stability Binds: High-Accuracy Stochastic-Gradient Langevin Sampling under Finite Budgets

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Superlinear drifts can make explicit Langevin samplers unstable. But stabilization is not free: once it becomes necessary, the stabilizer itself can become a major source of sampling error. This creates a two-stage problem. A sampler must first remain stable, and its stabilization must then disturb the intended dynamics as little as possible. We propose Relative-Growth Localized Taming (RELTA), an overdamped stochastic-gradient Langevin sampler with localized quadratic stabilization. It acts strongly in the tails while remaining inactive throughout the localized typical region. Despite using overdamped dynamics, stochastic gradients, and no Metropolis correction, RELTA achieves high finite-budget sampling accuracy across controlled, correlated, and real-data targets, and can outperform carefully tuned kinetic and exact-gradient methods under finite gradient-access budgets. We quantify how strongly stabilization perturbs the sampling dynamics as the step size decreases, and show how this perturbation contributes to stationary bias. Our results suggest a simple principle: when stability binds, stabilize first, but intervene as little as possible.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Langevin sampling, stochastic-gradient MCMC, superlinear drifts, taming, finite-budget sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BwyXOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies explicit overdamped stochastic-gradient Langevin samplers for targets πβ∝e−βU\pi_\beta \propto e^{-\beta U} with superlinearly growing gradients, where untamed SGLD can diverge. It considers drift-tamed updates Xn+1=Xn−λH(Xn,Zn+1)/Dλ(Xn)+2λ/β ξn+1X_{n+1} = X_n - \lambda H(X_n,Z_{n+1})/D_\lambda(X_n) + \sqrt{2\lambda/\beta}\,\xi_{n+1}. It argues that a stabiliser has two separable properties: the far-tail strength needed for Lyapunov stability, and the order at which it perturbs the dynamics on bounded-score states, which the authors call the sampling order.

The family Dλ,p=(1+(KλSs0)p)1/pD_{\lambda,p} = (1 + (K\lambda S_{s_0})^p)^{1/p} uses the localised score Ss0(x)S_{s_0}(x). Every member has the same far-tail scale KλSs0K\lambda S_{s_0}, but its relative perturbation is O(λp)O(\lambda^p). RELTA is the member p=2p = 2. It is attributed to prior work (Zhou and Chen, 2026); the contribution in this paper is the analysis and evaluation of the sampler.

The theoretical results are:

Theorem 1: the Lyapunov drift inequality, uniform second moments and uniqueness of the invariant law hold uniformly in pp. Proposition 2: the invariant law converges weakly and in W1W_1 to πβ\pi_\beta as λ→0\lambda \to 0. Theorem 2: the leading O(λ)O(\lambda) stationary weak-bias coefficient acquires an extra stabiliser term when p=1p = 1, but equals the stochastic-gradient Euler coefficient for every p>1p > 1.

On the numerics, the main findings are:

  • RELTA has much lower W1W_1 than TUSLA, mTULA and kTULA in a large-step stress band.
  • It beats tuned SGHMC at smaller step sizes. At λ=0.12\lambda = 0.12, SGHMC is more accurate but loses tail stability, and tamed SGHMC loses accuracy.
  • It attains lower W1W_1 than pSGLD and stability-constrained BAOAB at 2,000–10,000 full-gradient-equivalent (FGE) budgets, and than MALA, HMC and NUTS at 5,000–10,000 FGE. Accuracy-tuned BAOAB is more accurate at 10,000 FGE but fails on tail recovery.
Critical strengths

The paper gives a simple, well-motivated and transferable design principle. Separating how strongly a stabiliser acts in the tail from how much it perturbs ordinary states is a useful way to think about taming. The pp-family makes the separation concrete. The proof of Theorem 1 uses only D≥KλSD \ge K\lambda S and D≤1+KλSD \le 1 + K\lambda S, so stability is uniform in pp. Theorem 2 then shows that any p>1p > 1 removes the stabiliser from the leading stationary bias.

This gives a clean criterion for comparing stabilisers. For example, it suggests why globally active taming with relative perturbation of order λ1/2\lambda^{1/2} (TUSLA) or λ\lambda (mTULA). The principle should apply beyond this sampler, including to tamed kinetic schemes. For me, this is the main finding in the paper.

Disjoint tuning and held-out targets with frozen hyperparameters, exact references, and explicit FGE accounting are above the norm for this literature. So are the paired comparisons, the explicit stability criteria and the per-seed outputs. The paper reports where RELTA struggles (SGHMC at λ=0.12\lambda = 0.12, accuracy-tuned BAOAB at 10,000 FGE) and states plainly that the stabiliser is inactive in the regression experiment. The pp ablation shares oracle and noise streams across pp and shows 60/60 paired wins for p=2p = 2 over p=1p = 1. This provides convincing evidence for the mechanism in the regime where the stabiliser is active.

Critical weaknesses

My main criticism of this paper is that it is written for specialists in tamed Langevin algorithms and is hard to follow for the broader ICLR audience. The introduction uses the tamed update, HH, ZZ, DλD_\lambda, hλh_\lambda, KK, Ss0S_{s_0} and "sampling order" before defining them. The target πβ\pi_\beta and h=∇Uh = \nabla U appear only in Section 3.

Equation (1) is never identified as an Euler–Maruyama discretisation of the overdamped Langevin diffusion dXt=−∇U(Xt) dt+2/β dWtdX_t = -\nabla U(X_t)\,dt + \sqrt{2/\beta}\,dW_t, whose invariant law is πβ\pi_\beta. Nor is it explained why one would sample with such a scheme. II, QQ and VV are introduced without interpretation: they are the radial dissipativity of the drift, the second moment of the stochastic gradient, and a quadratic Lyapunov function. The relative-growth condition (3) and the lower bound (4) are each justified by a single clause. The reader only sees why DD must scale like KλSK\lambda S after working through the proof of Theorem 1 in the appendix.

Terms such as "strict stability", "stability binds", "bounded-score states" and FGE are used informally and some of the notation is overloaded:

  • hh denotes both ∇U\nabla U and the step size in Sections 5.3–5.5 and Appendix B.

  • KK is both a constant and a compact set.

  • RR is both the Markov kernel and a radius.

In my opinion, the new contribution is quite narrow, and the main experiments are not sufficiently convincing. RELTA itself is attributed to Zhou and Chen (2026), whose paper already claims polynomial moment stability and first-order stationary accuracy in both W1 and W2. The new content therefore appears to be the pp-family viewpoint, Theorem 2 and the empirical study.

If I've followed this correctly, then Theorem 2 seems to follow fairly directly, via a standard Talay–Tubaro-type expansion, once Dλ,p−1=1+O(λp)D_{\lambda,p}^{-1} = 1 + O(\lambda^p) on bounded-score sets. The genuinely difficult parts are assumed rather than verified, even for the quartic benchmark: weighted generator consistency (A3(c)), and uniform higher-order moments and Poisson-equation regularity (A4). The theorem also seems to show only that p>1p > 1 matches unstabilised Euler, not that it improves on it. It does not establish that CSC_S is non-negligible or of a particular sign, and it concerns smooth observables, whereas every experiment reports W1W_1.

Empirically, only the pp ablation tests the new ingredient. The finite-budget comparisons with pSGLD, BAOAB and corrected MCMC (Tables 4, 5 and 8) run RELTA at λ∈[0.006,0.04]\lambda \in [0.006, 0.04]. At those step sizes the stabiliser presumably rarely acts, so should RELTA not behave essentially like SGLD? These tables may therefore mainly measure effective sample size per FGE, which favours cheap mini-batch steps on a 3,200-component finite-sum target. That is a generic SG-MCMC advantage, not a consequence of find a better stabiliser.

Questions for authors
  • Please state precisely which results are new relative to Zhou and Chen (2026). Does the quadratic denominator (8) already appear there, and do Theorem 1 and Proposition 2 overlap with its stability and first-order accuracy results?

  • Why is the lower bound (4) needed? I can reconstruct the argument from the proof of Theorem 1: D≥KλSD \ge K\lambda S keeps the positive λ2Q/D2\lambda^2 Q/D^2 term below a fraction CQ/KC_Q/K of the inward drift term. This is the crux of the method, however, and should be explained, with intuition, where (4) is introduced.

  • What fraction of iterations at stationarity have Ss0(Xn)>0S_{s_0}(X_n) > 0, and what is the typical value of D−1D - 1 when the stabiliser is active? If SGLD matches RELTA’s typical-region accuracy, the advantage over pSGLD, BAOAB and corrected MCMC should be attributed to SGLD under FGE accounting, with RELTA contributing tail safety.

Reviewer_K414OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies explicit stochastic-gradient Langevin dynanics sampling (SGL) with outlier drifts in areas with exceedingly large forces, where taming is needed for stability but biases the chain. It analyzes RELTA, a drift-tamed SGLD from prior work with a localized score S = [‖x‖^{2r} − s0]+ and denominator D = √(1 + (K \lambda S)^2). It embeds RELTA in a family D{\lambda ,p} = (1 + (K\lambda S)^p)^{1/p}, whose members share a tail scale but perturb bounded-score states at O(λ^p). The theory gives a Lyapunov bound uniform in p (Thm 1), stationary consistency (Prop 2), and a Talay–Tubaro-type result: p = 1 changes the O(λ) stationary-bias coefficient, while p > 1 does not (Thm 2). On a d = 50 product-form quartic benchmark, RELTA beats unlocalized tamed schemes and SGHMC at moderate λ and beats pSGLD, BAOAB, HMC, NUTS and MALA at matched full-gradient-equivalent (FGE) budgets; it matches SGLD on a Diabetes BNN, and a p ∈ {1, 2, 4} ablation favours larger p.

Critical strengths
  1. The authors present a usable result. Prop 1 and Thm 2 state precisely why p = 1 taming changes the leading stationary bias and p > 1 does not. The proofs are short and correct given their assumptions. This matters because it is a usable design rule: tame only where the score is large, and make the local perturbation higher-order.
  2. The protocol is clean. Tuning and held-out targets are disjoint, configurations are frozen, seeds are paired, and raw outputs are supplied. The BNN result, where the stabiliser never binds, is reported honestly (B.10). This matters because it makes the weaknesses below checkable from the paper itself.
Critical weaknesses
  1. The method itself is fairly simple given the literature on second order moment normalization (Adam being the most famous example). RELTA multiplies the SGLD drift by 1/D, with D = √(1 + (KλS)²). Inside the radius set by s0 it is plain SGLD. Outside, it is a smooth, state-dependent step-size cap that shrinks the drift where it would make an explicit Euler step unstable. This is the standard taming idea, with a threshold and a squared denominator.
  2. The experiments are somewhat limited and limited to a GMM and a BNN regression. This is too little given that 50% of the experiments are toy examples.
  3. Given that 50% are toy examples, I would like to see a comparison between an analytically sampleable distribution and all listed samplers. The Wasserstein1 distances seem very small and I have seen similar values with analytically sampleable distributions where the sampling variance simply introduces similar values. The quartic sum example can probably be rewritten into a proper Gaussian Mixture Model or a comparable GMM can be constructed.
Questions for authors

See Weaknesses above.

Reviewer_mykJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies RELTA (Relative-Growth Localized Taming), a stabilization scheme designed for nonconvex stochastic-gradient Langevin learning introduced by the prior work RELTA-SGLD. In particular, the authors study its sampling behavior and stabilization-induced stationary error. The core of the method: modify Langevin sampling so that it remains numerically stable without unnecessarily distorting the distribution being sampled. Taming the gradient with a denominator that reduces excessively large drift updates solves one problem but introduces another: by not following the original gradient, the resulting sampler exhibits different dynamics and in general has a different stationary distribution. “When Stability Binds” calls attention to this stability-accuracy tradeoff, referring to a regime where satisfying the stability constraint trumps preserving the original Langevin dynamics. The paper argues that a good stabilizer should operate strongly in the tails but interfere as little as possible with ordinary states.

Critical strengths
  • The paper compares RELTA against 14 baselines across controlled, correlated, and real-data settings, providing evidence of favorable finite-budget sampling accuracy and numerical stability.
  • Using 14 baselines is great (positive), although I would make it clearer that these 14 methods are evaluated across different experiments, not together under one common benchmark protocol.
  • The paper addresses a meaningful tradeoff between numerical stability and sampling accuracy, providing a principled explanation of how stabilization can introduce additional sampling bias.
  • The authors provide an array of theoretical results.
Critical weaknesses
  • Last sentence of the abstract: a TLDR-like recipe message could have a place here, but I think that sentence is vague and broad enough not to be useful to the reader-practitioner. If the sentiment is to broaden the message to a larger audience, I think it can work.
  • L43: “replacing h”. What hh? It comes out of nowhere, it is unclear what the authors mean there and then.
  • Is “typical-region” common nomenclature? By “typical-region dynamics,” the authors likely mean the dynamics in the bulk (non-tail region) of the distribution, correct?
  • L53: What is pp? The authors write: “For the denominator family introduced below,”. Why not say that _after_ introducing the family? It will make the exposition more linear and therefore easier to go through.
  • L61: “Here” repetition.
  • Baselines: consider having a dedicated named paragraph in the experiments section (like the authors have already for “Benchmark” and “Evaluation”). It would be convenient to see at a glance what are those 14 baselines the authors mention in the introduction.
  • About the introduction: I think the authors should make it clearer what they do in the paper, i.e. carry out an in-depth study of RELTA and, based on their findings, formulate a recipe for how to stabilize Langevin sampling.
  • The introduction needs a rewrite to make it easier to read and more sensible in how it lays out its arguments. For how simple the projected plan is, it show read far more smoothly.
  • Section 3 before 3.1: the authors should fill the gaps between equations and introduced notations to inform the reader on what those are. The paper should read more fluidly.
  • L138: “The family introduced below”. Unclear.
  • I find the paper difficult to follow: the authors introduce several concepts without adequately motivating them or explaining their relationship.
  • The authors occasionally blur the distinction between inactive states (S=0S=0, exactly no intervention) from active bounded-score states (S>0S>0, intervention of order O(λ2)O(\lambda^2)) by describing both as typical-region dynamics.
Questions for authors
  • The authors define the sampling order with name ρ\rho at L52 and pp at L160. Can the authors clarify?
  • (Real-data result) On Bayesian neural regression, RELTA and matched-step SGLD produce identical predictions. The authors explicitly acknowledge that RELTA’s stabilizer never activates on the ordinary test trajectories. Couldn’t the authors find a real-data scenario where a harmful drift would need correction, i.e. where the stabilizer would kick in? If not, what does it say about the problem tackled?
  • Assumption A4 is substantial: it includes uniform higher-moment bounds and weighted expansion remainder conditions. Have those conditions been verified for the experimental targets and parameter choices?
  • It should be made crystal-clear what the contributions are compared to the paper that introduces RELTA, i.e. the RELTA-SGLD paper, whose contributions include the observation that quadratic taming perturbs the drift by O(λ2)O(\lambda^2) on bounded-score regions (Lemma 4.3). In this paper being reviewed, Proposition 1 generalizes the argument to the family with pp / 1/p1/p exponents, etc. The authors then experimentally compare p=1,2,4p=1,2,4, finding that larger pp generally improves sampling accuracy without sacrificing their measured stability. So the generalization to pp is novel, along with the accompanying theoretical results and sampling experiments; correct?
  • Section 5.6 shows that increasing pp from 2 to 4 improves sampling accuracy while preserving the reported stability. Moreover, as p→∞p\to\infty, the proposed denominator converges to Dλ,∞(x)=max⁡(1,KλSs0(x))D_{\lambda,\infty}(x)=\max(1,K\lambda S_{s_0}(x)), which appears to retain the same far-tail scaling and denominator bounds used in the Lyapunov stability proof. So here is my question: Given that p=4p=4 empirically improves over p=2p=2 in 59/60 comparisons without sacrificing stability, is there any reason to prefer the original quadratic RELTA denominator (p=2p=2) beyond its simplicity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.