- Rating
- 2
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies explicit overdamped stochastic-gradient Langevin samplers for targets with superlinearly growing gradients, where untamed SGLD can diverge. It considers drift-tamed updates . It argues that a stabiliser has two separable properties: the far-tail strength needed for Lyapunov stability, and the order at which it perturbs the dynamics on bounded-score states, which the authors call the sampling order.
The family uses the localised score . Every member has the same far-tail scale , but its relative perturbation is . RELTA is the member . It is attributed to prior work (Zhou and Chen, 2026); the contribution in this paper is the analysis and evaluation of the sampler.
The theoretical results are:
Theorem 1: the Lyapunov drift inequality, uniform second moments and uniqueness of the invariant law hold uniformly in . Proposition 2: the invariant law converges weakly and in to as . Theorem 2: the leading stationary weak-bias coefficient acquires an extra stabiliser term when , but equals the stochastic-gradient Euler coefficient for every .
On the numerics, the main findings are:
- RELTA has much lower than TUSLA, mTULA and kTULA in a large-step stress band.
- It beats tuned SGHMC at smaller step sizes. At , SGHMC is more accurate but loses tail stability, and tamed SGHMC loses accuracy.
- It attains lower than pSGLD and stability-constrained BAOAB at 2,000–10,000 full-gradient-equivalent (FGE) budgets, and than MALA, HMC and NUTS at 5,000–10,000 FGE. Accuracy-tuned BAOAB is more accurate at 10,000 FGE but fails on tail recovery.
Critical strengths
The paper gives a simple, well-motivated and transferable design principle. Separating how strongly a stabiliser acts in the tail from how much it perturbs ordinary states is a useful way to think about taming. The -family makes the separation concrete. The proof of Theorem 1 uses only and , so stability is uniform in . Theorem 2 then shows that any removes the stabiliser from the leading stationary bias.
This gives a clean criterion for comparing stabilisers. For example, it suggests why globally active taming with relative perturbation of order (TUSLA) or (mTULA). The principle should apply beyond this sampler, including to tamed kinetic schemes. For me, this is the main finding in the paper.
Disjoint tuning and held-out targets with frozen hyperparameters, exact references, and explicit FGE accounting are above the norm for this literature. So are the paired comparisons, the explicit stability criteria and the per-seed outputs. The paper reports where RELTA struggles (SGHMC at , accuracy-tuned BAOAB at 10,000 FGE) and states plainly that the stabiliser is inactive in the regression experiment. The ablation shares oracle and noise streams across and shows 60/60 paired wins for over . This provides convincing evidence for the mechanism in the regime where the stabiliser is active.
Critical weaknesses
My main criticism of this paper is that it is written for specialists in tamed Langevin algorithms and is hard to follow for the broader ICLR audience. The introduction uses the tamed update, , , , , , and "sampling order" before defining them. The target and appear only in Section 3.
Equation (1) is never identified as an Euler–Maruyama discretisation of the overdamped Langevin diffusion , whose invariant law is . Nor is it explained why one would sample with such a scheme. , and are introduced without interpretation: they are the radial dissipativity of the drift, the second moment of the stochastic gradient, and a quadratic Lyapunov function. The relative-growth condition (3) and the lower bound (4) are each justified by a single clause. The reader only sees why must scale like after working through the proof of Theorem 1 in the appendix.
Terms such as "strict stability", "stability binds", "bounded-score states" and FGE are used informally and some of the notation is overloaded:
-
denotes both and the step size in Sections 5.3–5.5 and Appendix B.
-
is both a constant and a compact set.
-
is both the Markov kernel and a radius.
In my opinion, the new contribution is quite narrow, and the main experiments are not sufficiently convincing. RELTA itself is attributed to Zhou and Chen (2026), whose paper already claims polynomial moment stability and first-order stationary accuracy in both W1 and W2. The new content therefore appears to be the -family viewpoint, Theorem 2 and the empirical study.
If I've followed this correctly, then Theorem 2 seems to follow fairly directly, via a standard Talay–Tubaro-type expansion, once on bounded-score sets. The genuinely difficult parts are assumed rather than verified, even for the quartic benchmark: weighted generator consistency (A3(c)), and uniform higher-order moments and Poisson-equation regularity (A4). The theorem also seems to show only that matches unstabilised Euler, not that it improves on it. It does not establish that is non-negligible or of a particular sign, and it concerns smooth observables, whereas every experiment reports .
Empirically, only the ablation tests the new ingredient. The finite-budget comparisons with pSGLD, BAOAB and corrected MCMC (Tables 4, 5 and 8) run RELTA at . At those step sizes the stabiliser presumably rarely acts, so should RELTA not behave essentially like SGLD? These tables may therefore mainly measure effective sample size per FGE, which favours cheap mini-batch steps on a 3,200-component finite-sum target. That is a generic SG-MCMC advantage, not a consequence of find a better stabiliser.
Questions for authors
-
Please state precisely which results are new relative to Zhou and Chen (2026). Does the quadratic denominator (8) already appear there, and do Theorem 1 and Proposition 2 overlap with its stability and first-order accuracy results?
-
Why is the lower bound (4) needed? I can reconstruct the argument from the proof of Theorem 1: keeps the positive term below a fraction of the inward drift term. This is the crux of the method, however, and should be explained, with intuition, where (4) is introduced.
-
What fraction of iterations at stationarity have , and what is the typical value of when the stabiliser is active? If SGLD matches RELTA’s typical-region accuracy, the advantage over pSGLD, BAOAB and corrected MCMC should be attributed to SGLD under FGE accounting, with RELTA contributing tail safety.