When Stability Binds: High-Accuracy Stochastic-Gradient Langevin Sampling under Finite Budgets

Superlinear drifts can make explicit Langevin samplers unstable. But stabilization is not free: once it becomes necessary, the stabilizer itself can become a major source of sampling error. This creates a two-stage problem. A sampler must first remain stable, and its stabilization must then disturb the intended dynamics as little as possible. We propose Relative-Growth Localized Taming (RELTA), an overdamped stochastic-gradient Langevin sampler with localized quadratic stabilization. It acts strongly in the tails while remaining inactive throughout the localized typical region. Despite using overdamped dynamics, stochastic gradients, and no Metropolis correction, RELTA achieves high finite-budget sampling accuracy across controlled, correlated, and real-data targets, and can outperform carefully tuned kinetic and exact-gradient methods under finite gradient-access budgets. We quantify how strongly stabilization perturbs the sampling dynamics as the step size decreases, and show how this perturbation contributes to stationary bias. Our results suggest a simple principle: when stability binds, stabilize first, but intervene as little as possible.