acceptodds
Under review as a conference paper at ICLR 2027

TamedMALA: Critically Scaled Score Taming for Robust Langevin Sampling

Abstract

The Metropolis-adjusted Langevin algorithm (MALA) can be highly efficient when the score provides a reliable local proposal direction, but excessively large scores can instead produce proposals that are almost always rejected. We introduce a proposal-scale-coupled family of score tamers, combined with an exact Metropolis–Hastings correction. For fixed-dimensional product blocks, we derive the leading Hastings expansion and show a scaling transition governed by the coupling exponent : for , taming worsens the proposal-variance scaling, whereas for the classical MALA scaling is preserved. At the critical coupling , we obtain an explicit nonnegative asymptotic efficiency penalty on regular targets, clarifying that taming should act as a robustness safeguard rather than improve well-behaved MALA. We further show that uniquely balances saturated deterministic drift with proposal noise. The resulting coordinate-wise TamedMALA achieves essentially MALA-level stationary efficiency across six independent tune-and-evaluate repetitions, while maintaining reliable recovery from deliberately pathological score states where MALA exhibits rejection-induced sticking. A short-chain tuner selects the additional taming scale automatically with stable stationary performance.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Markov chain Monte Carlo, MCMC, Metropolis-adjusted Langevin algorithm, MALA, Langevin Monte Carlo, robust sampling, gradient taming, optimal scaling, Bayesian inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MT5GOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes tamed MALA, a version of MALA where the score is tamed to avoid instabilities. Here, the score is tamed by block (or coordinate-wise for block size 11) instead of a global shrinking. A high-dimensional scaling result is given under standard i.i.d. and regularity assumptions. The experiments show that tamedMALA is robust.

Critical strengths
  1. It's a simple method that works. The experiments show that tamedMALA suffers negligible loss of performance compared to default MALA (unlike other robust baselines), but still recovers well outside stationary.

  2. The presentation is generally readable and not dense, unlike other papers written with AI assistance.

Critical weaknesses
  1. The novelty of the method seems limited, and the theoretical results are simple with no real technical innovation. In particular, no theoretical result is available for outside-stationary or light-tailed scenarios. The paper also studies exclusively MALA and leaves out unadjusted versions.

  2. No description of the stress-test targets is provided, making it difficult to see what makes the algorithms work or fail in each example, and to judge whether the test set is sufficiently exhaustive.

Questions for authors

I'm inclined to accept simple papers that work. My main question is already stated in the Critical Weakness section, namely:

  1. Please provide a description of the stress-test targets, in particular, specific features that make each target difficult. From your description in Section 5.5 it seems like those targets are all perturbed only in one dimension. I believe this limits the power of your conclusions.

  2. What happens for the block size m>1m>1? What is the rationale for m=1m=1?

Minor suggestions: please avoid putting heavy formula in the abstract with unexplained terms. Additionally formulas (5) and (6) should be moved after Eq. (7) for a smoother reading.

2 comments
AuthorsOpenReview ↗

Theoretical Scope, Robustness Experiments, and Blockwise Taming

We thank the reviewer for the constructive criticisms. In particular, we have addressed the following points according to your comment:

1. Theoretical scope and contributions

Our core contribution is to characterize the effect of proposal-scale-dependent taming with an MH step. The principal theoretical step is the forward/reverse Hastings expansion in Equation (8), which identifies the taming-induced terms in the MALA error. This yields the scaling regime in Theorem 4.2, the sharpness result in Corollary 4.3, and the taming penalty in Proposition 4.4.

We have expanded Section 2 to discuss TULA and related unadjusted schemes. Because our analysis concerns an MH-adjusted algorithm with exact target invariance, we focused on stationary acceptance and empirical pathological-state recovery.

2. Added details on stress-test targets

We have expanded Section 5.5 to include detailed descriptions of the two stress-test targets. For the sextic product target, V(x)=∑ixi6/6V(x)=\sum_i x_i^6/6, the score grows as −xi5-x_i^5. An extreme tail state produces a raw Langevin drift at risk of overshooting the typical region. For the planar-singular target, V(x)=∣x∣2/2−2log⁡∣x1∣V(x)=|x|^2/2-2\log|x_1|, the pathological score instead arises near the finite-position density zero at x1=0x_1=0, where g1=2/x1−x1g_1=2/x_1-x_1 diverges. In both experiments, the reviewer is correct that only x1x_1 is displaced from stationarity. However, because acceptance is determined by a single joint MH ratio, one extreme score can cause rejection of the full high-dimensional proposal.

3. Coordinate-wise taming and the case for m>1m>1

The theoretical analysis is formulated for fixed-dimensional blocks and is not restricted to m=1m=1. Our empirical default is m=1m=1 because it prevents an extreme score component from unnecessarily attenuating otherwise moderate gradients.

To illustrate this mechanism, we rewrite the blockwise taming transformation from Equation (2) fixing p=2p=2 and γ=1\gamma=1 as

\qquad a_b:=\left(1+\frac{\delta\\|g^{(b)}\\|^2}{c^2}\right)^{-1/2},$$ Here, $a_b$ denotes the scalar attenuation factor shared by all coordinates in block $b$. If one component $g_j$ becomes arbitrarily large while another component $g_i$ remains moderate, then $a_b\to0$, suppressing the drift of both components. Coordinate-wise taming avoids this within-block cross-attenuation because each coordinate has its own multiplier. The joint MH acceptance decision nevertheless continues to couple coordinates. This provides a mechanism-level justification for choosing $m=1$, particularly for localized score pathologies. We do not claim that $m=1$ is universally optimal. Other block sizes may be appropriate when coordinates have a meaningful physical grouping. For example, in three-dimensional electronic variational Monte Carlo, $m=3$ provides a natural particle-wise grouping that preserves rotational equivariance of the tamed drift. Our stationary experiments also include a globally tamed variant, but intermediate block sizes have not been systematically evaluated. Their empirical behaviors remain an open direction for future investigation. **4. Minor fixes** We have re-written the section around equations (5), (6), and (7). Equation (5) and (6) are now grouped under 'settings and notation' section.
Reviewer_MT5GOpenReview ↗

Acceptance

I'd like to thank the authors for the clarifications. I keep my original positive score.

Reviewer_z7EuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a step-size-coupled score-taming family for the Metropolis-adjusted Langevin algorithm (MALA), motivated by rejection-induced sticking under pathological score magnitudes. For regular product targets with fixed-dimensional blocks, it derives stationary high-dimensional Hastings and acceptance limits, identifies the coupling regime that preserves the classical MALA scaling, and quantifies the asymptotic local-efficiency cost of taming. The default configuration bounds the deterministic proposal displacement at the scale of the proposal noise. Experiments evaluate stationary sampling efficiency and recovery from deliberately pathological initializations. The reported results show approximately MALA-level stationary efficiency alongside improved recovery over MALA in the tested settings. The recovery results provide empirical evidence of robustness, rather than a general stability guarantee.

Critical strengths

Useful theoretical analysis of the proposed taming family.

The paper characterizes the stationary scaling and acceptance behavior and derives an explicit asymptotic local-efficiency penalty under its stated assumptions. These results help explain how the taming parameters affect sampling efficiency and provide a theoretical basis for the proposed design.

Empirical evidence of robustness while retaining competitive stationary efficiency.

The experiments show that the proposed method retains approximately MALA-level stationary efficiency while recovering reliably from tested pathological states where MALA exhibits sticking. The planar-singular experiment also shows improved recovery over tuned MALTA with similar stationary efficiency. Separating tuning from evaluation and keeping the selected parameters fixed during recovery tests strengthens this evidence.

Critical weaknesses

Incomplete experimental specifications and inconsistent reporting limit independent verification.

The supplied manuscript does not fully specify several target distributions and their parameters, the diagnostic panels underlying the headline ESS metric, or the numerical tuning grids. These details are needed to reproduce and interpret the empirical comparisons. In addition, Figure 1b and Table 4 appear to report inconsistent values for the same sensitivity analysis: for example, the planar-singular result at (p=1) is approximately 0.66 in the figure but 0.756 in the table. The experimental specifications and this discrepancy should be clarified to make the numerical evidence reproducible and internally consistent.

Questions for authors

Consistency between Figure 1b and Table 4.

Figure 1b and Table 4 appear to describe the same six-repetition (p)-ablation with frozen production parameters, but their reported values differ. For example, the planar-singular result at (p=1) is approximately 0.66 in the figure versus 0.756 in the table. Could you explain the discrepancy, identify the correct results, and clarify whether resolving it affects the interpretation in Section 5.3?

Specifications needed to interpret and reproduce the stationary benchmarks.

Could you provide the exact target-specific diagnostic panels used to compute the headline (Q_{0.25}) bulk ESS per gradient evaluation, the missing target definitions and parameter values, and the numerical tuning grids—or indicate precisely where these specifications are available? These details are needed to understand what the aggregate efficiency metric captures and to independently reproduce the reported comparisons.

1 comment
AuthorsOpenReview ↗

Inconsistency betwen Figure 1b and Table 4 fixed, add details for reproducing the experiment

We thank the reviewer for the constructive criticism. In particular, we appreciate the identification of the inconsistency in the (p)-ablation results and the missing reproducibility specifications. We have made the following modifications to the manuscript according to your comments.

1. Resolving Figure 1b and Table 4 numerical consistency

We audited the codebase and traced this discrepancy to the (p)-ablation computation pipeline. This appears to be a mistake. We conducted six repetitions of the (p)-ablation but reported only the results from the first three repetitions in Figure 1b. We have modified Figure 1b to report the results from all six repetitions, yielding 0.755805±0.1233280.755805\pm0.123328 which agrees with Table 4 (now Table 6 in Appendix D), which already uses all six repetitions.

2. Added target specifications and diagnostic panels

We have expanded the experimental documentation. Section 5.1 and Table 1 now describe the main benchmark targets, and Appendix C provides additional benchmark definitions. Furthermore, Table 3 in Appendix C now lists all target-specific diagnostic panels used in the stationary benchmark. The final benchmark uses eight chains with 30,000 retained transitions each. We now report these values in Section 5.1.

3. Added numerical tuning and evaluation protocol

We have substantially expanded Appendix E. It now specifies the target-scale initialization and parameter grids, adaptive bracketing and refinement rules, parameter bounds, and evaluation budgets.

Reviewer_iE96OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The general scope of the paper is to design Metropolis-Hastings algorithms to sample approximately from a general probability distribution π\pi. A standard choice is to consider Langevin proposal that drifts the points towards regions of large mass under π\pi. It has been observed in the literature that one might gain stability in practice by scaling down this drift by the magnitude of the score of π\pi, for instance when π\pi has super linear gradients (see e.g. https://arxiv.org/abs/1710.05559).

The current paper considers a general family of tamed Langevin proposal with various parameters, and seeks theoretical results to tune these parameters to achieve optimal scaling of the MCMC algorithm.

Critical strengths

In principle, a precise tuning and comparison of different families of tamed MALA proposals, including the class discussed in the papers, would be interesting for MCMC practioners.

Critical weaknesses

The paper is badly written, and I suspect a large part of it and of the results have been produced by generative AI.

First, the paper does a poor job at introducing the topic and discussing the literature. While it is acknowledged that tamed MALA proposals are not new in the MCMC literature, the authors do not discuss precisely how their class of proposal compares to existing methods. What the authors try to achieve by introducing this class of proposal seems very unclear by reading the paper, and the manner in which a "optimal scaling" is quantified seems very vague compared to the litterature (e.g. mixing times, ...). In particular, there is a striking low number of references to the literature (12 in total), and the related work (Section 2) is a few lines long and very vague.

Second, while the paper mainly claims to advance the theoretical understanding of MALA-type algorithms, the mathematical contributions presented in section 4 are again unclear. The main contributions Theorem 4.2 and Proposition 4.3 deal with quantities that are never defined in the paper (e.g. Δd\Delta_d or Φ\Phi), and the logical implications inside these statements are not explained. The paper poorly explains why the results that are claimed answer the questions that have been raised in the introduction. In the same vein, the description of the experiments in Section 5 is very vague and it is hard to understand what is being done and shown in the results, how it compares with the literature, and why this is relevant.

Finally, I have doubts on the soudness of the proofs in Appendix A, most of the formulas are not standard and are given without justifications or references. Considering as an assumption that all integration by parts used in the proofs are true in Asssumption 4.1 seems weird.

Questions for authors

Can you formalize the claims precisely with respect to the literature and the results of Theorem 4.2 ? How does it relates for instance with guarantees in mixing times or of original tamed schemes e.g. in https://arxiv.org/abs/1710.05559 ? Can you clarify the setup of your experiments in Section 5 and your interpretation of the results for MCMC practioners ?

3 comments
AuthorsOpenReview ↗

Improvement in Positioning, Theoretical Exposition, and Experimental Documentation

We thank the reviewer for the critical review. We have substantially revised the manuscript to strengthen literature reviews, clarify motivation and theoretical contribution, improve exposition of mathematical arguments, and provide additional details for reproducibility. Main changes are as follows:

1. Improved introduction, literature review, and positioning

Score-taming and bounded drifts proposals are not new. Our paper asks a more specific question: How strongly can we tame the score while preserving the d−1/3d^{-1/3} scaling of regular MALA? We have expanded Section 2 to discuss relevant work. It now includes tamed ULA variants, metropolized taming, Barker proposal, proximal proposal, and optimal-scaling theory. We’ve also clarified how existing tamers relate to our proposed tamer family. Our principal technical contributions are:

a. Leading Hastings perturbation Equation (8) identifies intrinsic O(ε3)O(\varepsilon^3) MALA error and taming-induced O(ε2γ+1)O(\varepsilon^{2\gamma+1}) term.

b. Scaling and acceptance limits Theorem 4.2 derives limiting Hastings distribution, acceptance probability, and ESJD. Corollary 4.3 establishes the sharp proposal-variance scaling.

c. Cost of critical taming Proposition 4.4 derives a strictly positive asymptotic efficiency penalty at γ=1\gamma=1 for regular product targets.

d. The drift–noise criterion Section 4.4 shows that p=2p=2 uniquely keeps the saturated deterministic proposal displacement on the scale of the Gaussian noise at γ=1\gamma=1.

2. Improved theoretical exposition

We have substantially reorganized and expanded Appendix A to make the mathematical arguments more transparent. Specifically:

a. Lemma A.1 derives the required integration-by-parts identity under smoothness, polynomial-growth, and moment conditions of Assumption 4.1.

b. Appendix A.2–A.3 derive the gaussian forward/reverse Hastings identity and its one-block perturbative expansion, including the taming-induced term and the remainder order.

c. Lemmas A.4–A.5 establish MH swap identity and derive the leading mean–variance relationship using bounded truncation and moment estimates.

d. Appendix A.5 applies a triangular-array Lyapunov CLT and establishes the limiting Hastings distribution, acceptance probability, and normalized ESJD.

e. Appendices A.6–A.7 provide the sharpness argument and the Gaussian-moment / integration-by-parts derivation of the positive efficiency penalty.

We also rewrote section 4.2-4.3 to clarify the logical progression from Hastings expansion to the scaling transition, sharpness, limiting ESJD, and efficiency penalty.

Minor fixes: added definition for Δd\Delta_d in Equation (5), added definition of Φ\Phi in section 4.1.

3. Added details about experimental documentation and contribution

We have expanded Section 5 and Appendices C–E to document details in target distributions, diagnostic panels, parameter tuning protocols, and evaluation protocols.

Our experiments support two observations: First, through lower quartile ESS tuning, TamedMALA can achieve approximately MALA-level of stationary efficiency across the benchmark targets. Second, TamedMALA shows improved recovery in pathological initializations. Our empirical claim is that TamedMALA combines great stationary efficiency with improved finite-budget recovery in pathological configurations.

4. Mixing time & geometric ergodicity

We have audited the relationship between our sampler and existing convergence theory. Atchadé (2006, Proposition 2.1) established geometric ergodicity for MH algorithms with bounded proposal drift under super-exponential-tail and regular-contour conditions. Our tamer satisfies the globally bounded condition since ∣Tδ,c,p,γ(g(b))∣≤cδ−γ/p|T_{\delta,c,p,\gamma}(g^{(b)})|\le c\delta^{-\gamma/p} Thus, Atchadé's result applies to TamedMALA under the same target assumptions. (In particular, the sextic product target used in stress test satisfies tail and contour conditions)

We do not have a dimension-explicit mixing time guarantee for TamedMALA. We made this remark in section 2 paragraph “optimal scaling vs mixing time”.

5. Research provenance, use of AI

We’ve had the idea of this work for a considerable time. This research originated from our earlier investigations of samplers for variational monte carlo. We initially used a hard-sign-like tamer for MALA and observed unfavorable dimension scaling (d−1d^{-1} for T(g)=c⋅Sign(g)T(g)=c\cdot\text{Sign}(g)). This experience motivates the current manuscript that puts considerable focus on studying tamer that guarantees MALA-equivalent scaling behavior. As disclosed in the AI-use statement at the end of the manuscript, we used generative AI tools to assist with drafting the manuscript and producing code for experimental verification. However, we have reviewed and verified all AI-generated contents. We take responsibility for the correctness of this work.

We hope this addresses the reviewer’s concerns and presents the paper in a clearer light.

Reviewer_iE96OpenReview ↗

post rebutal

I re-read entirely the revised paper.

My overall impression remains negative. The paper is poorly written, and the theoretical section is difficult to follow even for a mathematically trained ICLR audience. The notation and the meaning of the main theoretical results need much clearer explanations. Moreover, these theoretical results do not establish the claimed practical advantage over existing taming methods.

I therefore maintain my recommendation to reject. I encourage the authors to rewrite and carefully check the exposition themselves, with particular attention to clarity and precision, and to strengthen the theoretical contribution by identifying conditions under which their method provably outperforms existing taming methods.

AuthorsOpenReview ↗

Further clarification on theoretical contributions and mathematical exposition

We thank the reviewer for taking the time to go through the revised manuscript.

Respectfully, we disagree with the assessment that our work lacks contribution. Our theoretical objective is to characterize the asymptotic scaling properties of the proposed tamer family. We have established critical coupling, derived proposal-variance scaling, and quantified asymptotic efficiency penalty induced by taming. Our experimental results complement the theoretical analysis by demonstrating stationary efficiency comparable to MALA on regular targets and improved efficiency on near-singular regions compared with existing tamers.

Furthermore, the reviewer has stated that our "theoretical results do not establish the claimed practical advantage". However, existing theoretical analyses from precedents MALTA (Atchade, 2006) and TMALA/TMALAc (Brosse et al, 2019) do not establish guarantees of better stationary sampling efficiency or faster recovery from pathological regions either. Such performance is evaluated empirically in these works, as it is in ours.

In particular, regarding https://arxiv.org/abs/1710.05559 which the reviewer mentioned, Brosse et al explicitly stated in Section 3 that their theoretical analysis follows that of MALTA, referring to Atchade (2006, Proposition 2.1). We have established that this result directly applies to TamedMALA under the same target assumptions. Additionally, our work provides a theoretical characterization of high-dimensional scaling and the asymptotic efficiency cost of score taming. We believe that our current theoretical and empirical contributions should be assessed on their own merits, while comparative guarantees for off-stationary performance remain directions for future research.

Regarding the mathematical exposition, we believe the revised manuscript provides appropriate notations and assumptions. All results have been presented in a logically organized and mathematically justified manner. While the other reviewers have not raised concerns regarding the theoretical exposition, we nevertheless welcome specific feedback on any remaining ambiguities so that we can address them directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.