acceptodds
Under review as a conference paper at ICLR 2027

ZOLD: A Zeroth-Order Langevin Dynamics Framework for LLM Fine-Tuning

Abstract

Zeroth-order (ZO) optimization has emerged as a memory-efficient paradigm for fine-tuning LLMs, estimating gradients from forward evaluations alone. However, existing approaches refine the ZO gradient estimator within an SGD-style update and thereby inherit a structural limitation of SGD: the absence of a mechanism for global exploration of the loss landscape. In this paper, we propose a novel framework, Zeroth-Order Langevin Dynamics (ZOLD), which can be viewed as a zeroth-order analogue of stochastic gradient Langevin dynamics. A precise coupling between the smoothing radius and the inverse temperature ties the ZOLD dynamics to a Gibbs measure concentrating on the global flat minima of the loss, i.e., the global minimizers of a Hessian-trace regularized objective. Within this framework, we further introduce layerwise normalized ZOLD (LN-ZOLD) that addresses the layerwise scale heterogeneity of pretrained LLMs at no additional memory or forward evaluation. On the theoretical side, we establish non-asymptotic Wasserstein-1 error bounds for ZOLD with respect to the flatness-biased Gibbs measure. Empirically, ZOLD and LN-ZOLD improve over MeZO in accuracy across OPT-13B/30B/66B, LLaMA3, and Phi-2, while remaining substantially faster than HiZOO and FZOO at comparable accuracy.

Primary area:
optimization
Keywords:
Zeroth-Order Optimization, Large Language Model Fine-Tuning, Memory-Efficient Training, Langevin Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_14AJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a zeroth-order optimization framework for LLM fine-tuning that incorporates Langevin dynamics to encourage global exploration and convergence toward flat minima. The authors further introduce a layerwise normalized variant and establish non-asymptotic Wasserstein-1 convergence guarantees. Experiments across several language models and downstream tasks demonstrate competitive accuracy and computational efficiency compared with existing ZO methods.

Critical strengths
  1. Improving the effectiveness and efficiency of zeroth-order LLM fine-tuning is an important research problem. Exploring optimization strategies beyond conventional ZO-SGD is potentially valuable.
  2. The paper provides extensive theoretical analysis, including non-asymptotic Wasserstein-1 bounds for ZOLD with respect to a flatness-biased Gibbs measure.
  3. The experiments cover multiple model families, including OPT models up to 66B parameters, and a variety of downstream tasks. The proposed methods achieve favorable accuracy-efficiency trade-offs compared with several existing ZO baselines, particularly HiZOO and FZOO.
Critical weaknesses
  1. The core motivation for introducing Langevin noise is not sufficiently justified. My primary concern is the motivation behind adding Langevin noise to zeroth-order optimization. ZO methods inherently suffer from highly noisy gradient estimates, especially in high-dimensional settings such as LLM fine-tuning. The variance of standard ZO gradient estimators can grow with the parameter dimension, making the optimization process already highly stochastic. Given this inherent stochasticity, it is unclear why introducing additional Gaussian noise is necessary or beneficial. Moreover, the paper argues that conventional ZO-SGD methods lack a mechanism for global exploration. However, this characterization overlooks the substantial stochasticity already present in ZO updates.

  2. The algorithmic novelty is limited. The proposed ZOLD algorithm appears to be a straightforward application of Langevin dynamics to standard ZO optimization, essentially replacing the stochastic gradient in SGLD with a conventional two-point ZO gradient estimator. Zeroth-order Langevin methods have already been investigated in previous literature [1,2]. Furthermore, the connection between Gaussian smoothing, Hessian-trace regularization, and flat minima has been established in existing works [3] .

  3. A major limitation of the experimental evaluation is the lack of ablation studies directly demonstrating the effectiveness of the injected Langevin noise.

  4. There is a substantial gap between the theoretical guarantees and the practical algorithms. The theoretical analysis focuses on vanilla ZOLD with fixed inverse temperature and smoothing radius. However, the main experiments employ an annealing schedule, which is not covered by the theoretical results. Furthermore, the analysis does not directly apply to LN-ZOLD, since layerwise normalization modifies the expected update direction.

  5. The experimental evidence is insufficient to support the claimed global exploration and flat-minima benefits. The paper's central claim is that Langevin dynamics enables global exploration and favors global flat minima. However, the experimental evidence supporting this mechanism is largely restricted to a two-dimensional synthetic example. For the LLM experiments, the authors primarily report downstream task performance without directly examining whether ZOLD discovers flatter solutions, escapes unfavorable local minima, or explores the loss landscape more effectively than existing ZO methods. In addition, the inverse-temperature values used in the LLM experiments are extremely large, resulting in a relatively small injected noise magnitude. It would be useful to quantify the magnitude of the Langevin perturbations relative to the inherent noise of ZO gradient estimates and demonstrate that they meaningfully affect the optimization trajectory.

[1] Liu, L., & Wang, Z. (2020). One-Point Gradient Estimators for Zeroth-Order Stochastic Gradient Langevin Dydamics.

[2] Roy, A., Shen, L., Balasubramanian, K., & Ghadimi, S. (2022). Stochastic zeroth-order discretizations of Langevin diffusions for Bayesian inference. Bernoulli, 28(3), 1810-1834.

[3] Bruno, S., Hwang, Y., An, J., Sabanis, S., & Lim, D. Y. (2025). Flatness-Aware Stochastic Gradient Langevin Dynamics. arXiv preprint arXiv:2510.02174.

Questions for authors

I do not have any questions.

Reviewer_j3MqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that the entire ZO-SGD family shares the update θk+1=θk−λG^\theta_{k+1}=\theta_k-\lambda\hat G, inheriting SGD's stationarity-only guarantees, and proposes replacing the base update with stochastic gradient Langevin dynamics: θk+1=θk−λG^(θk)+2λβ−1ξk+1\theta_{k+1}=\theta_k-\lambda\hat G(\theta_k)+\sqrt{2\lambda\beta^{-1}}\xi_{k+1}. The key skill is a coupling η=β−(1+τ)/4\eta=\beta^{-(1+\tau)/4} between smoothing radius and inverse temperature, which turns Gaussian smoothing's implicit Hessian-trace regularization into a target Gibbs measure over the flatness-biased objective v(θ)=u(θ)+η22tr(H)v(\theta)=u(\theta)+\tfrac{\eta^2}{2}\mathrm{tr}(H). LN-ZOLD adds a layerwise normalization G(l)/(1+λ∣∣θ(l)∣∣)G^{(l)}/(1+\sqrt{\lambda}||\theta^{(l)}||) inspired by tamed Euler schemes. Theorem 4.4 gives a non-asymptotic W1W_1 bound to πβ,η∗\pi^*_{\beta,\eta} with complexity O~(d3/ϵ2β(1+τ)/2poly(1/c^))\tilde O(d^3/\epsilon^2\beta^{(1+\tau)/2}\mathrm{poly}(1/\hat c)). Experiments on OPT-13B/30B/66B, LLaMA3, Phi-2 report +2.1/+2.8/+4.5 average over MeZO.

Critical strengths
  1. A genuinely different framing. Diagnosing the ZO-SGD family as "estimator bolted onto SGD" and swapping the base update for Langevin dynamics is a real conceptual departure, and the η\eta–β\beta coupling is a non-obvious way to make smoothing-induced curvature regularization part of the target distribution.
  2. A rare non-asymptotic guarantee tied to flat minima. Theorem 4.4 gives a W1W_1 convergence rate to the flatness-biased Gibbs measure under weak assumption which goes beyond stationary-only analyses of prior ZO work, and the empirical section spans five model families up to 66B with a per-step cost comparison shows the advantage.
Critical weaknesses
  1. The method is essentially an additive noise term bolted onto an already-high-variance estimator, and the paper never checks whether the injected Langevin noise survives the ZO estimation variance. ZO gradient estimates at LLM dimension already have variance Θ(d/q)\Theta(d/q) which is the dominant problem the field exists to mitigate and a term of size 2λ/β\sqrt{2\lambda/\beta} may be negligible against it. Nothing in the paper quantifies the ratio ∣∣2λ/βξ∣∣||\sqrt{2\lambda/\beta}\xi|| versus the estimator noise λ∣∣G^−EG^∣∣\lambda||\hat G-\mathbb E\hat G||: if the latter dominates, the Langevin term cannot meaningfully drive exploration.
  2. LN-ZOLD is a non-contributing component, and the empirical section is too thin to support the claims. The layerwise normalization does not work well: on the main table, for llama and Phi-2, LN-ZOLD even loses ZOLD on average and the conclusion itself admits LN-ZOLD "does not uniformly outperform vanilla ZOLD." Shipping a component that is neutral-to-negative as a headline contribution without bad case analysis is inappropriate. Broader evaluation is similarly insufficient: models are scaled but tasks are not made harder, there is no full comparison against FO fine-tuning (nor against FOLD, the natural first-order Langevin counterpart, to size the remaining gap), multi-seed evaluation covers only SST-2 and WSC, and nowhere is the baselines' learning-rate selection reported or swept.
Questions for authors

Please see weaknesses.

  1. Could you report, across training, the magnitude of the injected Langevin noise relative to the ZO estimator's stochastic variance (and its dd/query dependence)? Does the injected noise remain visible above estimation noise?
  2. Does ZOLD actually escape basins on real LLMs? The 2D illustration is insufficient as evidence.
  3. Given LN-ZOLD is neutral or worse than vanilla ZOLD, how to explain this phenomenon? Would MeZO or other ZO-baselines given the same layerwise normalization close the gap?
Reviewer_xq2LOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the author argues that existing zeroth-order methods for LLM fine-tuning primarily refine gradient estimators within an SGD-style update, leaving the lack of an explicit mechanism for global exploration largely unaddressed. The paper proposes Zeroth-Order Langevin Dynamics (ZOLD), which introduces Langevin noise and couples the smoothing radius with inverse temperature to support exploration and favor flat minima, together with a layerwise normalized variant that addresses parameter-scale heterogeneity. The authors provide non-asymptotic theoretical error bounds under stated assumptions and report improved average performance over MeZO across several LLM families, with lower per-step computational costs than HiZOO-L and FZOO.

Critical strengths
  • The paper is well-organized overall.
  • The author compared the proposed method on multiple benchmarks and on different LLMs.
  • The paper provides detailed mathematical derivations and proofs that offer theoretical support for the proposed framework. In particular, the non-asymptotic analysis helps characterize its convergence behavior under explicitly stated assumptions.
  • The paper includes useful ablation studies examining the effects of parameter coupling and temperature scheduling, helping clarify the role of these design choices.
Critical weaknesses
  • The claim that existing ZO methods lack a mechanism for global exploration seems overstated. ZO gradient estimates already introduce randomness into parameter updates, which may itself facilitate exploration. Although the paper establishes theoretical benefits of explicit Langevin noise, it does not sufficiently explain how its exploration behavior differs from that induced by the inherent ZO estimation noise, or when the latter is insufficient. A clearer discussion and controlled ablations isolating the effect of the added noise would strengthen this claim.

  • The experimental comparison does not include representative low-rank or subspace-based ZO methods, such as LOZO and SubZero, despite citing these works. As well as including PeZO (Tan, 2025) and CurvZO (Wang, 2026) since these methods also aim to improve the efficiency and effectiveness of ZO fine-tuning, including them would help establish the practical advantages of ZOLD. It would also be useful to discuss whether the proposed Langevin mechanism can complement low-dimensional perturbations.

  • The main text focuses on per-step runtime without adequately addressing end-to-end training time and computational cost. This is a major omission: the potentially much longer training time of zeroth-order methods compared with first-order fine-tuning is a central practical concern, and faster individual steps do not establish overall training efficiency. Table 6 in Appendix F should be moved into the main paper, with an explicit comparison against first-order LoRA covering accuracy, total training time, and memory usage. These results are essential for assessing whether ZOLD offers a practically meaningful efficiency trade-off.

  • The runtime comparison in Table 6 is difficult to interpret. Table 5 specifies a 20k-iteration budget for ZOLD and LN-ZOLD, whereas Table 6 reports time to the best-accuracy checkpoint without providing the corresponding step counts or clearly specifying how that checkpoint is selected. This retrospective metric may underestimate practical training costs, since the best checkpoint is generally not known in advance. If selection uses test accuracy, it would also require ground-truth labels unavailable in deployment; if it uses validation accuracy, the selection and stopping protocols should be stated explicitly. Are all baselines evaluated using the same checkpoint-selection procedure and comparable training budgets? The authors should clarify these issues and report full-budget training time alongside time to a common accuracy target, where applicable.

  • Another question is that several entries in Table 6 warrant clarification. On OPT-125M/WSC, ZOLD and LN-ZOLD report identical mean accuracy and standard deviation (65.06 ± 1.11). MeZO and HiZOO-L also report identical standard deviations on OPT-2.7B (± 0.56) and OPT-13B (± 0.1.47) for both datasets. Given the stochasticity of ZO optimization and the additional noise introduced by ZOLD, these repeated statistics raise questions about the reliability of the reported results. It would be great if the authors could double check the reported results and clarify about this. Maybe provide per-seed results, confirm the evaluation split and sample size, and assess whether the observed gains are statistically meaningful.

  • The evaluation lacks broader coverage of challenging knowledge and reasoning benchmarks, such as MMLU and GSM8K.

  • One major concern is that the empirical evidence and results do not sufficiently substantiate the paper’s central explanation that existing ZO methods suffer from inadequate exploration and that additional Langevin noise leads to better solutions. Added noise may also slow convergence, and the reported accuracy gains are not consistent across tasks and baselines.

Questions for authors

Please refer to the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.