acceptodds
Under review as a conference paper at ICLR 2027

GaNsZO: Gauss-Newton Step-sized Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning

Abstract

Fine-tuning large language models at scale is increasingly constrained by the memory cost of backpropagation. Zeroth-order (ZO) optimizers (e.g., MeZO) alleviate this bottleneck by relying solely on forward passes, but they often suffer from slow convergence. Recent works such as HiZOO utilize diagonal Hessian estimators as preconditioner for acceleration, yet this incurs additional memory costs comparable to training parameters. Moreover, the noisy diagonal Hessian estimates often lead to training instabilities. In this work, we introduce the uss-ewton tepsized eroth-rder (GaNsZO) algorithm, a ZO optimizer. GaNsZO estimates the directional second derivative along the descent direction and uses it as a stepsize scaling factor, reducing memory overhead while improving robustness. To estimate efficiently, GaNsZO exploits the model's composite structure and computes Gauss-Newton quadratic form along that direction, saving one forward pass compared to standard central-difference schemes. Empirically, on OPT and Llama models, GaNsZO consistently outperforms HiZOO in terms of accuracy and training stability. In certain tasks (e.g., SST-2), GaNsZO approaches the performance of first-order fine-tuning. In terms of efficiency, GaNsZO reduces memory consumption by up to 43% compared to HiZOO, while requiring significantly fewer forward passes to reach comparable loss, leading to up to end-to-end speedup in selected settings. Finally, we show that GaNsZO combines naturally with PEFT and recent ZO techniques (e.g., LOZO) and produces further improvements.

Primary area:
optimization
Keywords:
zeroth order optimization, large language model fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_djZPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GaNsZO, a zeroth-order optimizer for LLM fine-tuning that adjusts MeZO's random-direction updates using a single scalar σt\sigma_t. This scalar is an exponential moving average of the Gauss–Newton curvature along the sampled direction zz. Experiments on OPT and Llama show modest gains in average scores over MeZO and HiZOO, lower peak memory usage than HiZOO, and wall-clock time savings.

Critical strengths

The design is simple and cheap. Extracting a scalar curvature signal from the hidden-state difference already available from the two forward passes is an appealing idea.

Critical weaknesses
  1. The gains may simply come from automatic learning-rate decay.

The effective learning rate is η/σt\eta/\sigma_t, and Fig. 4 shows that it decreases smoothly throughout training. The paper itself describes GaNsZO as a "learning rate scheduler." However, MeZO is only tested with two constant learning rates and no comparable decay schedule. Therefore, the current experiments cannot distinguish the benefit of curvature information from that of a learning-rate schedule.

  1. The gains are comparable to the noise.

On OPT, the improvement over HiZOO is modest and not statistically significant, while most per-task differences are within one standard deviation. The claim that EMA reduces variance also appears inconsistent with Table 5.

Questions for authors
  1. There are too many internal inconsistencies. For example, Fig. 1b shows a mean of approximately 1.22, a 99.9th percentile of approximately 1.03, and a maximum of only about 2.7. These statistics cannot all be correct. Fig. 4c also does not match Table 1: the caption specifies Llama SST-2, but the values look more like the OPT results. Tables 8 and 1 report different numbers for the same method. Could the authors explain these discrepancies?
  2. I do not see what the theoretical analysis adds. It considers a version that clips σ\sigma to [c0,c1][c_0, c_1] and takes μ→0\mu \to 0, whereas the experiments do not use this version. The resulting convergence rate has the same deffd_{\mathrm{eff}} dependence as standard ZO-SGD and does not show any benefit from curvature.
  3. Since the method is inspired by PseuZO, why is PseuZO not included as a baseline? I also consider FZOO and HELENE relevant baselines.
6 comments

Thank you for pointing out the reporting discrepancies and for suggesting stronger controls.

Learning-rate decay (W1).

We tested MeZO with linear, cosine, and exponential decay on OPT-1.3B. All three schedules decay from 10−610^{-6} to 10−710^{-7} over 20K steps (lines 839–844). On ten tasks and three seeds, they raise the average from 62.3 to 63.3, 63.2, and 63.5, respectively. GaNsZO reaches 63.9 (Tables 4 and 7). The results confirm that decay accounts for a substantial share of the gain over constant-rate MeZO. GaNsZO retains a modest average advantage over the schedules tested and generates its effective rate through online curvature estimates without a separate decay rule.

Effect size and EMA (W2).

We agree that the reported three-seed results do not establish statistical significance, and the improvements vary by task. Across the model-level task averages, GaNsZO scores 81.5 versus 80.5/78.2 for MeZO/HiZOO on RoBERTa-large, 63.9 versus 62.3/62.9 on OPT-1.3B, and 76.0 versus 73.9/73.7 on Llama-3.1-8B, where the Hessian baseline is HiZOO-L (Tables 6 and 8). These are descriptive gains across multiple architectures and scales. To be precise, our stability claim concerns within-run behavior: GaNsZO maintains a numerically stable, temporally smooth effective step size and avoids the loss explosions and per-coordinate oscillations that aggressive HiZOO smoothing induces (Fig. 4(a));Also, Table 18 reports a lower across-iteration variance of the effective step than HiZOO in its adaptive, non-degenerate regime.

Figure and table discrepancies (Q1).

We apologize for the error in Fig. 1(b). The 99.9th percentile was recorded incorrectly and has been corrected in the revised figure. Fig. 4(c) evaluates the EMA sweep after 10K steps, including SST-2 on Llama-3.1-8B. The shorter budget explains why its scores can be lower than the main results. The sweep tests sensitivity to the choice of α\alpha (lines 478–485). The former Table 8, now Table 11, reports a seed-0 ablation of the autodiff target. Table 1 instead reports means across three seeds, with standard deviations in Table 6. We now make the step budget and seed count explicit in the respective captions.

Scope of the theory (Q2).

Clipping the previous EMA state to [c0,c1][c_0,c_1] provides uniform bounds for the analysis. The current denominator still incorporates the directional curvature. In our experiments, the curvature scalar stays within a well-behaved range without clipping in the trajectories shown in Fig. 4(a). The limit μ→0\mu\to0 replaces the finite difference by its directional derivative for the analysis. The experiments use a finite μ\mu. We agree that Theorem 4.1 matches the effective-dimension order of fixed-step ZO-SGD. Appendix C.1 adds a separate result that quantifies when the curvature scalar improves the rate. On the positive-definite quadratic family analyzed there, accurate directional curvature and exact directional derivatives improve the condition-number dependence from Θ(κ)\Theta(\kappa) to O(κ)O(\sqrt{\kappa}) for fixed relative curvature error below one (lines 1377–1457).

Additional baselines (Q3).

We added PseuZO, FZOO, and HELENE on OPT-1.3B for SST-2, RTE, and BoolQ (Table 9 and lines 883–894). PseuZO uses an EMA of Jacobian estimates with a sliding window. FZOO normalizes using loss variation across perturbations and accelerates their batched evaluation. HELENE combines gradient annealing with layerwise clipping of diagonal curvature estimates. Different from these methods, GaNsZO uses a single Gauss–Newton scalar along the sampled direction and reuses the two forward evaluations used for gradient estimation. In this single-seed comparison with one fixed learning rate per method, GaNsZO scores 90.6, 63.1, and 65.7, compared with 89.5, 63.0, and 64.1 for FZOO, the strongest of the three additional baselines. We present this comparison as a limited check under the same training pipeline.

Thank you for the additional experiments and clarifications. The new results, however, reinforce my concerns about the magnitude and robustness of the performance gains. In particular, the added baselines—PseuZO, FZOO, and HELENE—were evaluated with only one fixed learning rate per method, yet FZOO already performs close to GaNsZO. It therefore remains unclear whether GaNsZO would retain a meaningful advantage if these baselines received comparable learning-rate tuning.

Thank you for emphasizing the importance of tuning the additional baselines. We extended the comparison for FZOO, the strongest of the three, on OPT-1.3B with SST-2, RTE, and BoolQ (seed 0). Starting from FZOO’s calibrated peak learning rate, we applied the same linear, cosine, and exponential schedules used for MeZO, each ending at 10% of the peak.

methodscheduleSST2RTEBoolQAVG
GaNsZOconst90.663.165.773.1
GaNsZOcosine91.768.264.274.7
FZOOconst89.563.064.172.2
FZOOlinear89.458.863.870.7
FZOOcosine90.160.363.771.4
FZOOexponential89.459.663.270.7

Constant learning rate gives FZOO its best average, while cosine gives its best SST-2 score. GaNsZO with a constant base rate remains ahead of the best tested FZOO configuration on each task and leads the best FZOO average by 0.9 points. We also evaluated cosine decay on GaNsZO’s base rate, so the effective step size becomes ηt/σt\eta_t/\sigma_t. This improved the average to 74.7, driven primarily by RTE, while lowering the BoolQ score. The main comparison relies on GaNsZO with a constant base rate.

These results show that the tested schedules do not close the gap at FZOO’s calibrated peak rate. We will continue this research, and we hope that our responses and new experimental results can address your concerns.

Reviewer_qnXfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Gauss-Newton step sized Zeroth-Order (GaNsZO) algorithm to improve the convergence and performance of zero-order optimization for LLMs. The core idea is adding a step size control algorithm on top of HiZOO. Experiment shows that the proposed method outperforms MeZO and HiZOO over multiple benchmarks and models.

Critical strengths
  1. The findings and analysis of HiZOO is interesting and valuable. It apparently shows a mismatch between HiZoo and the real behavior of Hesssians. Regardless of whether GaNsZO is the right solution to this problem, it opens a good research direction and practical engineering challenges.

  2. The proposed method is simple and with small additional cost. The experiments are over multiple configurations, setups, and tasks.

Critical weaknesses
  1. GaNsZO effectively modifies the learning rate dynamically during the training. So a natural baseline is some other learning rate scheduler on top of MeZO to compare with. This comparison is missing.

  2. The performance of HiZOO baseline in the paper seems worse than their original paper and sometimes worse than MeZO. This brings my concern about whether the baseline is faithfully re-produced.

Questions for authors
  1. Why is the HiZOO worse than MeZO in the experiments? Especially Figure 2, HiZOO is way much worse than MeZO. Also in some tables, HiZOO has lower numbers than MeZO. The numbers on LLama also does not match their original paper. I'm concerned about whether the re-production of HiZOO in the paper is faithful.

  2. Figure 4b shows that the effective LR of GaNsZO is like a exponential decay. The author should add some experiments comparing MeZO with several dynamic learning rate annealing methods as baselines. Also the author should do some literature search and add related works on LR scheduling, and as baselines for comparison.

  3. Does MeZO experiment in the paper also uses one-sided gradient estimator as GaNsZO?

  4. A related work is missing Seung, Hyunseok, Jaewoo Lee, and Hyunsuk Ko. "Low-rank curvature for zeroth-order optimization in llm fine-tuning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 30. 2026.

2 comments

Thank you for raising the questions about learning-rate scheduling and the HiZOO comparison.

Learning-rate schedules (Weakness 1 and Q2).

Constant-rate MeZO follows the original OPT protocol of Malladi et al. (2023). Following your suggestion, we tested linear, cosine, and exponential decay on OPT-1.3B. All three schedules start at 10−610^{-6} and end at 10−710^{-7} over 20K steps (lines 839–844). Across ten tasks and three seeds, they improve the average from 62.3 for constant-rate MeZO to 63.3, 63.2, and 63.5, respectively. Exponential decay gives the strongest scheduled baseline, consistent with the trend in Fig. 4(b). GaNsZO averages 63.9 and obtains a declining effective learning rate from the curvature estimate without a separate scheduler (Tables 4 and 7). The comparison shows that decay is effective and that GaNsZO retains a modest average lead over the tested schedules.

HiZOO reproduction (Weakness 2 and Q1).

We implemented HiZOO and HiZOO-L in the same MeZO training pipeline and used the same data splits and evaluation protocol across methods (lines 307–314). Table 5 reports the learning-rate and EMA grids. Table 6 reports the results for seeds 0, 42, and 2026. HiZOO improves the OPT-1.3B average from 62.3 to 62.9, with task-level gains and losses. On Llama-3.1-8B, HiZOO-L averages 73.7 versus 73.9 for MeZO. The original HiZOO paper reports results for Llama3-8B, so its absolute scores correspond to a different checkpoint.

Figure 2 compares training loss at equal numbers of forward evaluations. Since HiZOO uses three evaluations per update and MeZO uses two, HiZOO makes only two-thirds as many updates at a fixed query budget. This helps explain how a modest score gain at a fixed step budget can coexist with weaker query efficiency. Published comparisons also show that HiZOO’s gains over MeZO depend on the task and tuning setup, for example, in PaZO’s Table 2 [1] and FZOO's Table 2 [2]. Our narrower conclusion is that, in the settings we tested, the benefits of maintaining a per-parameter diagonal were limited relative to its cost. This motivated the scalar alternative in Sec. 3.2.

MeZO estimator (Q3).

MeZO uses the standard two-sided estimator in our experiments, whereas GaNsZO uses a one-sided difference so that the same two forward evaluations can be reused for curvature estimation. The centered estimator gives MeZO lower finite-difference truncation error under standard smoothness conditions. In a Taylor expansion at a fixed direction, the leading error is typically O(μ2)O(\mu^2) for the centered difference versus O(μ)O(\mu) for the forward difference.

Related work (Q4).

Thank you for pointing out Seung et al. (AAAI 2026). We have cited LOREN in the related work. LOREN uses a low-rank block-diagonal curvature model to adapt the perturbation distribution. GaNsZO uses a single directional curvature scalar to set the length of the sampled update.

[1] Hanzhen Zhao, Shihong Ding, Cong Fang, Zhouchen Lin, PaZO: Preconditioned Accelerated Zeroth-Order Optimization for Fine-Tuning LLMs, Neurips 2025.

[2] Dang, Sizhe and Guo, Yangyang and Zhao, Yanjun and Ye, Haishan and Zheng, Xiaodong and Dai, Guang and Tsang, Ivor, FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed, ICLR 2026.

Dear reviewer qnXf,

Thank you again for your time and thoughtful feedback. With the discussion period closing tomorrow, we would appreciate any feedback on our rebuttal and would be happy to provide any further details.

Reviewer_M6i1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GaNsZO, a zeroth-order optimizer for LLM fine-tuning that replaces HiZOO's per-coordinate diagonal Hessian with one scalar, the Gauss-Newton curvature along the current perturbation direction. The scalar is computed from the finite-difference change of the last hidden state and a Hessian-vector product through the output head and loss. It is smoothed by an EMA and divides the MeZO step. Theorem 4.1 gives an iteration bound linear in an effective dimension. On OPT-1.3B, Llama-3.1-8B and RoBERTa-large with GLUE/SuperGLUE tasks, the paper reports higher average accuracy than MeZO and HiZOO, memory close to MeZO, about 5× fewer forward passes to reach MeZO's final training loss, and gains when combined with LoRA, prefix tuning, Rademacher directions and LOZO.

Critical strengths
  1. The method adds one scalar of optimizer state and keeps two forward passes per step. On Llama-3.1-8B it uses 21.9 GiB against 20.6 GiB for MeZO and 38.4 GiB for HiZOO (Fig. 3), and its per-step time matches MeZO (Table 2). Because it only rescales the step, it composes with LoRA, prefix tuning, Rademacher directions and LOZO, and the average gain over the matching baseline holds in every block of Table 3.

  2. Sec. 3.2 and Fig. 1 give a useful diagnostic of HiZOO. Under its default EMA coefficient of 1e-8, nearly all diagonal estimates stay close to their initial value of 1, and at 1e-6 the estimates blow up and training reaches NaN. So HiZOO in its stable regime behaves much like MeZO with a few rescaled coordinates. This is a good argument for replacing the diagonal with a scalar.

Critical weaknesses
  1. The experiments do not separate curvature information from a learning-rate schedule, and Sec. 5.5 itself calls GaNsZO a learning rate scheduler. Fig. 4b shows η/σt\eta/\sigma_t decaying from about 1e-6 to about 3e-7, while MeZO and HiZOO are tuned over two constant rates, {1e-6, 1e-7} (Table 4). With η∈\eta \in {1e-2, 1e-3}, each step is roughly 1% or 0.1% of the Gauss-Newton step of Eq. (3), so the global rate still sets the scale. The average gains in Table 5 are 1.6 and 2.1 points over three seeds, and several per-task gaps are within one standard deviation. Three runs would locate the gain: MeZO following the smoothed η/σt\eta/\sigma_t of a GaNsZO run, MeZO with a tuned decaying schedule, and GaNsZO with σ^t\hat\sigma_t computed on an independent direction.

  2. Theorem 4.1 shows no benefit from the curvature scalar, and one proof step is not justified. Remark C.7 needs (1−α)c0(1-\alpha)c_0 and (1−α)c1(1-\alpha)c_1 of order deffL1d_{\mathrm{eff}} L_1. This clips σt\sigma_t to a constant-factor band and turns Algorithm 2 into ZO-SGD with a fixed step of order 1/(deffL1)1/(d_{\mathrm{eff}} L_1), whose rate is the effective-rank rate already shown for MeZO (Malladi et al., 2023). In Eq. (18), σt−1⟨∇L,z⟩⟨∇LB,z⟩\sigma_t^{-1}\langle \nabla L, z\rangle\langle \nabla L_{\mathcal{B}}, z\rangle is lower-bounded by substituting the upper bound (16) for σt\sigma_t. This holds only when the product is nonnegative, and σt\sigma_t depends on the same batch and direction through σ^t\hat\sigma_t. The proof should bound this correlation term and state what the result gains over constant-step ZO-SGD.

Questions for authors

See weakness

2 comments

Thank you for identifying the missing learning-rate controls and the theory gap.

weakness 1

Learning-rate decay. In ZO-baselines, the learning rate directly multiplies the estimated directional derivative, so a suitable decay schedule can improve the update at no extra query cost. Our original constant-rate baseline followed the OPT protocol in Malladi et al. (2023), with the rate selected from the grid in Table 5. Motivated by the decline in GaNsZO’s effective learning rate in Fig. 4(b), we evaluated three explicit schedules on OPT-1.3B. All start at η0=10−6\eta_0=10^{-6} and end at 0.1η00.1\eta_0 after 20K steps. With p=t/20,000p=t/20{,}000, the respective multipliers are 1−0.9p1-0.9p (linear), 0.1+0.45(1+cos⁡(πp))0.1+0.45(1+\cos(\pi p)) (cosine), and 0.1p0.1^p (exponential). See lines 839–844.

Across ten tasks and three seeds, constant-rate MeZO averages 62.3, compared with 63.3, 63.2, and 63.5 for linear, cosine, and exponential decay. GaNsZO averages 63.9 (Tables 4 and 7). Decay therefore explains a substantial share of the gain over constant-rate MeZO. GaNsZO also has the highest observed average and produces a declining effective rate from online curvature estimates without a separate schedule. We agree that the global multiplier still controls the update scale but the comparison supports a modest advantage over the schedules tested.

And we appreciate the two additional suggestions. We prioritized standard schedules because they can be applied to MeZO directly at no extra query cost. Replaying a smoothed GaNsZO curve requires a prior GaNsZO run to supply the schedule. Estimating curvature along a separate direction requires an extra forward evaluation per step and gives up the reuse of the two evaluations that makes GaNsZO inexpensive. For this revision, we therefore focused on prescribing decay schedules for MeZO under the same two-forward-pass budget.

weakness 2

Eq. (18) and the theoretical comparison. We agree that a pointwise substitution of the denominator bound is invalid when the product of directional derivatives is negative. Let a=⟨∇L(θt),z⟩a=\langle\nabla L(\theta_t),z\rangle and bB=⟨∇L(θt;B),z⟩b_{\mathcal B}=\langle\nabla L(\theta_t;\mathcal B),z\rangle. Averaging over the gradient minibatch first replaces abB/σta b_{\mathcal B}/\sigma_t by a2/σta^2/\sigma_t in expectation. We can then apply the upper bound in Eq. (16) to the nonnegative numerator (revised Eq. (18), lines 1248–1253).

The clipping in Remark C.7 applies to the previous EMA state. The current denominator still depends on the newly sampled curvature. We agree that the original bound gives the same order in effective dimension as fixed-step ZO-SGD. Appendix C.1 adds a separate result for positive-definite quadratics with exact directional derivatives. If the relative error of the actual curvature denominator is at most δ<1\delta<1, the scaled step retains at least 1−δ21-\delta^2 of the exact line-search decrease. For the two-dimensional family analyzed there, this improves the condition-number dependence from Θ(κ)\Theta(\kappa) for the best fixed step to O(κ)O(\sqrt{\kappa}) for fixed δ\delta, at the same relative accuracy. See Eqs. (26)–(30) and lines 1439–1457.

Dear reviewer M6i1,

Thank you again for your time and thoughtful feedback. With the discussion period closing tomorrow, we would appreciate any feedback on our rebuttal and would be happy to provide any further details.

Comments on the paper

1 comment

We greatly appreciate the time and effort of the area chair and reviewers during the review phase and the subsequent discussion phase. Overall, the three reviewers were most concerned about the lack of comparison with MeZO with LR scheduler. We highly respect this suggestion and have conducted a detailed and comprehensive comparison. In addition, we have also made revisions to the theoretical section, errors in the diagrams, and unclear annotations, and added new experiments to our revised paper. Hope our comments and the revision paper can address the concerns of the area chair and all the reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.