- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GaNsZO, a zeroth-order optimizer for LLM fine-tuning that adjusts MeZO's random-direction updates using a single scalar . This scalar is an exponential moving average of the Gauss–Newton curvature along the sampled direction . Experiments on OPT and Llama show modest gains in average scores over MeZO and HiZOO, lower peak memory usage than HiZOO, and wall-clock time savings.
Critical strengths
The design is simple and cheap. Extracting a scalar curvature signal from the hidden-state difference already available from the two forward passes is an appealing idea.
Critical weaknesses
- The gains may simply come from automatic learning-rate decay.
The effective learning rate is , and Fig. 4 shows that it decreases smoothly throughout training. The paper itself describes GaNsZO as a "learning rate scheduler." However, MeZO is only tested with two constant learning rates and no comparable decay schedule. Therefore, the current experiments cannot distinguish the benefit of curvature information from that of a learning-rate schedule.
- The gains are comparable to the noise.
On OPT, the improvement over HiZOO is modest and not statistically significant, while most per-task differences are within one standard deviation. The claim that EMA reduces variance also appears inconsistent with Table 5.
Questions for authors
- There are too many internal inconsistencies. For example, Fig. 1b shows a mean of approximately 1.22, a 99.9th percentile of approximately 1.03, and a maximum of only about 2.7. These statistics cannot all be correct. Fig. 4c also does not match Table 1: the caption specifies Llama SST-2, but the values look more like the OPT results. Tables 8 and 1 report different numbers for the same method. Could the authors explain these discrepancies?
- I do not see what the theoretical analysis adds. It considers a version that clips to and takes , whereas the experiments do not use this version. The resulting convergence rate has the same dependence as standard ZO-SGD and does not show any benefit from curvature.
- Since the method is inspired by PseuZO, why is PseuZO not included as a baseline? I also consider FZOO and HELENE relevant baselines.
6 comments
Thank you for the additional experiments and clarifications. The new results, however, reinforce my concerns about the magnitude and robustness of the performance gains. In particular, the added baselines—PseuZO, FZOO, and HELENE—were evaluated with only one fixed learning rate per method, yet FZOO already performs close to GaNsZO. It therefore remains unclear whether GaNsZO would retain a meaningful advantage if these baselines received comparable learning-rate tuning.
Thank you for emphasizing the importance of tuning the additional baselines. We extended the comparison for FZOO, the strongest of the three, on OPT-1.3B with SST-2, RTE, and BoolQ (seed 0). Starting from FZOO’s calibrated peak learning rate, we applied the same linear, cosine, and exponential schedules used for MeZO, each ending at 10% of the peak.
| method | schedule | SST2 | RTE | BoolQ | AVG |
|---|---|---|---|---|---|
| GaNsZO | const | 90.6 | 63.1 | 65.7 | 73.1 |
| GaNsZO | cosine | 91.7 | 68.2 | 64.2 | 74.7 |
| FZOO | const | 89.5 | 63.0 | 64.1 | 72.2 |
| FZOO | linear | 89.4 | 58.8 | 63.8 | 70.7 |
| FZOO | cosine | 90.1 | 60.3 | 63.7 | 71.4 |
| FZOO | exponential | 89.4 | 59.6 | 63.2 | 70.7 |
Constant learning rate gives FZOO its best average, while cosine gives its best SST-2 score. GaNsZO with a constant base rate remains ahead of the best tested FZOO configuration on each task and leads the best FZOO average by 0.9 points. We also evaluated cosine decay on GaNsZO’s base rate, so the effective step size becomes . This improved the average to 74.7, driven primarily by RTE, while lowering the BoolQ score. The main comparison relies on GaNsZO with a constant base rate.
These results show that the tested schedules do not close the gap at FZOO’s calibrated peak rate. We will continue this research, and we hope that our responses and new experimental results can address your concerns.
Thank you for pointing out the reporting discrepancies and for suggesting stronger controls.
Learning-rate decay (W1).
We tested MeZO with linear, cosine, and exponential decay on OPT-1.3B. All three schedules decay from 10−6 to 10−7 over 20K steps (lines 839–844). On ten tasks and three seeds, they raise the average from 62.3 to 63.3, 63.2, and 63.5, respectively. GaNsZO reaches 63.9 (Tables 4 and 7). The results confirm that decay accounts for a substantial share of the gain over constant-rate MeZO. GaNsZO retains a modest average advantage over the schedules tested and generates its effective rate through online curvature estimates without a separate decay rule.
Effect size and EMA (W2).
We agree that the reported three-seed results do not establish statistical significance, and the improvements vary by task. Across the model-level task averages, GaNsZO scores 81.5 versus 80.5/78.2 for MeZO/HiZOO on RoBERTa-large, 63.9 versus 62.3/62.9 on OPT-1.3B, and 76.0 versus 73.9/73.7 on Llama-3.1-8B, where the Hessian baseline is HiZOO-L (Tables 6 and 8). These are descriptive gains across multiple architectures and scales. To be precise, our stability claim concerns within-run behavior: GaNsZO maintains a numerically stable, temporally smooth effective step size and avoids the loss explosions and per-coordinate oscillations that aggressive HiZOO smoothing induces (Fig. 4(a));Also, Table 18 reports a lower across-iteration variance of the effective step than HiZOO in its adaptive, non-degenerate regime.
Figure and table discrepancies (Q1).
We apologize for the error in Fig. 1(b). The 99.9th percentile was recorded incorrectly and has been corrected in the revised figure. Fig. 4(c) evaluates the EMA sweep after 10K steps, including SST-2 on Llama-3.1-8B. The shorter budget explains why its scores can be lower than the main results. The sweep tests sensitivity to the choice of α (lines 478–485). The former Table 8, now Table 11, reports a seed-0 ablation of the autodiff target. Table 1 instead reports means across three seeds, with standard deviations in Table 6. We now make the step budget and seed count explicit in the respective captions.
Scope of the theory (Q2).
Clipping the previous EMA state to [c0,c1] provides uniform bounds for the analysis. The current denominator still incorporates the directional curvature. In our experiments, the curvature scalar stays within a well-behaved range without clipping in the trajectories shown in Fig. 4(a). The limit μ→0 replaces the finite difference by its directional derivative for the analysis. The experiments use a finite μ. We agree that Theorem 4.1 matches the effective-dimension order of fixed-step ZO-SGD. Appendix C.1 adds a separate result that quantifies when the curvature scalar improves the rate. On the positive-definite quadratic family analyzed there, accurate directional curvature and exact directional derivatives improve the condition-number dependence from Θ(κ) to O(κ) for fixed relative curvature error below one (lines 1377–1457).
Additional baselines (Q3).
We added PseuZO, FZOO, and HELENE on OPT-1.3B for SST-2, RTE, and BoolQ (Table 9 and lines 883–894). PseuZO uses an EMA of Jacobian estimates with a sliding window. FZOO normalizes using loss variation across perturbations and accelerates their batched evaluation. HELENE combines gradient annealing with layerwise clipping of diagonal curvature estimates. Different from these methods, GaNsZO uses a single Gauss–Newton scalar along the sampled direction and reuses the two forward evaluations used for gradient estimation. In this single-seed comparison with one fixed learning rate per method, GaNsZO scores 90.6, 63.1, and 65.7, compared with 89.5, 63.0, and 64.1 for FZOO, the strongest of the three additional baselines. We present this comparison as a limited check under the same training pipeline.