- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies "plug-in" adaptive ZO optimization, where zeroth-order gradient estimators (ZOGEs) are naively substituted into adaptive first-order optimizers (Adam, RMSProp, regularized Muon, R-AdaZO). The authors argue this naive substitution is problematic: the estimation noise propagates not only into the current update direction but also into the optimizer's persistent states (e.g. second-moment EMAs), creating a systematic optimizer-induced bias that standard ZO analysis ignores.
They derive joint bias/covariance expansions for the plug-in transition, separating finite-radius smoothing bias from finite-sampling bias, showing the leading sampling-bias term scales as . They then propose C-SQE (Corrected Same-Query Extrapolation): partition the queries into two balanced halves, form half-sample estimates, and apply a Richardson-style extrapolated transition , with a rational correction keeping Adam-style second moments nonnegative. This reduces sampling bias to while reusing the same queries (no extra budget) and preserving the leading covariance (Thm 4.1). Convergence theory (Thm 4.2, Cor 4.2) shows the residual update-bias contribution improves from (R-AdaZO: ). Synthetic and a LLM soft-prompt fine-tuning experiments show C-SQE's advantage against simple plug-in optimizers.
Critical strengths
-
A genuinely overlooked problem, cleanly formalized. The observation that ZO noise contaminates persistent optimizer state (not just the step) is correct and under-appreciated; the joint bias/covariance expansion separating radius-bias from sampling-bias is a principled contribution, and the "same-query" extrapolation is an elegant, budget-neutral correction (no extra forward evals).
-
Broad, optimizer-agnostic framing with matching theory. The analysis and C-SQE construction cover Adam/RMSProp/regularized Muon/R-AdaZO under one transition-map abstraction, and the synthetic experiments cleanly confirm the predicted bias-slope improvement, which is convincing evidence that the mechanism is real.
Critical weaknesses
- C-SQE corrects only the bias, but the leading variance problem is left uncorrected, and this is exactly what dominates in the deployed regime. The paper's own expansions show the leading covariance term is and is explicitly preserved by C-SQE. The correction only moves the (sub-leading) sampling bias from to , and the convergence gains (, ) are improvements to high-order terms. At small , especially the LLM experiments using , variance overwhelmingly dominates the total error, so the asymptotic bias-order improvement that the theory celebrates is largely irrelevant precisely where ZO is actually used. This is a fundamental tension between where the theory bites (large ) and where ZO lives (small ).
- The real-model evidence is narrow and partly within noise, leaving practical significance unestablished. The only LLM experiment is 64-dim soft-prompt tuning on a frozen OPT-125M with and 3 seeds and the model backbone is never actually optimized, so this is a 64-parameter problem, not LLM fine-tuning in any meaningful sense. The convincing results are synthetic with huge ; the regime that matters is demonstrated only on a toy-scale, partly-negative setup.
Questions for authors
Please see weaknesses.
- Could you report, at the deployed (when small), the empirical split of total MSE into bias vs variance for plug-in vs C-SQE? The core question is whether the bias reduction is even measurable when the variance dominates.
- Any result where actual model weights (not a 64-dim prompt) are updated, e.g. MeZO-style full/LoRA fine-tuning ,even on a small model (like OPT125M)? Does the state-bias correction still help once is large?
- Given variance is the leading term, did you consider control-variate / antithetic sampling combined with C-SQE, and would that be the more impactful contribution?