acceptodds
Withdrawn from ICLR 2027

Wanda-Guided Sparse Perturbations for Efficient Zeroth-Order Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zeroth-order (ZO) optimization provides a memory-efficient alternative to backpropagation for fine-tuning large language models (LLMs), but its estimator variance grows with the ambient parameter dimension, often causing slow and unstable optimization. We propose Wanda-Importance Sparsity-MeZO (WIS-MeZO), a saliency-guided ZO method that concentrates sparse perturbations on task-relevant coordinates. WIS-MeZO uses weight magnitudes and downstream input-feature statistics to allocate the perturbation budget while keeping the pretrained model dense and requiring neither gradients nor pre-training data. We characterize support quality through the gradient energy capture ratio and show that, under stated conditions, WIS-guided supports preserve more gradient energy than random sparse supports. We further establish an -smooth convergence guarantee governed by the effective search dimension and captured gradient energy. Experiments across two model families, scales from 3B to 32B, and 16 benchmarks show improved aggregate performance over MeZO and random sparse perturbations, competitive or stronger results than magnitude-based Sparse-MeZO, and fewer optimization steps to reach comparable validation losses.

Primary area:
optimization
Keywords:
Optimization, zeroth order, wanda

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uKJqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WIS-MeZO, a memory-efficient zeroth-order (ZO) fine-tuning method for large language models. It selects a task-relevant sparse set of coordinates using a Wanda-style importance score that combines pretrained weight magnitudes with activation statistics from a calibration batch, then performs ZO updates only on this subset while keeping the model dense overall. The paper analyzes support quality via a gradient-energy capture metric and provides convergence results. Empirically, WIS-MeZO reportedly outperforms or matches MeZO and random/magnitude-based sparse variants, achieving comparable validation loss in fewer optimization steps with lower memory overhead.

Critical strengths

Pretrained LLMs are often driven by only a small subset of neurons. Based on this observation, one can fine-tune only those neurons that are deemed most critical, rather than updating the entire network. The paper adopts a Wanda-style criterion to identify important neurons and performs fine-tuning using a sparse mask that specifies this subset. Because fine-tuning updates only a small number of parameters, the zeroth-order gradient estimator has lower variance, which in turn should speed up convergence. This intuition sounds reasonable.

Critical weaknesses
  1. Calibration batch selection & impact. How is the calibration dataset (or calibration batch) chosen? How does it affect performance? The current method uses a lazy update strategy to update the mask MM. Have you tested an alternative where the mask MM is updated for every batch, without using a separate calibration batch?

  2. Fairness with batch size & memory accounting. If the batch size is too large, it may exceed GPU memory. When measuring peak GPU memory, does it include the additional memory overhead from the large batch size? Also, using a larger batch size can be unfair to other zeroth-order methods: larger batches reduce the variance of the gradient estimator, so zeroth-order comparisons should typically use the the same batch size. This paper would benefit from detailed ablation experiments.

  3. Theory missing the calibration batch. The theoretical analysis does not explicitly account for the calibration batch (which is central to how the mask is constructed). Please include this in the theory.

  4. Broader zeroth-order baselines. In the experiments, it would be better to add more types of zeroth-order optimization methods as baselines, e.g., low-rank ZO approaches.

  5. Query counting / single-query vs. multi-query. Single-query typically still requires two forward passes, using either central-difference or forward-difference schemes. For multi-query, if you fix the baseline loss, the number of additional queries can be reduced—this should be stated and accounted for clearly.

  6. Notation clarification. Here, dd denotes the total number of parameters (not the size of a single parameter matrix m×nm \times n). Please clarify this in the notation.

Questions for authors

Please address the first four critical weaknesses outlined above.

Reviewer_Wz6vOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes WIS-MeZO, a sparse zeroth-order fine-tuning method that uses the existing Wanda weight-activation saliency score to select coordinates for ZO perturbations. The authors introduce a gradient-energy capture metric to characterize support quality and provide convergence analysis depending on the effective perturbation dimension and captured gradient energy. Experiments on LLaMA and Qwen models from 3B to 32B parameters show improvements over MeZO, random sparse perturbations, and Sparse-MeZO on a range of downstream tasks.

Critical strengths
  1. The paper studies an important problem: reducing the high-dimensional variance of zeroth-order optimization for memory-efficient LLM fine-tuning. Sparse perturbation is a practical approach to this problem, particularly for large models. The experiments cover a relatively broad range of downstream tasks and model scales, including LLaMA and Qwen models from 3B to 32B parameters. The reported results consistently show that WIS-based support selection can outperform random masking and, in aggregate, magnitude-based Sparse-MeZO.

  2. The gradient-energy perspective is useful for analyzing the quality of sparse perturbation supports. The empirical diagnostic shows that the WIS support often captures substantially more gradient energy than random or magnitude-only supports at the same density. The paper also provides detailed memory and throughput measurements, including a bit-packed mask implementation that substantially reduces the additional storage overhead at the 32B scale.

Critical weaknesses
  1. The algorithmic novelty over Sparse-MeZO appears limited. Both methods follow essentially the same sparse-MeZO paradigm: a fixed subset of coordinates is selected and ZO perturbations are restricted to this support. The main methodological change is the support-ranking criterion, where WIS-MeZO replaces magnitude-based selection with the existing Wanda score Si,j=∣Wi,j∣∥Xj∥2S_{i,j}=|W_{i,j}|\|X_j\|_2. Wanda itself is an established pruning criterion rather than a new mechanism introduced in this work. Therefore, the core algorithm appears closer to applying an existing saliency heuristic within Sparse-MeZO than to introducing a substantially new ZO optimization framework. The gradient-energy analysis adds theoretical motivation, but does not substantially change the underlying optimization procedure. The experimental comparison is insufficient relative to recent ZO fine-tuning literature. The main ZO baselines are MeZO, Sparse-MeZO, and Random-MeZO. However, the paper itself discusses substantially more recent methods such as AGZO, FZOO, and low-rank/structured ZO methods, but does not compare against them experimentally. This is particularly problematic for AGZO, since AGZO also explicitly uses downstream activation information to construct more informative perturbation spaces. Thus, comparing WIS-MeZO only with magnitude-based Sparse-MeZO does not establish whether the proposed use of activations is competitive with existing activation-guided ZO methods. The large-scale 7B--32B experiments are even more limited, as they primarily compare WIS-MeZO with vanilla MeZO.

  2. The theory does not establish that WIS-MeZO has a better convergence guarantee than Sparse-MeZO. Theorem 5.4 only shows that WIS captures more energy than a random support under the sufficient condition ρ∗>r/κξ\rho^*>r/\kappa_\xi. It does not establish superiority over the magnitude-based support used by Sparse-MeZO. Similarly, Theorem 5.8 is essentially a generic fixed-mask sparse-ZO convergence result parameterized by ρmin⁡\rho_{\min}; it does not prove that the WIS mask maintains a larger ρmin⁡\rho_{\min} than alternative support-selection strategies throughout optimization. This is especially relevant because the primary WIS mask is computed only once at initialization and remains fixed while the model changes. The theoretical motivation for using the full Wanda score rather than activation information alone is not sufficiently isolated. For a linear layer, the paper writes the per-example gradient as Gi,j=δiXjG_{i,j}=\delta_i X_j, so under the weak-correlation assumption the expected gradient energy is directly related to the feature second moment E[Xj2]E[X_j^2]. The weight magnitude ∣Wi,j∣|W_{i,j}| enters the analysis only through converting this quantity into the Wanda score, with the quality of the resulting bound depending on the potentially unfavorable ratio wmin⁡/wmax⁡w_{\min}/w_{\max}. An activation-only baseline, e.g., selecting coordinates based directly on ∥Xj∥2\|X_j\|_2 or its second moment, is therefore important to determine whether the Wanda weight factor itself is actually beneficial. This ablation is missing.

Questions for authors
  1. Could the authors clarify the methodological novelty relative to Sparse-MeZO? If Sparse-MeZO is viewed as "sparse ZO + a support-selection rule," is WIS-MeZO essentially the same framework with the magnitude criterion replaced by the existing Wanda saliency score? Which component of the optimization algorithm itself is fundamentally new?

  2. Why are recent ZO fine-tuning methods such as AGZO, LOZO/LOZO+, and FZOO not included in the experimental comparison? In particular, AGZO appears highly relevant because it also uses downstream activations to guide ZO perturbations. Could the authors provide a direct comparison under matched models, tasks, memory budgets, and query budgets?

  3. In the proof of Theorem 5.8, the analyzed estimator appears to be G^=(1/r)⟨G,Z⊙M⟩(Z⊙M)\hat{G}=(1/r)\langle G,Z\odot M\rangle(Z\odot M), whereas Algorithm 1 uses a finite-difference estimator with perturbation radius ϵ\epsilon. How is the finite-difference approximation error controlled? Should the convergence guarantee contain an additional term depending on ϵ\epsilon?

  4. Does the theory provide any guarantee that the WIS support has a larger ρmin⁡\rho_{\min} than the Sparse-MeZO support over the entire optimization trajectory? Theorem 5.4 appears to compare WIS only with uniform random masking, while Theorem 5.8 assumes ρmin⁡>0\rho_{\min}>0 for the chosen fixed mask.

  5. Since Gi,j=δiXjG_{i,j}=\delta_i X_j suggests that activation energy itself is directly related to gradient energy under Assumption 5.1, could the authors compare WIS against an activation-only support rule based on ∥Xj∥2\|X_j\|_2 or V^j\hat{V}_j? This would clarify whether multiplying by ∣Wi,j∣|W_{i,j}| provides additional value beyond the activation statistics.

  6. For the 7B, 14B, and 32B experiments, why is Sparse-MeZO (and other recent ZO methods) absent from the main comparison? Since scalability is a central claim, comparisons against stronger sparse or structured ZO baselines at these model scales would be important.

Reviewer_4ah3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WIS-MeZO, which only perturbs a sparse set of weights in MeZO, decided by the Wanda score (weight magnitude times activation norm). It gives a gradient energy analysis and a convergence bound, and shows gains over MeZO, random masks and Sparse-MeZO on GLUE, SuperGLUE and commonsense tasks with 3B to 32B models.

Critical strengths
  1. Simple and cheap, it needs only one calibration pass.
  2. The gradient energy plot (Fig. 2) clearly shows that the choice of weights matters.
Critical weaknesses
  1. It is a direct combination, basically Wanda plus Sparse-MeZO, and close methods like SensZOQ and AGZO are not compared.
  2. The gradient G_ij = δ_i x_j does not depend on W_ij, so the theory does not explain why |W| should be in the score. An activation-only mask is a missing ablation.
  3. Gains over Sparse-MeZO are small in several settings.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.