- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes WIS-MeZO, a memory-efficient zeroth-order (ZO) fine-tuning method for large language models. It selects a task-relevant sparse set of coordinates using a Wanda-style importance score that combines pretrained weight magnitudes with activation statistics from a calibration batch, then performs ZO updates only on this subset while keeping the model dense overall. The paper analyzes support quality via a gradient-energy capture metric and provides convergence results. Empirically, WIS-MeZO reportedly outperforms or matches MeZO and random/magnitude-based sparse variants, achieving comparable validation loss in fewer optimization steps with lower memory overhead.
Critical strengths
Pretrained LLMs are often driven by only a small subset of neurons. Based on this observation, one can fine-tune only those neurons that are deemed most critical, rather than updating the entire network. The paper adopts a Wanda-style criterion to identify important neurons and performs fine-tuning using a sparse mask that specifies this subset. Because fine-tuning updates only a small number of parameters, the zeroth-order gradient estimator has lower variance, which in turn should speed up convergence. This intuition sounds reasonable.
Critical weaknesses
-
Calibration batch selection & impact. How is the calibration dataset (or calibration batch) chosen? How does it affect performance? The current method uses a lazy update strategy to update the mask . Have you tested an alternative where the mask is updated for every batch, without using a separate calibration batch?
-
Fairness with batch size & memory accounting. If the batch size is too large, it may exceed GPU memory. When measuring peak GPU memory, does it include the additional memory overhead from the large batch size? Also, using a larger batch size can be unfair to other zeroth-order methods: larger batches reduce the variance of the gradient estimator, so zeroth-order comparisons should typically use the the same batch size. This paper would benefit from detailed ablation experiments.
-
Theory missing the calibration batch. The theoretical analysis does not explicitly account for the calibration batch (which is central to how the mask is constructed). Please include this in the theory.
-
Broader zeroth-order baselines. In the experiments, it would be better to add more types of zeroth-order optimization methods as baselines, e.g., low-rank ZO approaches.
-
Query counting / single-query vs. multi-query. Single-query typically still requires two forward passes, using either central-difference or forward-difference schemes. For multi-query, if you fix the baseline loss, the number of additional queries can be reduced—this should be stated and accounted for clearly.
-
Notation clarification. Here, denotes the total number of parameters (not the size of a single parameter matrix ). Please clarify this in the notation.
Questions for authors
Please address the first four critical weaknesses outlined above.