- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes HiZO (HEBBIAN-INFORMED ZEROTH-ORDEROPTIMIZATION) which changes the normal perturbation in MeZO to a hebbian-informed distribution. The idea is valid and a reasonable research direction. Experiments are tested against MeZO and AGZO on Roberta-Large with improvement from the proposed method.
Critical strengths
The paper proposes a simple and straightforward alternative of generating perturbation for zero-order optimization. The method is well-motivated and experiments are conducted to validate the effectiveness with analysis against real gradients. The results seem promising on the tested scenarios.
Critical weaknesses
-
The runtime & memory consumption of HiZO is about 1.5x of MeZO. However, that comparison is not available for AGZO. This is a critical missing baseline.
-
Lack of experiments on large model and data. For ZO methods, the main claim is savings on memory cost, compared with FO methods. This is more of a problem for large models. For tested models (Roberta-large, Qwen3-1.7B), practically, there's not much motivation to use ZO methods than FO methods. Would like to see some experiments on larger models at 8B scale. Another aspect is what happens for long context models. Since the computation now relies on activation, requiring storing such for long context is somtimes out-weights the model itself.
-
A strong baseline is missing. It compares with AGZO and also outperforms it. Dong, Xun, et al. "ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces." arXiv preprint arXiv:2607.01125 (2026).
Questions for authors
-
What's the wall clock and memory consumption of AGZO?
-
What's the wall clock and memory consumption for Qwen3 1.7B model?
-
Can the authors test on 8B-scale models?
-
How does it work under long context? What exactly needs to be stored? In MeZO, memory savings come from storing the random seed, instead of perturbation vector. But for HiZO, it seems the covariance needs to be stored?
-
Can the author compare with ZO-Act?
Thank you for highlighting the missing AGZO profile and for asking how the method behaves on larger models and longer sequences.
Q1–Q2 and W1 — Runtime and memory. We profiled Llama3-8B on RTE using a single A800 80 GB GPU. AGZO was close to MeZO on both measures. Subsampling reduced HiZO's per-step time, and the separable variant reduced its memory overhead in this profile.
Q3, Q5, and W2–W3 — 8B evaluation and ZO-Act. In addition to the RTE profile, we ran MeZO, AGZO, HiZO, and our reproduction of ZO-Act on SST-2 with Llama3-8B. The evaluation loss at 3,000 updates was 0.335 for HiZO, 0.383 for MeZO, and about 0.442 for each activation-guided baseline. Our ZO-Act run used learning-rate warmup. This comparison covers convergence on one task. In the eight-direction configuration, ZO-Act uses nine forward evaluations per update, while HiZO uses two per minibatch. In their Llama3-8B RTE experiment, Dong et al. report 42.0 minutes and 16.1 GB for ZO-Act with a budget of 4,500 forward queries. This corresponds to an average of 5.04 seconds per optimizer step. We are running more experiments on Llama3-8B and will share the results when they are available.
Q4 — What is stored for long sequences? HiZO regenerates the perturbation from the random seed and the cached diagonal variance weights. For an m×n weight matrix, these weights form an m×n map. Its size is independent of sequence length. HiZO computes the statistics during the forward pass, and the activation workspace and covariance computation can still grow with the number of tokens. S-HiZO uses a fixed number of activation rows for the covariance product and retains the dense map. SA-HiZO stores only O(m+n) persistent guidance values per matrix, with temporary dense buffers still needed in the current implementation.