acceptodds
Under review as a conference paper at ICLR 2027

HiZO: Hebbian-informed Zeroth-order Optimization for Efficient Fine-tuning

Abstract

Zeroth-order optimization enables language model fine-tuning without backpropagation, but isotropic perturbations do not exploit the task-dependent information contained in forward activations. We introduce Hebbian-informed Zeroth-order Optimization (HiZO), which uses this information to guide the perturbation distribution and improve convergence. HiZO uses the magnitude of centered input–output covariance to allocate perturbation variance across weight coordinates while preserving the total variance budget. The resulting guidance reflects both learned weights and downstream input statistics. These statistics are collected during existing loss evaluations and require no additional model forward passes. We also develop subsampled and separable variants to reduce the computation and storage needed for guidance. For smooth nonconvex objectives, we establish convergence bounds in the directional-derivative limit that characterize the roles of covariance–gradient alignment and estimator variability. A complementary one-layer analysis identifies conditions under which Hebbian covariance provides information about task-gradient magnitudes. Language-model experiments and controlled ablations show faster optimization with Hebbian guidance while retaining comparable or better predictive performance. These findings suggest that forward activations offer a practical source of task-dependent information for improving zeroth-order convergence.

Primary area:
optimization
Keywords:
Zeroth-order optimization, efficient fine-tuning, Hebbian learning, activation-guided optimization, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KicCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HiZO (HEBBIAN-INFORMED ZEROTH-ORDEROPTIMIZATION) which changes the normal perturbation in MeZO to a hebbian-informed distribution. The idea is valid and a reasonable research direction. Experiments are tested against MeZO and AGZO on Roberta-Large with improvement from the proposed method.

Critical strengths

The paper proposes a simple and straightforward alternative of generating perturbation for zero-order optimization. The method is well-motivated and experiments are conducted to validate the effectiveness with analysis against real gradients. The results seem promising on the tested scenarios.

Critical weaknesses
  1. The runtime & memory consumption of HiZO is about 1.5x of MeZO. However, that comparison is not available for AGZO. This is a critical missing baseline.

  2. Lack of experiments on large model and data. For ZO methods, the main claim is savings on memory cost, compared with FO methods. This is more of a problem for large models. For tested models (Roberta-large, Qwen3-1.7B), practically, there's not much motivation to use ZO methods than FO methods. Would like to see some experiments on larger models at 8B scale. Another aspect is what happens for long context models. Since the computation now relies on activation, requiring storing such for long context is somtimes out-weights the model itself.

  3. A strong baseline is missing. It compares with AGZO and also outperforms it. Dong, Xun, et al. "ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces." arXiv preprint arXiv:2607.01125 (2026).

Questions for authors
  1. What's the wall clock and memory consumption of AGZO?

  2. What's the wall clock and memory consumption for Qwen3 1.7B model?

  3. Can the authors test on 8B-scale models?

  4. How does it work under long context? What exactly needs to be stored? In MeZO, memory savings come from storing the random seed, instead of perturbation vector. But for HiZO, it seems the covariance needs to be stored?

  5. Can the author compare with ZO-Act?

1 comment

Thank you for highlighting the missing AGZO profile and for asking how the method behaves on larger models and longer sequences.

Q1–Q2 and W1 — Runtime and memory. We profiled Llama3-8B on RTE using a single A800 80 GB GPU. AGZO was close to MeZO on both measures. Subsampling reduced HiZO's per-step time, and the separable variant reduced its memory overhead in this profile.

MethodSeconds per updateReported GPU memory (GB)
MeZO1.4816.1
AGZO1.4616.1
HiZO2.3431.1
S-HiZO1.7431.1
SA-HiZO2.2216.1

Q3, Q5, and W2–W3 — 8B evaluation and ZO-Act. In addition to the RTE profile, we ran MeZO, AGZO, HiZO, and our reproduction of ZO-Act on SST-2 with Llama3-8B. The evaluation loss at 3,000 updates was 0.335 for HiZO, 0.383 for MeZO, and about 0.442 for each activation-guided baseline. Our ZO-Act run used learning-rate warmup. This comparison covers convergence on one task. In the eight-direction configuration, ZO-Act uses nine forward evaluations per update, while HiZO uses two per minibatch. In their Llama3-8B RTE experiment, Dong et al. report 42.0 minutes and 16.1 GB for ZO-Act with a budget of 4,500 forward queries. This corresponds to an average of 5.04 seconds per optimizer step. We are running more experiments on Llama3-8B and will share the results when they are available.

Q4 — What is stored for long sequences? HiZO regenerates the perturbation from the random seed and the cached diagonal variance weights. For an m×nm\times n weight matrix, these weights form an m×nm\times n map. Its size is independent of sequence length. HiZO computes the statistics during the forward pass, and the activation workspace and covariance computation can still grow with the number of tokens. S-HiZO uses a fixed number of activation rows for the covariance product and retains the dense map. SA-HiZO stores only O(m+n)O(m+n) persistent guidance values per matrix, with temporary dense buffers still needed in the current implementation.

Reviewer_ERCzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HiZO, a zeroth-order optimization method that uses the magnitude of centered input–output activation covariance to allocate perturbation variance across model parameters. The theoretical analysis shows that convergence improves under covariance-gradient alignment. The experiments demonstrate faster optimization than MeZO (in optimization steps) on roberta-large, and the shuffled-guidance ablation supports the importance of Hebbian information.

Critical strengths
  1. The paper proposes using input–output activation covariance to guide ZO perturbations, and it provides theoretical analyses of convergence.
  2. The ablation studies on shuffled Hebbian scores, normalized vs. unnormalized guidance, and weight-only vs. input-covariance-only guidance are well designed.
Critical weaknesses
  1. The paper lacks citations and comparisons with several relevant methods. For example,
  • Second-Order Fine-Tuning without Pain for LLMs: A Hessian Informed Zeroth-Order Optimizer, which uses diagonal Hessian information to adapt ZO updates
  • AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments, which uses moment information
  • ConMeZO: Adaptive Descent-Direction Sampling for Gradient-Free Finetuning of Large Language Models, which samples directions from a cone centered on a momentum-based direction estimate

Other zeroth-order methods take (almost) the same memory as MeZO, such as

  • Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures, which uses low-rank perturbations
  • Zeroth-Order Sharpness-Aware Learning with Exponential Tilting, which reweighs perturbations with loss values
  • Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning, which selects a subset of parameters

To be more convincing, the authors could compare with more baselines on both accuracy and memory.

  1. I am concerned about evaluation: The optimization speedup is measured in optimization steps rather than wall-clock time, and the reported wall-clock speedups are estimates. Also, the datasets the paper evaluates on are SST-2, SNLI, COPA, and SQuAD only, which are much narrower than those in other zeroth-order papers. Please evaluate on more datasets, at least as MeZO does.

If my concerns with experiments are addressed, I would love to increase my score.

Questions for authors

The convergence analysis assumes a uniformly positive covariance–gradient alignment margin, but the experiments report average alignment values. Could the authors show the distribution of alignment ratios across training iterations and layers?

2 comments

Thank you for the careful review and for suggesting a closer look at the alignment distribution. We agree that the average alone leaves out important variation.

Alignment over training and across layers. We measured Hebbian alignment through step 3,000 while fine-tuning Llama3-8B on SST-2. The isotropic reference is 1. Across the 300 checkpoints, the model-wide median was 2.33, the middle 50% ranged from 2.17 to 2.51, and the minimum was 1.70. The lowest ratio observed for any reported layer at any checkpoint was 1.05. The heatmap and per-layer interquartile ranges show that the first transformer block and the language-model head had the highest median alignment. Together, the first block and the language-model head account for about 10% of the model's parameters and accounted for about 75% of the model-wide alignment gain above the isotropic baseline, averaged across checkpoints. All observed ratios were above the isotropic reference. The theorem requires a uniform margin along the analyzed trajectory, which is a stronger condition.

We also ran an ablation with Hebbian guidance restricted to those two parts, using isotropic perturbations elsewhere. At 3,000 updates, evaluation loss was 0.372 with the restricted guidance, 0.383 for MeZO, and 0.335 for full HiZO. In this run, guiding the two highest-scoring parts captured some of the improvement, and applying guidance more broadly performed better.

Timing, memory, and task coverage. We agree that the time-to-target estimates in the submission need direct validation. On RTE with Llama3-8B, full HiZO took 1.58 times as long per update as MeZO and used roughly twice as much GPU memory in our profile. Activation subsampling reduced the time overhead, and the separable variant brought reported memory back to MeZO's level in this profile. These measurements describe the cost of an update. Direct elapsed time to the same validation target remains to be measured. The new runs add RTE profiling and SST-2 convergence at 8B. We agree that a broader task suite, such as MeZO's, is needed to assess how well these results generalize.

Related work and baselines. Thank you for the specific references. Appendix A already discusses HiZOO's diagonal curvature, LOZO's low-rank perturbations, and Sparse MeZO's parameter selection. We will make those distinctions clearer and add AdaMeZO's moment estimates, ConMeZO's momentum-guided cone sampling, and the loss-based perturbation weighting in zeroth-order sharpness-aware learning with exponential tilting. AGZO is our primary comparison for activation-based guidance. We also ran a ZO-Act reproduction on 8B SST-2. We agree that these address only part of the request for head-to-head accuracy and memory comparisons with other memory-efficient ZO methods.

Reviewer_ERCzOpenReview ↗

Reply to Rebuttal

Thanks for the rebuttal. The revision's experimental setup still seems to list only roberta-large, OPT-1.3B, and Qwen3-1.7B, and compares primarily with MeZO and AGZO. I appreciate the authors' acknowledgment that wall-clock measurements, broader task coverage, and comparisons with other ZO optimizers remain incomplete. These are central to my original concerns, so I prefer to retain my score.

Reviewer_k4vrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes Hebbian-informed Zeroth-Order Optimization (HiZO) for efficient fine-tuning of large language models (LLMs). Unlike MeZO, which employs isotropic Gaussian perturbations, HiZO adjusts parameter-wise perturbation variances based on centered input-output covariance statistics computed from forward activations. To reduce computational and memory overhead, the authors further introduce two efficient variants: S-HiZO, which employs activation subsampling, and SA-HiZO, which uses a separable approximation. Theoretically, the paper derives a convergence bound based on the alignment between the perturbation covariance and the true gradient, and establishes sufficient conditions under which Hebbian covariance aligns with gradient magnitudes in a single-layer setting. Experiments report improvements in optimization efficiency and downstream task performance across several language models.

Critical strengths

None

Critical weaknesses
  • Unconvincing motivation and limited practical significance

    • The paper aims to enable efficient LLM fine-tuning but does not provide a convincing motivation for adopting zeroth-order (ZO) optimization in contemporary LLM fine-tuning scenarios. ZO optimization is particularly meaningful when true gradients are inaccessible, such as in black-box optimization or optimization with non-differentiable objectives. However, this work focuses on conventional supervised fine-tuning, where gradient information is readily available. Given the extensive development and widespread adoption of efficient first-order fine-tuning methods, demonstrating improvements over MeZO alone is insufficient to establish the practical significance of the proposed approach.

    • The experimental settings are also limited and do not adequately reflect contemporary LLM fine-tuning scenarios. The evaluations are conducted on RoBERTa-large, OPT-1.3B, and Qwen3-1.7B, using tasks such as SST-2, SNLI, COPA, and SQuAD. These settings are insufficient to demonstrate the practical relevance of HiZO for modern LLM fine-tuning workloads. In particular, the paper does not report the performance of pretrained checkpoints or include first-order fine-tuning baselines. Without these comparisons, it is difficult to assess whether the proposed method delivers meaningful improvements over the pretrained models or offers a competitive alternative to established fine-tuning approaches.

  • Insufficient experimental validation and baseline comparisons

    • Despite the inherently stochastic nature of ZO optimization, the paper reports only single-run results without repeated trials, variance estimates, or error bars. Considering that the reported improvements in downstream performance are often only a few percentage points, the current results do not provide sufficient evidence that these improvements are statistically robust or reproducible.

    • Comparisons with existing ZO optimization methods are also limited. Although numerous variants and extensions of MeZO have been proposed, AGZO is the only additional ZO method included in the comparisons. Furthermore, AGZO is evaluated only in terms of downstream accuracy and is excluded from the convergence and computational efficiency comparisons. Consequently, it remains unclear whether HiZO offers meaningful advantages over existing ZO approaches, particularly in terms of its claimed optimization efficiency.

    • The claimed "3× faster" convergence is also not sufficiently substantiated. This result is measured by the number of optimization steps required to reach a manually selected validation-loss threshold relative to MeZO. The corresponding training-time speedup is then estimated by combining this step reduction with per-step runtime measurements, rather than demonstrated through direct end-to-end training-time comparisons. Such a limited evaluation does not convincingly establish practical improvements in training efficiency.

  • Insufficient justification of the central design choice

    • The central contribution of HiZO is the use of Hebbian covariance as a parameter-wise importance measure to guide perturbation variances. However, the paper does not sufficiently justify why this statistic should be more effective than simpler alternatives. For a linear layer, the proposed covariance can be expressed in terms of (WΣxW\Sigma_x), which does not necessarily capture the importance of individual parameters with respect to the true task gradient. In particular, gradients in deep neural networks depend not only on forward activation statistics but also on the downstream objective and backpropagated error signals. Therefore, the connection between Hebbian covariance and task-relevant parameter importance remains unclear.

    • The paper provides an alignment ablation comparing Hebbian guidance against weight-only and input-covariance-only alternatives. However, this analysis is restricted to a small MLP and does not examine their relative optimization performance in actual LLM fine-tuning experiments. As a result, it remains unclear whether Hebbian covariance offers meaningful benefits over simpler statistics, such as weight magnitudes or activation statistics, that may require less computational and memory overhead.

    • Fundamentally, HiZO applies diagonal anisotropic Gaussian perturbations within an existing ZO optimization framework. Therefore, the effectiveness of the proposed method largely depends on whether Hebbian covariance provides a useful parameter-wise importance measure. Given that this choice constitutes the central methodological contribution, the current empirical evidence is insufficient to establish its necessity or superiority over alternative perturbation guidance strategies.

  • Limited theoretical novelty and relevance

    • It is unclear whether the theoretical analysis provides substantial new insights into the effectiveness of the proposed method. The main convergence result relies on the HCA condition, which assumes favorable alignment between the perturbation covariance and the true gradient. Under such an assumption, improved convergence behavior is largely expected and does not, by itself, establish an intrinsic advantage of Hebbian-informed perturbations. Moreover, the theoretical justification for the proposed Hebbian covariance is primarily developed in a simplified single-layer setting, leaving its relevance to deep LLMs unclear. Overall, the analysis mainly characterizes the convergence behavior when a suitably aligned perturbation covariance is available, rather than explaining why the proposed Hebbian guidance should reliably produce such a covariance in practice.
Questions for authors

See weaknesses.

2 comments

Thank you for taking the time to review our paper and for pressing us on the motivation, baselines, and scope of the theory.

Why use ZO for supervised fine-tuning? Gradients are available in our experiments, and efficient first-order methods are important baselines. We focus on settings where full-parameter adaptation is desirable and the memory cost of backpropagation is a constraint. For example, the MeZO paper reports that, in its single-GPU experiments on an A100 80 GB, MeZO could fine-tune OPT-30B and full-parameter Adam could fine-tune OPT-2.7B. Within that setting, we ask whether forward activations can make ZO perturbations more useful. Full HiZO adds time and memory overhead over MeZO, so the practical value depends on how much of the reduction in updates survives that overhead.

Baselines, scale, and repeated runs. Table 1 in the submission reports RoBERTa-large zero-shot accuracies of 79.0% on SST-2 and 50.2% on SNLI, and first-order fine-tuning accuracies of 91.9% and 77.5%, respectively. The table summarizes the ZO results across three seeds. For HiZO, accuracy was 93.8 ± 3.2% on SST-2 and 75.7 ± 1.3% on SNLI (mean ± sample standard deviation). We agree that further experiments are needed to assess how robust these gains are at larger scales. We are running experiments on Llama3-8B and will share the results when they are available.

What the speed claim measures. In Table 2 in the submission, MeZO took roughly three times as many updates as the HiZO variants to reach the shared, manually selected validation-loss thresholds on RoBERTa-large. Appendix E combines those counts with measured per-step costs to estimate training time. We agree that directly timing runs to a common target would give stronger evidence. We will describe the step-count result and the measured overhead explicitly wherever we discuss speed.

Why use Hebbian covariance? What does the theory establish? We agree that WΣxW\Sigma_x is a forward statistic and that the gradient also depends on the objective and downstream errors. The question is whether its magnitude provides a useful signal for allocating a fixed perturbation budget. In the MLP ablation, the full statistic aligned better than either weight-only or input-covariance-only guidance when evaluated at the same checkpoints and with matched total variance. This is evidence for combining the two sources of information in that setting. The theoretical contribution is to quantify how alignment and the Gaussian estimator's second moment jointly affect the convergence constant and to give sufficient conditions for alignment in a one-layer model with squared loss, where the learned weights are proportional to the task solution and the scores are nonuniform. The bound is conditional on alignment. Whether Hebbian guidance produces it in deep models is an empirical question.

Thank you for the response and clarifications.

My primary concern regarding the practical motivation and significance of ZO-based LLM fine-tuning remains unresolved. While I acknowledge the potential memory advantages of ZO optimization, the response does not convincingly establish the practical relevance of the proposed approach relative to established first-order fine-tuning methods, particularly in contemporary LLM settings. Furthermore, the response largely reiterates existing evidence without providing additional experiments to address my concerns regarding the advantages of Hebbian covariance over simpler guidance strategies, experimental robustness, and actual training efficiency. Therefore, I maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.