acceptodds
Under review as a conference paper at ICLR 2027

LIZO: Enhancing Zeroth-Order LLM Fine-Tuning via Lateral Inhibition-Inspired Parameter Perturbation

Abstract

Fine-tuning is widely used to adapt large language models (LLMs) to downstream tasks, while zeroth-order (ZO) optimization provides a memory-efficient alternative through forward-only gradient estimation. However, existing ZO methods lack a principled characterization of parameter importance for guiding where perturbations should be introduced. In biological neural systems, lateral inhibition (LI) emphasizes the contrast between individual neuronal activity and its surrounding population. Inspired by this principle, we formulate an energy function to characterize the relative importance of individual parameters within their population and derive an LI score for parameter selection. Based on perturbation analysis, we show that parameters with higher LI scores are more sensitive to perturbations. We therefore propose Lateral-Inhibition-inspired Zeroth-Order Optimization (LIZO), which preserves parameters with higher LI scores and restricts perturbations and updates to parameters with lower scores. Extensive experiments on multiple LLMs and downstream tasks show that LIZO consistently improves over existing ZO methods across a range of settings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Zeroth-Order Optimization, LLM, Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XNfuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes LIZO, a sparse zeroth order method for LLM finetuning. The authors use normalized parameter values (based on the mean and std of all parameter values of the model ) as an importance metric. They then use the lower 25% of parameters based on the importance metric to perturb during zeroth-order finetuning. The paper proves that such low-scoring parameters exhibit smaller expected change of score when using Gaussian perturbation. Experimental results show LIZO improves over several ZO baselines in LLM finetuning (particularly in classification tasks) in the paper's evaluation settings.

Critical strengths

1- The importance metric used is simple and easy to implement.

2- The reported classification results are promising.

3- The mask swapping ablation is useful.

Critical weaknesses

1- The analysis of magnitude-base perturbation (Both figure 1 and the results provided in figure 5) is not convincing as the values apparently are for a single training seed and only one task is used (assuming figure 5 is also on SST-2. The authors don't mention what task figure 5 is about). The differences seem insignificant and more tasks and more training seeds are needed. Particularly for these small benchmarks that are very fast to train on.

2- The novelty appears limited. LIZO applies an existing energy-based score to ZO finetuning. The point of S-MEZO was to show that using sparse ZO finetuning helps and that a magnitude-based rule can be helpful. There are of course a number of other existing metrics that can be used for sparse ZO finetuning. This paper then uses the metric proposed in SimAM (which is cited). There are several other metrics from the pruning and sparsity literature that might outperform magnitude-based metrics, and applying an existing metric to sparse ZO finetuning does not by itself show a substantial contribution.

3- Experimental results are limited. Only two models and only one (if the same across all tasks) training seed and older benchmarks which might not be useful today. Also LIZO does not "consistently improves over existing ZO methods across a range of settings" as the abstract suggests (LIZO is outperformed by the baselines in open ended QA tasks ). Larger models with current important benchmarks (math, coding, ...) and more training seeds are needed.

4- The paper does not quantify how different its masks are from S-MEZO. Figure 6 is not enough. Since the proposed metric essentially becomes magnitude based if the mean of parameters is 0, what is driving the improvements over S-MEZO if the improvements persist with more training seeds.

Questions for authors

1- The appendix mentions results from existing papers are used. How have authors made sure that the evaluation data is identical in different methods? Are the evaluation settings identical?

2- "LIZO leads all seven Llama3-8B classification tasks" is incorrect. FZOO outperform LIZO on MultiRC. Also the bolded entry for that model-task is incorrect.

3- Is the mean and std computed over the entire model weights? Is bottom 25% over the entire model weights?

4- If like the paper claims for efficiency purposes all weights use the same mean and variance, equation 6 should produce at most two curves when plotted against absolute weight size. One for positive weights and one for negative weights. Why does figure 6 show a scatter? Were different means and variances used for different weights?

Reviewer_CzvfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose a new zeroth-order approach, LIZO, that improves parameter selection compared to prior works. Inspired by lateral inhibition, the authors introduce an importance score based on the deviation of individual parameters from the mean of their parameter population. A theoretical analysis is also provided, which shows that parameters with higher importance scores are more sensitive to random perturbations; it motivates to select the low-score parameters. Finally, the authors conduct experiments on two LLMs and several downstream tasks, reporting improvements over existing zeroth-order methods.

Critical strengths
  1. The paper presents a simple and intuitive approach to parameter selection in zeroth-order optimization. The proposed method is easy to implement and achieves consistent empirical improvements over several existing ZO baselines.

Additionally, I did not find any technical flaws in the proofs.

Critical weaknesses
  1. The proposed importance score is a monotonic function of the squared deviation from the population mean. Hence, selecting parameters with the lowest scores is equivalent to selecting parameters with the smallest absolute deviations from the mean. This makes the proposed method closely related to S-MeZO, with the main difference being the use of centered rather than absolute parameter magnitudes. The lateral inhibition formulation does not appear to introduce a fundamentally new selection mechanism.
  2. Although Proposition 1 establishes that selecting low-score parameters minimizes the expected squared change in the importance scores, it does not establish that this selection improves gradient estimation accuracy, reduces variance, or leads to faster convergence. Thus, the connection between the theoretical analysis and the empirical improvements remains unclear.
  3. Experiments do not include a random parameter selection baseline or an analysis of how the proposed masks differ from those of S-MeZO. Moreover, the main results are reported without standard deviations or confidence intervals across multiple random seeds. These additional experiments would help establish whether the observed improvements are statistically reliable and attributable to the proposed selection criterion. Furthermore, some baseline results are taken from the original papers, making it difficult to assess whether all methods were evaluated under strictly comparable experimental conditions.
Questions for authors
  1. Could the authors clarify what additional insight the lateral inhibition formulation provides beyond this simple centered-magnitude selection rule? In particular, under what conditions should this criterion significantly outperform the magnitude-based selection used in S-MeZO?
  2. The authors use a fixed parameter selection ratio of 25% across all experiments. How sensitive is the performance of LIZO to this choice across different models and downstream tasks? In particular, does the optimal selection ratio depend on the model architecture or the task?
Reviewer_4deCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LIZO, a zeroth-order LLM fine-tuning method that selects perturbation targets using a lateral-inhibition-inspired importance score. The score is derived from a least-squares energy between a parameter and its population, then approximated with global mean and variance (Eq. 6). Parameters in the lowest percentile of the score are perturbed for MeZO-style gradient estimation, while higher-score parameters are left fixed. The method is evaluated on Llama3-8B and OPT-13B across SuperGLUE and SQuAD/DROP against MeZO, FZOO, LOZO, and S-MeZO, with ablations on perturbation ratio and mask design.

Critical strengths

The motivation against pure magnitude-based selection (S-MeZO) is concrete: the random mask-swap experiment on SST-2 (Fig. 1) shows that magnitude alone is an unreliable perturbation criterion.

The paper gives a clear algorithmic story from an energy formulation to a practical mask (Algorithm 1), plus Proposition 1 linking higher LI scores to larger expected score distortion under perturbation.

Empirical gains over strong ZO baselines are consistent on both backbones, with average accuracy improvements of about 3.06 and 2.03 points on Llama3-8B and OPT-13B, and large F1 gains on QA tasks relative to S-MeZO.

Ablations are useful: bottom-25% vs top-25% LI selection (Fig. 4b), perturbation ratio (Fig. 4a), and mask-swap sensitivity (Fig. 5) support the design choice to perturb low-LI parameters.

GPU memory stays close to S-MeZO (Fig. 3), so the extra selection logic does not appear to break the memory advantage of sparse ZO fine-tuning.

Critical weaknesses

The biological lateral-inhibition narrative is only loosely tied to the actual criterion: after the local formulation in Eq. (3)–(5), LIZO uses a global z-score-style score (Eq. 6), which is closer to prior population-relative weighting (e.g., SimAM-style ideas) than to neighborhood inhibition in LLM weight space.

Proposition 1 analyzes sensitivity of the importance score, not sensitivity of the task loss or gradient quality; the step from “preserve high-LI parameters” to better ZO fine-tuning is motivated but not fully established theoretically.

The paper does not report variance over random seeds or statistical tests; with 1k (or fewer) training samples per task, it is hard to judge how stable the reported margins are.

Scale is limited to 8B/13B models; there is no evidence on larger models or on settings where the MeZO line is most stressed.

Baseline coverage omits some recent subspace/random-subspace ZO variants (e.g., SubZero), and there is no experiment combining LIZO with low-rank or faster ZO estimators (LOZO/FZOO).

Presentation has minor issues (e.g., “MultiiRC” in Fig. 3), and the conclusion does not discuss limitations or failure cases explicitly.

Questions for authors

Are LI scores computed per weight tensor (as in Algorithm 1) or with a single global statistic over all parameters? How sensitive are results to this choice?

How often is the perturbation mask refreshed during training, and what is the runtime overhead (beyond GPU memory in Fig. 3) relative to S-MeZO?

On OPT-13B DROP, LIZO F1 (30.25) remains far below Adam/LoRA; please comment on where LI-based selection fails.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.