LIZO: Enhancing Zeroth-Order LLM Fine-Tuning via Lateral Inhibition-Inspired Parameter Perturbation

Fine-tuning is widely used to adapt large language models (LLMs) to downstream tasks, while zeroth-order (ZO) optimization provides a memory-efficient alternative through forward-only gradient estimation. However, existing ZO methods lack a principled characterization of parameter importance for guiding where perturbations should be introduced. In biological neural systems, lateral inhibition (LI) emphasizes the contrast between individual neuronal activity and its surrounding population. Inspired by this principle, we formulate an energy function to characterize the relative importance of individual parameters within their population and derive an LI score for parameter selection. Based on perturbation analysis, we show that parameters with higher LI scores are more sensitive to perturbations. We therefore propose Lateral-Inhibition-inspired Zeroth-Order Optimization (LIZO), which preserves parameters with higher LI scores and restricts perturbations and updates to parameters with lower scores. Extensive experiments on multiple LLMs and downstream tasks show that LIZO consistently improves over existing ZO methods across a range of settings.