MAPS: Momentum-Aware Perturbation Sampling for Zeroth-Order Optimization

Zeroth-order (ZO) optimization enables memory-efficient LLM fine-tuning through gradient estimation by forward-only function queries, but most existing methods sample queries based on random isotropic perturbations on parameters that ignore informative optimization states accumulated during training. We introduce **Momentum-Aware Perturbation Sampling (MAPS)**, a plug-and-play framework that uses momentum state to guide subsequent ZO queries. MAPS first calibrates the momentum state to the scale of isotropic perturbations and then mixes momentum-guided directions with stochastic exploration. It further balances the momentum-driven exploitation with random exploration through block-wise gating and temporal scheduling, controlling when and where the momentum-guided queries are applied. Our theoretical analysis establishes the convergence guarantee for MAPS. Since MAPS only modifies perturbation sampling without changing subsequent operations or introducing any additional function evaluation budget, it is generally applicable and extensively evaluated on various base ZO methods, foundation models and tasks. Our extensive experiments show that MAPS improves downstream performance and achieves lower training loss, demonstrating the effectiveness of reusing optimizer states in perturbation sampling.