AGOZO: Activation-Guided Orthogonalized Zeroth-Order Optimization for LLM Fine-Tuning

Zeroth-order (ZO) fine-tuning reduces memory use by replacing backpropagation with forward evaluations, but limited queries produce noisy gradient estimates. We propose Activation-Guided Orthogonalized ZO (AGOZO), which combines activation-guided queries with gradient orthogonalization. AGOZO uses input activations to select a low-rank query subspace, estimates the gradient within that subspace, and replaces the estimate's nonzero singular values by one before mapping it back to the full weight matrix. Our theory quantifies how much gradient information this operation preserves. Assuming exact directional derivatives along Gaussian perturbations, we derive upper and lower bounds on the update's average alignment with the true gradient. These bounds agree up to constant factors when the subspace rank and query budget are small relative to the layer's output dimension. We also characterize when orthogonalization provides a stronger one-step loss-reduction guarantee than raw activation-guided ZO under a common smoothness assumption. This comparison considers growing matrix dimensions while keeping the number of queries fixed. We further quantify how sharing queries across layers weakens each layer's alignment. Across three language-model backbones and nine tasks, experiments with three seeds and a matched budget of 40,000 forward evaluations show that AGOZO achieves the highest reported mean validation score or a rounded tie on 16 of 27 model–task pairs.