- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper uses a ZO optimizer's existing momentum to guide subsequent random perturbations, and the experiment section evaluates the proposed method on up to 30B models.
Critical strengths
The paper evaluates the proposed method on 10 tasks, multiple model architectures, and model sizes up to 30B.
Critical weaknesses
- Using momentum-like information to guide ZO perturbations is not fundamentally new. For example, ConMeZO (2026) samples perturbations from a cone centered on a momentum-based direction. The paper acknowledges closely related work, such as ZO-AdaMU and Guided Evolutionary Strategies, but its distinction from existing momentum-guided sampling methods needs further clarification and experimental demonstration.
- The convergence shows that MAPS converges, but it is not shown to converge faster than LOZO-M. It also seems that the displayed upper-bound constants become less favorable as the guidance parameter increases.
Questions for authors
-
Since MAPS's improvements are generally modest, could the author supply multiple-seed results to showcase the statistical significance of performance improvements?
-
The theoretical convergence bound becomes looser with increasing momentum guidance, while the empirical results suggest that momentum guidance improves optimization. Could the authors measure the alignment between the accumulated momentum and the true gradient during training, and provide theoretical or empirical evidence connecting this alignment to the observed convergence improvements?
7 comments
-
MAPS guides selected parameter blocks at selected training steps
Section 4.2 introduces two gates. The temporal gate determines when to use momentum guidance. During warmup, all blocks use the base optimizer’s random sampler. After warmup, each step activates guidance with probability . At an active step, the block-wise gate applies guidance to a randomly selected subset of eligible blocks. The parameter controls the fraction selected. The remaining blocks use random perturbations. At an inactive step, all blocks use random perturbations.
ConMeZO uses the cone angle to control directional bias. Its warmup schedule adjusts the momentum coefficient used to estimate the cone axis. ZO-AdaMU anneals , , and to control perturbation sampling and step size. The two MAPS gates determine whether a given block uses a guided or random perturbation at the current step.
-
Multiple-seed results on OPT-1.3B and the comparison on OPT-13B
We compared three methods on WiC under a common training protocol. All methods fine-tuned OPT-1.3B on the same GPU with the same data splits and evaluation protocol. We used a batch size of 16 and trained for 10K steps. The table reports accuracy (%) for each training seed and the sample standard deviation across seeds.
Method Run 1 Run 2 Run 3 Mean ± SD MAPS-LOZO-M 59.72 57.37 58.93 58.67 ± 1.20 ZO-AdaMU 57.68 57.99 55.64 57.11 ± 1.28 ConMeZO 56.90 58.93 57.21 57.68 ± 1.10 MAPS-LOZO-M achieves the highest mean accuracy. Using unrounded scores, its mean gain is 0.99% over ConMeZO, and 1.57% over ZO-AdaMU.
We further compare the reported OPT-13B results across seven shared tasks. For each method, we use its best reported results under the experimental protocol and hyperparameter configuration.
Method SQuAD SST-2 WiC BoolQ ReCoRD RTE MultiRC MAPS-LOZO-M 84.1 93.3 61.6 71.2 82.0 74.0 64.1 MAPS-ZO-Adam 85.1 93.2 63.8 73.6 81.2 74.7 65.7 ZO-AdaMU 82.4 92.1 60.7 73.0 83.0 72.9 63.0 ConMeZO 83.7 92.4 58.3 69.3 80.9 64.5 58.3 Overall, the multi-seed results show that MAPS-LOZO-M achieves the better performance among the compared methods on OPT-1.3B. Moreover, the OPT-13B comparisons demonstrate consistent competitive performance across seven tasks: MAPS-LOZO-M outperforms ConMeZO on all seven tasks and ZO-AdaMU on five, while MAPS-ZO-Adam outperforms ZO-AdaMU on six. Notably, MAPS uses only 20K training steps, compared with the larger training budget 100K used by ZO-AdaMU. These results further support the effectiveness and applicability of MAPS across different model scales and optimizer configurations.
R3-W2 & R3-Q2
We thank the reviewer for this suggestion. Our main theorem shows that MAPS-LOZO-M retains the convergence rate of LOZO-M under the stated assumptions. This guarantee holds for arbitrary orientation of the historical momentum relative to the current gradient, including when they are orthogonal. Importantly, the theorem establishes the same asymptotic convergence order, rather than a faster convergence guarantee than LOZO-M. The effect of momentum guidance on practical optimization progress depends on whether the historical momentum provides useful directional information.
The convergence analysis in the appendix gives an exact expression for . Applying this identity to the current subspace gradient shows when guidance from , the accumulated ZO momentum transported to the current subspace, can be beneficial. Specifically, if then the squared projection of the gradient onto the momentum direction exceeds its expected squared projection onto an isotropically sampled random unit direction in the dimensional subspace. Under this condition, momentum guidance increases the expected squared directional derivative relative to Gaussian sampling, providing a more informative perturbation direction and making the descent term in the one-step bound more favorable, all else being equal.
The main theorem does not require this condition to hold. Instead, its proof uses the uniform lower bound which remains valid regardless of the momentum orientation, including when momentum is orthogonal to the current gradient. This conservative bound does not exploit the additional directional benefit when historical momentum is well aligned with the current gradient. As increases, the lower bound becomes weaker, resulting in a looser convergence upper bound. This reflects the worst case nature of the theoretical guarantee, rather than implying that stronger momentum guidance necessarily harms practical optimization. The empirical benefit depends on the directional information provided by historical momentum along the actual training trajectory.
The practical benefit of momentum guidance also depends on its strength. Table 9 illustrates this effect. At a fixed block ratio of , the highest WiC accuracy is obtained with , while the lowest training loss was obtained with . Increasing guidance strength further did not consistently improve either metric, suggesting that excessively strong guidance can limit the benefit of random exploration.
To examine the directional alignment during training, we extracted the accumulated zeroth-order momentum available before the current update at different OPT-1.3B/WiC training checkpoint. For each evaluation minibatch, we computed the first-order reference gradient by backpropagation and measured its signed cosine similarity with the historical momentum, . We also compared the alignment of the standard ZO and MAPS gradient estimates, and , using the same setting. Run 1 and Run 2 evaluate the same training checkpoints using different evaluation samples and perturbations. Cosines are reported in units of . Small cosine values are plausible in a high-dimensional parameter space, so we focus on relative differences in alignment. Gain is the percentage increase in average cosine for MAPS relative to standard ZO within each run.
| Run 1 | Run 2 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Step | Gain | Gain | Mean gain | ||||||
| 525 | −0.71 | 2.31 | 2.36 | +2.22% | +0.04 | 2.25 | 2.27 | +1.27% | +1.75% |
| 1025 | +1.15 | 2.12 | 2.18 | +3.12% | −0.72 | 1.97 | 2.06 | +4.67% | +3.90% |
| 1525 | −0.49 | 2.43 | 2.45 | +0.63% | −1.04 | 2.24 | 2.30 | +2.95% | +1.79% |
| 2025 | +3.10 | 2.26 | 2.36 | +4.30% | +0.86 | 2.23 | 2.26 | +1.24% | +2.77% |
| Mean | +0.76 | 2.28 | 2.34 | +2.57% | −0.22 | 2.17 | 2.22 | +2.53% | +2.55% |
The results show that the alignment between ZO accumulated momentum and the current gradient is not stable. Both the magnitude and sign of this alignment vary across checkpoints and evaluation samples. At step 1525, the momentum cosine is negative in both runs, yet incorporating momentum improved gradient estimate alignment by 0.63% and 2.95%, respectively. MAPS mixes normalized momentum into selected parameter blocks, and the two-point finite-difference estimator is used to estimate the gradient. Consequently, alone does not determine whether guidance improves the estimated direction. Across these results, MAPS improved gradient estimate alignment relative to standard ZO by an average of 2.57% and 2.53%. These results provide empirical evidence that momentum-guided sampling can improve estimator alignment, although the magnitude of the improvement varies across training states.
This alignment improvement also provides an explanation for the observed optimization progress. At a given checkpoint and for a fixed step size along the negative normalized gradient estimate, closer alignment with the gradient implies a larger first order predicted decrease in the same minibatch loss. This is consistent with the lower training loss observed with MAPS under the same query budget in Figure 1.
Thank you for highlighting the breadth of our evaluation and raising specific questions about the method. We address the concerns below, focusing on clarifying our contributions, theoretical analysis, and experimental evidence.
R3-W1 & R3-Q1
Thank you for the suggestions. In addition to the previously discussed ZO-AdaMU and Guided Evolutionary Strategies, we also consider the recently proposed ConMeZO. We clarify the methodological differences and provide additional experimental evidence below.
Relationship to and Differences from Existing Guided Sampling Methods.
ZO-AdaMU jointly designs the perturbation history state, sampling mechanism, and adaptive parameter updates. Its Eq. (4) combines two Gaussian samples, one centered at zero and the other centered at the historical perturbation state, to update the perturbation history.
z˙t+1∼N(0,αt+1),z¨t+1∼N(mt,1−αt+1),
mt+1=β1z˙t+1+(1−β1)z¨t+1.
Here, mt is its perturbation history state. ZO-AdaMU also constructs an adaptive scaling term from the squared samples. Its simulated annealing schedule controls αt, β1,t, and β2,t. These mechanisms form a dedicated ZO optimizer[1].It can be viewed as a specific improved version of zeroth-order Adam. MAPS directly reuses the directional state already maintained by the base optimizer, without introducing a separate perturbation history recurrence.
ConMeZO obtains its momentum direction from an exponential moving average of past ZO gradient estimates[2],
mt+1=βmt+(1−β)g(xt,zt).
ConMeZO then uses the normalized momentum direction as the axis of its cone sampler. This concentrates subsequent perturbations around that direction in the high-dimensional parameter space. MAPS directly reuses the momentum already maintained by the base ZO optimizer, such as the first moment of ZO-Adam or the low-rank momentum of LOZO-M. This avoids maintaining a separate state to track past ZO gradient estimates for sampling.
Guided ES relies on additional surrogate gradients as guidance signals, using them to construct a guiding subspace and modify the covariance of the search distribution[3]. MAPS follows a different design. MAPS does not introduce an additional surrogate model or an independent guiding subspace. It directly uses the momentum state already maintained by the base optimizer to construct guided perturbations. This allows historical optimization information to be incorporated into subsequent sampling without additional gradient computations or function evaluations.
MAPS uses the base optimizer states to guide sampling in its native perturbation space
ZO-AdaMU maintains a history of sampled perturbations. ConMeZO uses a moving average of ZO gradient estimates to define its cone axis. MAPS takes the momentum state P already available in the base optimizer and mixes it with a random perturbation R in the same representation. Eq. (5) defines the guided perturbation as $\widetilde{\mathbf{P}} = \sqrt d\,\frac{\mathbf{P}}{\max\{\|\mathbf{P}\|_F,\epsilon\}}, ~ \mathbf{S} = \sqrt{1-\lambda}\,\mathbf{R} + \sqrt{\lambda}\,\widetilde{\mathbf{P}}, ~ \mathbf{R}\sim\mathcal N(\mathbf{0},\mathbf{I}_d).$ And the guided perturbation has the same expected squared norm as the original Gaussian perturbation. This sampling mechanism can be instantiated with different optimizers by reusing their momentum states and perturbation representations. MAPS-ZO-Adam uses the existing first moment to guide the dense perturbation. ZO-Adam then applies its standard first-moment, second-moment, and parameter updates. We also evaluate this sampler in the LoRA setting. MAPS-LOZO-M uses the existing low-rank momentum N to guide the left factor U of the perturbation UV⊤.