acceptodds
Under review as a conference paper at ICLR 2027

Displacement-Aware Subspace Adaptation for Muon-Style Zeroth-Order Fine-Tuning

Abstract

First-order fine-tuning of large language models (LLMs) is often constrained by the substantial memory overhead of backpropagation, whereas zeroth-order (ZO) fine-tuning offers a memory-efficient alternative based solely on forward evaluations. However, high-dimensional ZO estimates can be noisy, making it difficult to reliably exploit matrix structure through Muon-style spectral updates. Recent ZO-Muon methods alleviate this issue by restricting ZO estimation to low-rank search subspaces, but repeatedly refreshing these subspaces at random may discard directions that have consistently driven recent parameter updates. Consequently, subsequent Muon steps may operate in less informative search spaces, as polar normalization can only exploit directions already captured by the selected bases. To address this issue, we propose Displacement-Aware Subspace ZO-Muon (DAZM), which uses accumulated parameter displacements produced by recent Muon updates to adapt the low-rank search subspace. Specifically, we accumulate the realized Muon updates over each interval, allowing directions that are repeatedly reinforced along the optimization trajectory to become prominent in the resulting displacement. We then retain the leading left and right singular directions as trajectory-informed exploitation bases, while filling the remaining dimensions with random directions orthogonal to the retained subspaces to preserve exploration. Theoretically, we show that under the temporal persistence of the local optimization geometry, displacement-aware subspace adaptation retain more informative gradient structure for subsequent Muon updates than purely random refreshes. Extensive experiments on multiple LLMs and downstream tasks demonstrate that our method achieves consistently superior performance over existing ZO-Muon and non-Muon ZO baselines.

Primary area:
optimization
Keywords:
Fine-tuning, Muon, Zeroth-order Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FyvNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Displacement-Aware Subspace ZO-Muon (DAZM), a memory-efficient zeroth-order (ZO) fine-tuning method for large language models. DAZM adopts the method from SubZero by decomposing the perturbation into the form UΣVTU\Sigma V^T. It accumulates Σ\Sigma and retains UU and VV. The paper shows theoretically and empirically that DAZM reduces variance compared with existing ZO-Muon variants and yields better downstream performance across multiple LLMs and tasks.

Critical strengths
  • The paper follows the approach from SubZero: it decomposes the perturbation into the form UΣVTU\Sigma V^T, applies momentum accumulation to the low-dimensional Σ\Sigma, and then carries out Muon-based optimization. This design sounds well motivated.

  • Under the full fine-tuning setting, the paper validates the effectiveness of the proposed algorithm.

Critical weaknesses
  1. Need sufficient experiments for the usefulness of U/VU/V low-rank subspace adaptation. Since the columns of UU and VV are high-dimensional vectors and are (almost) orthogonal, their average effect may be similar to that of random bases.

  2. GPU memory usage seems hard to justify. The paper reports lower memory than MeZO and SubZero (e.g., MM and Ψ\Psi in the supplementary material). However, this is not intuitive: each step requires orthogonalization of OO, which may incur non-trivial computational and memory costs.

  3. Question whether basis alignment is beneficial. Again, because UU and VV live in high-dimensional spaces and are nearly orthogonal, it is unclear whether aligning to these bases provides meaningful gains; this should be validated via ablation studies.

  4. Fair comparison across different ZO algorithms requires equal query budget. Different zeroth-order methods use different numbers of function queries per update (e.g., multi-query vs. single-query, central-difference vs. forward-difference). Experiments should be compared under the same total number of queries, or at least under matched wall-clock time.

  5. No LoRA is used. The proposed method performs full-parameter fine-tuning.

  6. Mathematical notation is inconsistent. For example, the SVD is sometimes written as UΣVU\Sigma V and sometimes with a transpose (e.g., VTV^T). This needs to be standardized.

Questions for authors

Please see the weaknesses listed above.

Reviewer_PWp5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Displacement-Aware Subspace ZO-Muon (DAZM), a zeroth-order fine-tuning method that adapts low-rank search subspaces using accumulated parameter displacements. The method retains leading singular directions of the accumulated reduced-space updates and supplements them with random orthogonal directions to preserve exploration. The paper provides theoretical analyses of convergence and gradient-signal capture, together with experiments on Qwen2.5-7B and OPT-13B across several language understanding and reasoning tasks.

Critical strengths
  1. ZO-Muon relies on low-dimensional search subspaces, whereas purely random subspace refreshes do not explicitly exploit information about directions along which recent parameter updates have occurred. Using realized parameter displacements as a subspace prior is a natural, appealing design choice, and it requires no additional forward queries to obtain this information.

  2. DAZM performs the SVD on a small reduced displacement core and supplements the displacement-informed directions with an orthogonal complement for exploration. This design avoids explicitly forming or storing the full parameter displacement matrix, making it well suited to memory-constrained fine-tuning settings.

  3. DAZM uses displacement-derived directions as exploitation bases to improve gradient-signal capture in the low-dimensional search subspace. Figure 4 provides a useful diagnostic by comparing the certified capture lower bound with a random-subspace benchmark, offering some insight into the proposed mechanism beyond downstream task performance alone.

Critical weaknesses
  1. Although the convergence analysis introduces terms for zeroth-order estimation error, Newton–Schulz approximation error, and momentum-transport mismatch across subspace refreshes, the conditions required for these terms to yield a meaningful guarantee for the practical DAZM algorithm remain insufficiently clear. The authors should clarify the assumptions underlying the finite-query estimation error bound and explain how the resulting error terms affect the convergence guarantee under the implemented subspace adaptation and approximate Muon updates.

  2. The theoretical capture bound depends on the alignment between historical displacement directions and the relevant gradient subspace, so it does not generally guarantee superiority over random subspaces. While Figure 4 provides a useful diagnostic of the certified lower bound, more direct evidence of actual gradient capture would strengthen the central claim. It would also be useful to investigate whether improvements in gradient capture correlate with downstream task performance.

  3. The existing ablations provide useful evidence for the adaptive subspace mechanism, but they do not fully isolate the contribution of accumulated displacement information. Comparisons with single-step or disrupted displacement histories would help establish the source and robustness of DAZM's gains.

  4. Although the paper reports GPU memory consumption and wall-clock training time, the additional costs of SVD, orthogonalization, and periodic subspace refresh are not sufficiently isolated. A more detailed breakdown of subspace-adaptation overhead, together with performance comparisons under matched wall-clock budgets, would better establish DAZM's practical efficiency relative to ZO-Muon and other baselines.

Questions for authors

See weaknesses.

Reviewer_uYhuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Displacement-Aware Subspace ZO-Muon (DAZM), which accumulates Muon parameter displacements over each subspace refresh interval and extracts their leading singular directions to construct the search subspace for the next stage, while incorporating orthogonal random directions to maintain exploration. The method performs displacement accumulation and subspace adaptation entirely in reduced coordinates without requiring additional function evaluations.

The theoretical analysis examines the ability of search subspaces to capture gradient information and establishes a lower bound on gradient capture when the retained displacement directions remain aligned with the subsequent gradient structure. Experiments are conducted on Qwen2.5-7B and OPT-13B across multiple language understanding and reasoning tasks. The results show that DAZM achieves better average performance than existing ZO-Muon and other zeroth-order optimization methods. The authors further validate the effectiveness of the proposed method through ablation studies, hyperparameter sensitivity analysis, and computational resource comparisons.

Critical strengths
  1. The paper addresses the quality of search subspaces in low-rank ZO-Muon optimization. Although random subspaces can reduce the effective dimension of zeroth-order gradient estimation, they fail to exploit information accumulated during previous optimization steps. The authors extract consistently reinforced update directions from historical parameter displacements and combine them with random exploration to construct new search subspaces. This design naturally leverages information already available along the optimization trajectory without requiring additional function evaluations, offering practical value in terms of memory efficiency, computational efficiency, and optimization accuracy.

  2. The authors conduct extensive evaluations on Qwen2.5-7B and OPT-13B across multiple downstream tasks, comparing DAZM with methods such as MeZO, SubZero, LOZO, LOZO+, and ZO-Muon. DAZM achieves average scores of 83.37 and 70.41 on the two models, outperforming ZO-Muon by 1.81 and 2.33 points, respectively. In addition, the paper includes ablation studies with Frozen and Random subspace strategies to validate the effectiveness of adaptive subspace updates, along with analyses of training time, GPU memory consumption, and hyperparameter sensitivity. These experiments provide solid empirical support for the effectiveness of the proposed method.

Critical weaknesses

The paper conducts ablation studies using Frozen and Random strategies, showing that both keeping the search subspace fixed and refreshing it entirely at random perform worse than DAZM. However, DAZM introduces two changes simultaneously: retaining some directions from the previous search subspace and selecting which directions to retain based on accumulated displacements. Therefore, the current ablations do not sufficiently isolate how much of the performance improvement comes from the directional information provided by historical displacements.

Furthermore, why do DAZM and ZO-Muon use different search rank ranges in Appendix Table 3? Could the choice of search rank itself directly affect gradient capture and optimization behavior?

Questions for authors

Could the authors further distinguish the benefits of subspace retention itself from those of displacement-aware direction selection? For example, comparing DAZM with a strategy that randomly retains a subset of directions from the previous search subspace, under the same retention dimension p, total search rank r, and computational budget, would help clarify the unique contribution of historical displacement information.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.