- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces Displacement-Aware Subspace ZO-Muon (DAZM), a memory-efficient zeroth-order (ZO) fine-tuning method for large language models. DAZM adopts the method from SubZero by decomposing the perturbation into the form . It accumulates and retains and . The paper shows theoretically and empirically that DAZM reduces variance compared with existing ZO-Muon variants and yields better downstream performance across multiple LLMs and tasks.
Critical strengths
-
The paper follows the approach from SubZero: it decomposes the perturbation into the form , applies momentum accumulation to the low-dimensional , and then carries out Muon-based optimization. This design sounds well motivated.
-
Under the full fine-tuning setting, the paper validates the effectiveness of the proposed algorithm.
Critical weaknesses
-
Need sufficient experiments for the usefulness of low-rank subspace adaptation. Since the columns of and are high-dimensional vectors and are (almost) orthogonal, their average effect may be similar to that of random bases.
-
GPU memory usage seems hard to justify. The paper reports lower memory than MeZO and SubZero (e.g., and in the supplementary material). However, this is not intuitive: each step requires orthogonalization of , which may incur non-trivial computational and memory costs.
-
Question whether basis alignment is beneficial. Again, because and live in high-dimensional spaces and are nearly orthogonal, it is unclear whether aligning to these bases provides meaningful gains; this should be validated via ablation studies.
-
Fair comparison across different ZO algorithms requires equal query budget. Different zeroth-order methods use different numbers of function queries per update (e.g., multi-query vs. single-query, central-difference vs. forward-difference). Experiments should be compared under the same total number of queries, or at least under matched wall-clock time.
-
No LoRA is used. The proposed method performs full-parameter fine-tuning.
-
Mathematical notation is inconsistent. For example, the SVD is sometimes written as and sometimes with a transpose (e.g., ). This needs to be standardized.
Questions for authors
Please see the weaknesses listed above.