- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SubZero+, a memory-efficient adaptive zeroth-order (ZO) fine-tuning method for large language models that operates in periodically refreshed random low-rank subspaces. The method combines two key ideas: (i) multi-query forward-difference averaging to reduce ZO estimator noise in a small coefficient space, and (ii) running Adam entirely on r×r coefficient matrices with per-block restarts when subspaces refresh, thus avoiding O(d) optimizer states. Theoretical results in the exact-directional limit show that multi-query averaging preserves conditional unbiasedness while reducing covariance, MSE, and squared-write inflation by exactly 1/K in coefficient space. Experiments on SuperGLUE with models from 1.3B to 32B parameters under full-parameter and LoRA fine-tuning demonstrate consistent improvements over strong ZO baselines at near-inference-level memory.
Critical strengths
-
Introduces a dual low-dimensional design: multi-query averaging and Adam on r×r coefficient matrices with periodic basis restarts, enabling adaptive ZO without O(d) memory.
-
Provides targeted theory (Theorems 1 and 2) that directly characterizes how multi-query averaging affects first and second moments in coefficient space, especially the squared-write inflation that is critical for Adam’s stability.
-
Practical algorithmic details (sign-corrected QR, local bias correction within refresh blocks, seed-based perturbation regeneration) are thoughtful and reproducible.
Critical weaknesses
-
Insufficient comparison with closely related published work. AGZO (ICML 2026) exploits activation-informed subspaces, directly challenging SubZero+'s reliance on random bases, yet is neither cited nor compared; MUZO(Peng et al., EMNLP 2025), which the paper identifies as the most directly comparable multi-query adaptive baseline, is discussed only in a memory comparison, while FZOO (ICLR 2026) is not evaluated.
-
The central “dual low-dimensionality” claim is not adequately isolated. The paper lacks controlled performance ablations comparing coefficient-space SGD with (K=99) against coefficient-space Adam with (K=1), so the reported gains cannot be attributed specifically to the proposed combination rather than multi-query averaging alone.
-
The robustness of the reported gains is uncertain, and the default query count is insufficiently justified. The paper reports no multi-seed uncertainty estimates, while its highly non-monotonic query-count ablation achieves its best average at (K=50), rather than the default (K=99), and covers only OPT-1.3B on three tasks.
-
Theoretical and practical claims exceed the available evidence. The theoretical results focus on conditional single-step identities in the exact-directional limit without quantifying finite-(\epsilon) bias or establishing convergence and stability for the complete algorithm with periodic moment resets; meanwhile, despite using 99 perturbation queries plus a baseline evaluation per update, the paper reports no wall-clock throughput or time-to-quality measurements.
reference:
-
AGZO: Activation-Guided Zeroth-Order Optimization for LLM Fine-Tuning Wei Lin, Yining Jiang, Qingyu Song, Qiao Xiang, Hong Xu ICML 2026
-
MUZO: Leveraging Multiple Queries and Momentum for Zeroth-Order Fine-Tuning of Large Language Models Yuezhang Peng, Yuxin Liu, Fei Wen, Xie Chen 2025.emnlp-main.432
-
FZOO: FAST ZEROTH-ORDER OPTIMIZER FOR FINE-TUNING LARGE LANGUAGE MODELS TOWARDS ADAM-SCALE SPEED Sizhe Dang , Yangyang Guo , Yanjun Zhao, Xiaodong Zheng ICLR 2026
Questions for authors
same as weakness
1. AGZO, MUZO, FZOO and related work (Related Work, Table 4)
We added AGZO, MUZO-Adam, FZOO and other adaptive baselines, and the revision now states each method's design focus. AGZO builds activation-guided subspaces and may complement SubZero+. MUZO pairs multi-query estimation with adaptive updates. FZOO pairs normalization with batched execution. SubZero+ designs estimation and update together: it averages multiple queries and keeps Adam states in the same two-sided random coefficient space. Our contribution is this joint design and its gains, especially under LoRA. Single-seed accuracy (%):
2. Separating multi-query estimation from coefficient-space Adam (Table 5)
We crossed K with the matrix-coefficient optimizer (SGD, SGDM or Adam) and shared all other settings: two-sided subspace, forward baseline, rank 16, ϵ=10−3, Adam for 1-D parameters, the same refresh schedule, a separate learning-rate search per configuration, and about 40K queries. Setup: OPT-1.3B/SST-2/LoRA, 3 seeds.
At K=99, Adam beats SGD and SGDM, and it gains 2.8 points over K=1.
3. Default K (original Table 5, now Table 8)
We found a flaw in the original learning-rate search and reran the experiment (OPT-1.3B/SST-2/FT, rank 24, seed 0, about 40K queries). Each K now has its own learning-rate search, with baseline calls counted. All experiments use K=99; we did not tune it per task.
4. Scope of the theory and empirical error (Table 11)
Our 1/K identities hold only in the exact-directional limit, with parameters, minibatch and subspace fixed. They do not show that full Adam with finite ϵ is unbiased, stable or convergent. With finite ϵ, averaging lowers variance but does not remove bias, and the revision now says so.
We also ran a diagnostic at three checkpoints of one OPT-1.3B/SST-2/FT run. We used 30 repeats per K over 146 matrix layers and compared against the projected backpropagation (BP) gradient. BP is used only for this diagnostic, never in training. Going from K=1 to K=99:
5. Training time and resources (Tables 12–15)
All times are seed-0 update-phase times on OPT-1.3B/SST-2/FT. They exclude loading, evaluation and checkpointing.
Dev loss ≤ 0.40: SubZero+ reaches it at 192.9 s (3,900 queries), versus 497.3 s (7,920 queries) for ZO-Muon, the fastest baseline. That is about 2.58× faster, or 1.92× with two-evaluation confirmation.
Stricter target, 0.32055:
At this target, SubZero+ is about 1.28× faster than ZO-Muon. Its peak memory is 4.04 GB, versus 3.40 GB for MeZO.
Full-run time (min):