acceptodds
Under review as a conference paper at ICLR 2027

SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces

Abstract

Zeroth-order (ZO) optimization with SGD in random subspaces enables memory-efficient fine-tuning of large language models without backpropagation. However, high gradient estimation noise fundamentally undermines adaptive optimizers like Adam. We propose SubZero+, which achieves practical adaptive ZO optimization through a carefully designed dual low-dimensionality strategy: (i) multi-query forward-difference gradient estimation in periodically refreshed random subspaces to mitigate noise amplification in moment buffers, and (ii) Adam updates with periodic restarts performed directly in low-dimensional space rather than full-parameter space. In experiments, this dual design retains memory overhead comparable to momentum-free ZO methods while achieving stronger optimization performance than the evaluated ZO baselines. Theoretically, in the exact-directional limit, -query averaging preserves conditional unbiasedness, while the coefficient estimator's covariance and mean-squared error, as well as query-induced second-moment inflation, scale exactly as . Extensive experiments across SuperGLUE with models from 1.3B to 32B parameters under both full fine-tuning and LoRA schemes demonstrate consistent improvements over competing ZO methods. SubZero+ significantly narrows the performance gap with first-order optimization while preserving ZO's inference-time memory efficiency. Code is available at [https://anonymous.4open.science/r/SubZero-Plus-E8A6](https://anonymous.4open.science/r/SubZero-Plus-E8A6).

Primary area:
optimization
Keywords:
Zeroth-order Optimization, Large Language Model Fine-tuning, Subspace Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iJVaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SubZero+, a memory-efficient adaptive zeroth-order (ZO) fine-tuning method for large language models that operates in periodically refreshed random low-rank subspaces. The method combines two key ideas: (i) multi-query forward-difference averaging to reduce ZO estimator noise in a small coefficient space, and (ii) running Adam entirely on r×r coefficient matrices with per-block restarts when subspaces refresh, thus avoiding O(d) optimizer states. Theoretical results in the exact-directional limit show that multi-query averaging preserves conditional unbiasedness while reducing covariance, MSE, and squared-write inflation by exactly 1/K in coefficient space. Experiments on SuperGLUE with models from 1.3B to 32B parameters under full-parameter and LoRA fine-tuning demonstrate consistent improvements over strong ZO baselines at near-inference-level memory.

Critical strengths
  1. Introduces a dual low-dimensional design: multi-query averaging and Adam on r×r coefficient matrices with periodic basis restarts, enabling adaptive ZO without O(d) memory.

  2. Provides targeted theory (Theorems 1 and 2) that directly characterizes how multi-query averaging affects first and second moments in coefficient space, especially the squared-write inflation that is critical for Adam’s stability.

  3. Practical algorithmic details (sign-corrected QR, local bias correction within refresh blocks, seed-based perturbation regeneration) are thoughtful and reproducible.

Critical weaknesses
  1. Insufficient comparison with closely related published work. AGZO (ICML 2026) exploits activation-informed subspaces, directly challenging SubZero+'s reliance on random bases, yet is neither cited nor compared; MUZO(Peng et al., EMNLP 2025), which the paper identifies as the most directly comparable multi-query adaptive baseline, is discussed only in a memory comparison, while FZOO (ICLR 2026) is not evaluated.

  2. The central “dual low-dimensionality” claim is not adequately isolated. The paper lacks controlled performance ablations comparing coefficient-space SGD with (K=99) against coefficient-space Adam with (K=1), so the reported gains cannot be attributed specifically to the proposed combination rather than multi-query averaging alone.

  3. The robustness of the reported gains is uncertain, and the default query count is insufficiently justified. The paper reports no multi-seed uncertainty estimates, while its highly non-monotonic query-count ablation achieves its best average at (K=50), rather than the default (K=99), and covers only OPT-1.3B on three tasks.

  4. Theoretical and practical claims exceed the available evidence. The theoretical results focus on conditional single-step identities in the exact-directional limit without quantifying finite-(\epsilon) bias or establishing convergence and stability for the complete algorithm with periodic moment resets; meanwhile, despite using 99 perturbation queries plus a baseline evaluation per update, the paper reports no wall-clock throughput or time-to-quality measurements.

reference:

  1. AGZO: Activation-Guided Zeroth-Order Optimization for LLM Fine-Tuning Wei Lin, Yining Jiang, Qingyu Song, Qiao Xiang, Hong Xu ICML 2026

  2. MUZO: Leveraging Multiple Queries and Momentum for Zeroth-Order Fine-Tuning of Large Language Models Yuezhang Peng, Yuxin Liu, Fei Wen, Xie Chen 2025.emnlp-main.432

  3. FZOO: FAST ZEROTH-ORDER OPTIMIZER FOR FINE-TUNING LARGE LANGUAGE MODELS TOWARDS ADAM-SCALE SPEED Sizhe Dang , Yangyang Guo , Yanjun Zhao, Xiaodong Zheng ICLR 2026

Questions for authors

same as weakness

1 comment

1. AGZO, MUZO, FZOO and related work (Related Work, Table 4)

We added AGZO, MUZO-Adam, FZOO and other adaptive baselines, and the revision now states each method's design focus. AGZO builds activation-guided subspaces and may complement SubZero+. MUZO pairs multi-query estimation with adaptive updates. FZOO pairs normalization with batched execution. SubZero+ designs estimation and update together: it averages multiple queries and keeps Adam states in the same two-sided random coefficient space. Our contribution is this joint design and its gains, especially under LoRA. Single-seed accuracy (%):

MethodFT SST-2BoolQWiCMultiRCLoRA SST-2BoolQWiCMultiRC
MeZO90.265.058.359.791.965.561.159.7
SubZero90.965.458.159.591.764.061.457.9
TeZO-SGD91.363.458.562.091.265.158.858.3
MUZO-Adam87.863.258.353.791.965.461.159.2
ZO-Muon91.766.258.860.987.863.658.961.1
AdaMeZO91.463.459.759.991.663.159.760.1
AGZO89.366.959.461.683.161.858.556.5
FZOO92.366.058.862.291.765.458.860.6
SubZero+92.167.759.461.491.965.464.662.9

2. Separating multi-query estimation from coefficient-space Adam (Table 5)

We crossed KK with the matrix-coefficient optimizer (SGD, SGDM or Adam) and shared all other settings: two-sided subspace, forward baseline, rank 16, ϵ=10−3\epsilon=10^{-3}, Adam for 1-D parameters, the same refresh schedule, a separate learning-rate search per configuration, and about 40K queries. Setup: OPT-1.3B/SST-2/LoRA, 3 seeds.

KSGDSGDMAdam
189.1 ± 1.087.4 ± 1.088.9 ± 0.7
9989.8 ± 0.890.2 ± 1.291.7 ± 0.5

At K=99K=99, Adam beats SGD and SGDM, and it gains 2.8 points over K=1K=1.

3. Default KK (original Table 5, now Table 8)

We found a flaw in the original learning-rate search and reran the experiment (OPT-1.3B/SST-2/FT, rank 24, seed 0, about 40K queries). Each KK now has its own learning-rate search, with baseline calls counted. All experiments use K=99K=99; we did not tune it per task.

KStepsRefresh FLRAcc. (%)
120,4005,1001e-490.4
508002001e-391.5
994081021e-392.0

4. Scope of the theory and empirical error (Table 11)

Our 1/K1/K identities hold only in the exact-directional limit, with parameters, minibatch and subspace fixed. They do not show that full Adam with finite ϵ\epsilon is unbiased, stable or convergent. With finite ϵ\epsilon, averaging lowers variance but does not remove bias, and the revision now says so.

We also ran a diagnostic at three checkpoints of one OPT-1.3B/SST-2/FT run. We used 30 repeats per KK over 146 matrix layers and compared against the projected backpropagation (BP) gradient. BP is used only for this diagnostic, never in training. Going from K=1K=1 to K=99K=99:

  • covariance trace fell from 139–927 to 2.0–9.8
  • MSE fell from 0.56–3.74 to 0.008–0.040
  • update cosine rose from 0.56–0.76 to 0.95–0.99

5. Training time and resources (Tables 12–15)

All times are seed-0 update-phase times on OPT-1.3B/SST-2/FT. They exclude loading, evaluation and checkpointing.

Dev loss ≤ 0.40: SubZero+ reaches it at 192.9 s (3,900 queries), versus 497.3 s (7,920 queries) for ZO-Muon, the fastest baseline. That is about 2.58× faster, or 1.92× with two-evaluation confirmation.

Stricter target, 0.32055:

MethodFirst hit (s)QueriesConfirmed (s)
MeZO499.57,920749.0
SubZero1050.615,8401313.0
MUZO-Adamnot reachednot reachednot reached
ZO-Muon497.37,920746.0
AdaMeZO7134.015,8408920.6
AGZO654.27,920981.6
FZOO1716.611,8802288.8
SubZero+387.67,800583.5

At this target, SubZero+ is about 1.28× faster than ZO-Muon. Its peak memory is 4.04 GB, versus 3.40 GB for MeZO.

Full-run time (min):

MethodFT SST-2BoolQWiCMultiRCLoRA SST-2BoolQWiCMultiRC
MeZO41.6141.148.0323.222.3126.228.7312.1
SubZero43.8144.450.4323.628.7131.335.4318.1
MUZO-Adam40.1139.946.6321.020.4124.127.0310.3
TeZO-Adam106.8206.2114.0387.630.6133.935.9318.9
ZO-Muon41.4141.647.6320.928.0130.932.9317.2
AdaMeZO297.6400.6323.1584.137.2139.142.6324.1
AGZO54.5169.159.3375.233.9139.339.1330.2
FZOO95.3205.7102.4381.218.7127.526.1315.9
SubZero+42.8145.449.7323.523.7125.429.8313.1
Reviewer_UhFzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SubZero+ is a zeroth-order method for fine-tuning language models without backpropagation. It averages forward-difference estimates in random subspaces and applies Adam directly to the resulting coefficient matrices. The bases remain fixed within each block. When they change, the method resets the moments and their bias-correction counter.

Theorems 1 and 2 describe the estimator's conditional mean, covariance, and coordinatewise second moments in the exact-directional limit. Experiments cover full-parameter and LoRA tuning. The main benchmarks range from 1.3B to 32B parameters.

Critical strengths

S1. Adaptive states remain compact. Each represented matrix requires two Adam moment buffers with only 2r22r^2 entries in total. Appendix A.2 separately accounts for cached bases and one-dimensional parameter states. This is a practical advantage over full-dimensional Adam states. Table 4 reports about 0.8–1.2% extra peak memory over MeZO for OPT-13B full-parameter tuning. The overhead depends on the setting: Figure 1(c) reports 4.04 versus 3.40 GB for OPT-1.3B, about 18.8% extra.

S2. The complete method performs well across several settings under matched forward budgets. Tables 1–3 give SubZero+ the highest reported ZO average in all seven model/tuning configurations. Its margins over the strongest evaluated ZO baseline by average range from 0.3 to 2.1 points. The main ZO comparisons match 40,000 training forward evaluations, including the shared baseline evaluations. This is useful evidence across model sizes and tuning schemes, although it does not establish a win on every task or statistical reliability.

Critical weaknesses

W1. The closest adaptive comparisons remain unresolved. Section 6.1 excludes MUZO from the performance tables because of its optimizer-state memory. However, "MUZO: Leveraging Multiple Queries and Momentum for Zeroth-Order Fine-Tuning of Large Language Models" (Peng et al.) includes MUZO-Adam with LoRA. Its Table 4 reports 13.1 GB versus 13.0 GB for MeZO-SGD on OPT-2.7B/SST-2 with LoRA. This different setup does not predict the outcome here, but it makes a matched LoRA comparison relevant.

"TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs" (Sun et al.) also includes a compact adaptive variant. Its Algorithm 1 contains separate TeZO, TeZO-m, and TeZO-Adam branches. Appendix A.2 of the submission names that algorithm but does not identify the branch used. The current evidence does not establish an advantage over this adaptive alternative.

W2. The reliability of the gains and the evaluation protocol are insufficiently documented. Tables 1–3 omit seed counts, per-run scores, and uncertainty estimates. Some average margins are small, including 66.3 versus 66.0 and 83.7 versus 83.2 in Table 1. Their size alone does not show that the gains are unreliable, but repeated paired runs are needed to assess them.

Appendix A.2 gives search grids and a validation-loss selection rule, which are useful. However, the paper does not specify evaluation split identities and sample counts, prompt templates, or task-specific scoring details. These omissions limit interpretation of the averages across classification, multiple-choice, and generation tasks.

Additional concerns.

(a) The experiments do not isolate the proposed interaction between averaging and adaptive updates. The main comparisons change the optimizer, query count, difference scheme, refresh interval, and one-dimensional parameter treatment together. Table 5 varies query count only within SubZero+. Theorem 2 establishes the expected squared input to Adam, but does not establish stable normalized updates. Figure 6 compares complete methods on one training-accuracy task and explicitly acknowledges that it does not isolate averaging.

A small query-count-by-optimizer comparison would test whether the combination explains the gains. This matters because the paper presents the interaction as its main explanation. The current results support the complete recipe more strongly than that explanation. I do not regard the absence of an Adam convergence theorem as a defect by itself.

(b) Table 5 has contradictory budget descriptions. Appendix A.1 fixes 30,000 perturbation queries and adds one baseline evaluation per update. Section 6.3 and Appendix A.2 instead describe a fixed total-forward budget including those baselines. Under the first description, K=1K=1 uses 60,000 total evaluations, while K=100K=100 uses 30,300. The actual counts are needed to interpret the ablation under equal cost. This issue does not affect the consistent 40,000-forward accounting for the main tables.

Questions for authors
  1. Can you provide a representative matched-budget comparison against MUZO-Adam with LoRA? Please identify the optimizer branch used for TeZO and provide a TeZO-Adam comparison if it was not evaluated.

  2. Can you compare K=1K=1 and K=99K=99 with both coefficient SGD and coefficient Adam on a representative task under equal total-forward budgets? Please keep the difference scheme, rank, and one-dimensional parameter treatment consistent, and tune learning rates for each configuration. Please state whether the shared refresh policy is measured in updates or forward calls.

  3. How many independent runs produced the main-table results? Please provide seed-level scores and uncertainty for representative paired comparisons, together with evaluation splits, sample counts, prompt templates, and scoring definitions.

  4. Which budget was actually used for Table 5? Please report steps, perturbation evaluations, baseline evaluations, and total evaluations for each KK. If total budgets differed, does the query-count conclusion persist after matching them?

3 comments

W1. Closest adaptive baselines and memory (Tables 6, 12, 13)

Thank you for pointing us to MUZO and TeZO-Adam. We have added both. The main text now separates TeZO-SGD (Tables 1–2) from TeZO-Adam (new comparison). Setup: OPT-1.3B/SST-2/FT, fixed configuration, 3 seeds (mean ± sample std.):

MethodAcc. (%)
MUZO-Adam87.6 ± 0.7
TeZO-Adam87.6 ± 1.2
ZO-Muon91.5 ± 0.5
SubZero+91.5 ± 0.6

Peak allocated training memory (GB):

MethodSST-2BoolQWiCMultiRCFT avgLoRA avg
MeZO3.408.753.5213.507.297.30
SubZero3.438.783.5513.537.327.30
AdaMeZO3.408.753.5213.507.297.30
AGZO3.408.753.5313.507.307.30
TeZO-Adam3.638.983.7513.737.527.34
ZO-Muon4.169.084.1613.837.817.31
MUZO-Adam14.5618.5514.5623.3017.757.31
FZOO (batched)13.5852.7813.96OOM26.77*24.43*
SubZero+4.048.824.0413.577.627.30

*Average over 3 tasks.

Under FT, SubZero+ uses 0.64 GB more than MeZO on SST-2 (0.33 GB more on average) and about 10 GB less than MUZO-Adam. Under LoRA, all methods except FZOO are within 0.05 GB of each other.

W2. Statistical reliability and evaluation protocol (Tables 5, 6)

We added three-seed results for the key configurations. On SST-2 FT, SubZero+ matches ZO-Muon's mean and is 3.9 points above our MUZO-Adam implementation. In the high-KK LoRA setting, Adam reaches 91.7 ± 0.5%, compared with 89.8 ± 0.8% for SGD and 90.2 ± 1.2% for SGDM.

We follow MeZO's task construction, prompt templates and evaluation pipeline:

  • Data: 1,000 training examples and 500 for hyperparameter selection. Evaluation uses up to 1,000 examples, or the full set if it is smaller.
  • Classification and multiple choice: we predict the candidate with the highest average per-token log-likelihood.
  • Generation: greedy decoding, scored with each task's standard metric.

All methods share the same data and pipeline for a given task. The FO baseline uses Adam with batch size 8, a learning rate from {1e-5, 5e-5, 8e-5}, no weight decay, 5 epochs and a linear schedule.

(a) Averaged estimation and the nonlinear Adam update (Tables 5, 11)

We crossed query count KK with the matrix optimizer. Setup: OPT-1.3B/SST-2/LoRA, 3 seeds, everything else identical to SubZero+:

KSGDSGDMAdam
189.1 ± 1.087.4 ± 1.088.9 ± 0.7
9989.8 ± 0.890.2 ± 1.291.7 ± 0.5

At K=99K=99, Adam outperforms both SGD and SGDM.

We also ran an exploratory diagnostic at three checkpoints of one OPT-1.3B/SST-2/FT run. For each checkpoint we used one minibatch, repeated each KK 30 times, and collected statistics over 146 matrix layers.

  • Reference: the backpropagation (BP) gradient, projected onto the same coefficient subspace. BP is used only for this diagnostic, never in training.
  • Metrics: the spread of repeated ZO estimates (covariance trace), their MSE against the reference, and the cosine similarity between the resulting update and the reference update under the same optimizer state.
StepKCov. traceMSEUpdate cos
401139.45.64e-10.756
40504.11.66e-20.980
40992.08.21e-30.990
2031926.53.74e00.563
2035018.97.64e-20.913
203999.83.96e-20.949
3661543.42.20e00.573
366508.83.57e-20.957
366994.71.90e-20.975

As KK increases, covariance and MSE fall and update cosine rises at all three checkpoints.

(b) Table 5 budget (original Table 5, now Table 8)

We found a flaw in the original learning-rate search and reran the experiment (OPT-1.3B/SST-2/FT, rank 24, seed 0, ~40K training queries). Each KK now has its own learning-rate search under the same budget, and baseline calls are counted. The revised results replace the original table. KK was not tuned per task: all experiments use K=99K=99.

KStepsRefresh FLRAcc. (%)
120,4005,1001e-490.4
508002001e-391.5
994081021e-392.0

Thanks for adding the adaptive baselines and rerunning the query-count ablation. The revised forward counts address my concern about the original Table 5, and the comparison between SGD, SGDM, and Adam at both query counts helps explain what Adam adds. My main concern is the comparison with MUZO-Adam under LoRA. The new accuracy comparison is for full fine-tuning, while the LoRA results report memory. Since the LoRA memory numbers are close, I still want to see how the two methods compare on accuracy at the same forward budget. I’m keeping my rating at weak reject.

Thank you for your comment. We kindly refer the reviewer to Table 4 for the comparison with MuZO-Adam under LoRA. Regarding the memory footprint of MuZO-Adam, please refer to Figure 5 and Appendix A.1. Tables 6&13 compares the memory usage of MuZO-Adam under the LoRA setting.

Reviewer_sUxpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SubZero+, a memory-efficient zeroth-order optimizer for LLM fine-tuning that builds on SubZero’s layer-wise random subspaces. The method combines multi-query forward-difference gradient estimation with Adam updates performed directly on low-dimensional coefficient matrices, resetting the optimizer states whenever the subspace bases are refreshed. In the exact-directional limit, the paper derives conditional identities showing that query averaging reduces the coefficient estimator’s covariance, mean-squared error, and additive second-moment inflation by a factor of (1/K). Experiments across models ranging from 1.3B to 32B parameters, under both full fine-tuning and LoRA, report improved average performance over the evaluated ZO baselines while retaining low memory overhead.

Critical strengths
  1. The paper addresses a meaningful problem: improving adaptive ZO optimization without introducing full-dimensional optimizer states that undermine its memory advantage. Accumulating gradient estimates and maintaining Adam’s moments directly in the coefficient space is a straightforward and practical design.
  2. The theoretical results are explicitly restricted to a fixed optimization state, minibatch, and subspace in the exact-directional limit. Under these conditions, the stated mean, covariance, and squared-estimate identities are mathematically sound, and the proofs are clear.
Critical weaknesses
  1. The methodological novelty appears incremental, and the positioning relative to closely related work needs to be more precise. SubZero (Yu et al., ICCV 2025) already provides two-sided random subspaces and a compact coefficient-momentum extension; MUZO-Adam (Peng et al., EMNLP 2025) combines multi-query averaging with adaptive updates; TeZO-Adam (Sun et al., arXiv 2025) maintains compact moment representations; ZO-Muon (Lang et al., arXiv 2026) performs multi-query estimation and spectral updates within a subspace; and AdaMeZO (Cai et al., ICML 2026) explores Adam-style adaptation without retaining moment buffers. These approaches differ in their subspace constructions and normalization rules, but they already establish several of the main ingredients motivating SubZero+. The distinguishing contribution therefore appears to be the specific combination of two-sided coefficient-space Adam, multi-query estimation, and periodic restarts. The paper should explain more clearly why this combination constitutes a substantive advance. Its broad statements that previous adaptive ZO methods consistently fail to outperform SGD, or that related multi-query methods primarily operate in the full parameter space, should also be qualified.
  2. The experiments do not sufficiently isolate the contributions of multi-query estimation and coefficient-space Adam. Relative to SubZero, the proposed method simultaneously changes the query count, optimizer, finite-difference scheme, subspace-refresh schedule, and treatment of one-dimensional parameters. Consequently, the main results establish the effectiveness of the complete configuration but do not identify which components drive the improvements or whether the two central designs provide complementary benefits. Table 5 varies the query count within SubZero+ and therefore does not resolve this issue. A controlled comparison of single-query versus multi-query estimation crossed with coefficient-space SGD versus Adam would be particularly informative. These variants should use a consistent estimator and subspace construction, matched total forward budgets, and separately tuned learning rates, with the refresh schedule explicitly controlled or justified. An additional SGDM comparison would help distinguish the benefit of coordinate-wise adaptivity from that of more stable momentum accumulation.
Questions for authors
  1. Why is MUZO-Adam omitted from the LoRA experiments, and which TeZO optimizer variant is used? Please provide at least one budget-matched comparison with the closest adaptive baselines.
  2. Under the same subspaces and query counts, does Adam outperform SGD or SGDM? A small controlled ablation would help isolate the contributions of query averaging and adaptive updates.
  3. Does Table 5 fix total forward evaluations or perturbation queries? Please report the steps and total forwards for each (K), and clarify how learning rates and subspace-refresh intervals were tuned.
  4. How many independent random seeds were used, and are the small gains reproducible? Please also clarify the data splits, evaluation metrics, and FO baseline configurations.
  5. Do gradient-estimate or moment-statistic measurements on actual LLMs support the proposed noise-reduction mechanism? Measurements at a few checkpoints on a small model would be sufficient.
1 comment

1. How SubZero+ differs from related ZO methods (Contribution & Related Work)

We don't claim to be the first to use multi-query estimation, low-rank subspaces or compact momentum. Our claim is that in adaptive ZO fine-tuning, gradient estimation and the update rule should be designed together, because the estimate also goes into Adam's moments. Our contribution is this joint design and the accuracy and memory gains it gives, especially for LoRA.

SubZero+ aggregates queries and keeps Adam moments in the same two-sided low-dim coefficient space, so memory stays low. Each update uses 99 perturbation queries plus 1 shared baseline query. This reduces the noise that enters Adam's moments (MUZO uses 8 queries and ZO-Muon uses 4). AdaMeZO uses random-state caching to cut MeZO-Adam's memory, but it doesn't reduce estimator variance. We revised Related Work and narrowed our earlier broad claims.

MethodGrad. spaceFwd/updateUpdate spaceRule
SubZeroTwo-sided low-dim2FullSGD
MUZOFull8FullAdam
TeZO-AdamLow-dim tensor2Low-dim tensorAdam
ZO-MuonOne-sided low-dim5One-sided low-dimMuon
AdaMeZOFull2FullAdam
SubZero+Two-sided low-dim100Two-sided low-dimAdam

SubZero+ has the best cross-task average among ZO methods in all six FT/LoRA settings. In LoRA, it beats the strongest baseline by 2.0, 1.7 and 2.1 points.

2. Controlled subspace-Adam comparison (Table 5)

We crossed KK with the optimizer used for the matrix coefficients. All other settings are fixed: same subspace and estimator, rank 16, perturbation 10−310^{-3}, Adam for 1-D parameters, and refresh after the same number of cumulative queries. Each variant gets its own learning-rate search, with a budget of about 40K queries. Setup: OPT-1.3B/SST-2/LoRA, 3 seeds:

KSGDSGDMAdam
189.1±1.087.4±1.088.9±0.7
9989.8±0.890.2±1.291.7±0.5

At K=99K=99, Adam beats SGD by 1.9 points and SGDM by 1.5 points. Going from K=1K=1 to K=99K=99 improves Adam by 2.8 points. With few queries, noisy estimates corrupt the moment statistics, which supports designing the two parts together.

3. MUZO-Adam and TeZO-Adam (Table 6)

The original experiments used TeZO-SGD; we now also compare against TeZO-Adam and MUZO-Adam. Setup: OPT-1.3B/SST-2, seeds 0/1/2:

MethodFT AccFT Mem (GB)LoRA AccLoRA Mem (GB)
MUZO-Adam87.6±0.714.5691.3±0.73.42
TeZO-Adam87.6±1.23.6390.2±1.23.45
ZO-Muon91.5±0.54.1688.8±0.93.42
SubZero+91.5±0.64.0491.7±0.53.41

In FT, SubZero+ ties ZO-Muon and beats MUZO-Adam by 3.9 points. In LoRA, it has the best accuracy and the lowest memory. The appendix describes how we implemented the new baselines.

4. Choice of K (now Table 8)

We found a flaw in the original learning-rate search and reran the experiment with a separate learning-rate search for each KK. Setup: OPT-1.3B/SST-2/FT, rank 24, seed 0, 40,800 queries including baseline calls. We use K=99K=99 everywhere and did not tune it per task.

KStepsFAccLR
120400510090.41e-4
5080020091.51e-3
9940810292.01e-3

5. Seeds and protocol

We reran the experiments in Tables 5 and 6 with seeds 0/1/2 and report mean ± sample std.

  • Data: We follow MeZO's tasks, prompts and evaluation. We use 1,000 training and 500 development examples and evaluate on at most 1,000 examples (872 for SST-2).
  • LoRA: applied to q_proj/v_proj, rank 8, alpha 16.
  • Training: FP16, batch size 16, max length 2048, constant learning rate, no weight decay.
  • Scoring: Classification and multiple-choice tasks use the highest average per-token log-likelihood. Generation tasks use greedy decoding and each task's standard metric.
  • FO baseline: Adam, batch size 8, learning rate from {1e-5, 5e-5, 8e-5}, 5 epochs, linear schedule.

6. Noise reduction on a real LLM (A.3.2, Table 11)

We took three checkpoints from one OPT-1.3B/SST-2/FT run. At each one, we used one minibatch and ran each KK 30 times over 146 matrix layers. The BP gradient, projected onto the same subspace, serves only as a diagnostic reference; training never uses it.

StepK=1 / 50 / 99: Cov. traceMSEUpdate cos
40139.4 / 4.1 / 2.00.564 / 0.0166 / 0.00820.756 / 0.980 / 0.990
203926.5 / 18.9 / 9.83.74 / 0.0764 / 0.03960.563 / 0.913 / 0.949
366543.4 / 8.8 / 4.72.20 / 0.0357 / 0.01900.573 / 0.957 / 0.975

At every checkpoint, larger KK lowers covariance and MSE and raises update cosine.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.