acceptodds
Under review as a conference paper at ICLR 2027

Zeroth-Order Riemannian Optimization on Fixed-Rank Manifolds for LLM Fine-Tuning

Abstract

Zeroth-order (ZO) optimization estimates descent directions from function values alone, which removes the activation storage required by backpropagation and makes it attractive for memory-efficient fine-tuning of large language models (LLMs). Existing low-rank ZO methods either parameterize the additive update through Euclidean factors with a non-unique representation, or restrict each finite-difference direction to be low-rank. In this work, we propose to constrain each additive update block to a fixed-rank matrix manifold while keeping the pretrained weights frozen, so the learned update remains rank-constrained throughout fine-tuning. On this geometry, we build a Riemannian ZO gradient estimator that projects a random matrix onto the tangent space, normalizes it to correct the scale mismatch induced by projection, and further propose Zeroth-order Riemannian Gradient Descent (ZO-RGD) and its smoothed variant, ZO-RGD-S. We show that the ambient finite difference matches its retracted counterpart, the normalized estimator recovers the Riemannian gradient in expectation, and ZO-RGD converges under trajectory-regularity conditions. Experiments on LLM fine-tuning tasks show that the proposed algorithms are competitive with strong forward-only baselines across model families and tasks, with gains in several settings.

Primary area:
optimization
Keywords:
LLM, memory-efficient fine-tuning, Riemannian optimization, zeroth-order optimization, fixed-rank matrix manifold

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_4gXgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates forward-only LLM adaptation as zeroth-order optimization of persistent additive updates constrained to fixed-rank matrix manifolds. Rather than optimizing redundant Euclidean factors B,AB,A, each learned update matrix is maintained directly on a rank-rr manifold. The proposed estimator projects Gaussian perturbations onto the tangent space and normalizes them to correct the rank- and shape-dependent perturbation scale. Based on this estimator, the paper introduces ZO-RGD and the scalar-smoothed ZO-RGD-S. The theoretical analysis shows first-order agreement between ambient and retracted finite differences, expected alignment of the normalized estimator with the Riemannian gradient up to a known scaling and bias, and a conditional non-asymptotic convergence result. Experiments cover OPT-1.3B/13B/30B, Llama-3.1-8B, and Qwen2.5-7B, together with controlled comparisons, geometry ablations, rank and precision diagnostics, and runtime/memory measurements.

Critical strengths

The paper does more than report endpoint benchmark numbers. It includes multiple model families, controlled self-run baselines, geometry/parameterization ablations, rank and numerical-stability analyses, and explicit runtime/memory profiling. In the controlled OPT comparison, ZO-RGD-S substantially improves RTE over MeZO/LOZO and also improves SQuAD, while the paper clearly separates the stronger first-order LoRA result from the forward-only ranking.

Critical weaknesses

The paper motivates its formulation by contrasting direct optimization of a fixed-rank update matrix with methods that operate through low-rank adapter factors, and describes RoZO as using geometry induced by low-rank adapter factors while the present work “takes the matrix update itself as the optimization variable on the fixed-rank manifold.” However, this distinction appears substantially narrower than stated. RoZO already formulates the low-rank update X=BAX=BA as a point on a fixed-rank manifold and performs tangent-space zeroth-order optimization together with rank-preserving geometric updates. Thus, both approaches appear to share the central formulation of maintaining a persistent low-rank additive update on a fixed-rank manifold, rather than merely imposing low rank on individual perturbations. The more specific novelty of the present paper seems to lie in its particular tangent-direction construction, ambient Gaussian sampling followed by orthogonal tangent projection and explicit norm normalization, and in the associated finite-difference analysis, rather than in the broader fixed-rank-manifold formulation itself. I would like the paper to sharpen this distinction and avoid presenting the latter as a major conceptual departure if the optimization object is already shared by prior/concurrent work.

Questions for authors

Could the authors provide a more direct head-to-head comparison with RoZO under the same backbone, tasks, rank, number of function evaluations, learning-rate tuning budget, and checkpoint-selection protocol? Given the conceptual proximity of the two methods, such a comparison would help establish whether the proposed ambient-Gaussian projection and normalization provide an advantage beyond an alternative construction of tangent-space ZO directions.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 4gXg

We thank the Reviewer for the careful reading and for the precise characterization of the relationship to RoZO, which is also discussed in part (B) of the general response.

Replies to weaknesses:

The distinction from RoZO is narrower than stated.

We agree with the Reviewer's assessment. RoZO [R3] likewise maintains a persistent rank-rr additive update X=BA\boldsymbol X=\boldsymbol B\boldsymbol A on the fixed-rank manifold using tangent-space ZO directions and SVD retraction, so our phrasing in Section 2 overstated the formulation-level distinction. Both works share the persistent fixed-rank manifold state; our specific contribution lies in the estimator and analysis identified by the Reviewer: constructing isotropic tangent directions via orthogonal projection of ambient Gaussian matrices and per-block norm calibration (requiring no factor-space preconditioning, momentum transport, or trust-region heuristics), proving leading-order agreement and gradient alignment for off-manifold ambient tangent-line queries (Propositions 1–2), and establishing the resulting conditional stationarity bound (Theorems 1–2).

Replies to questions:

A head-to-head comparison with RoZO under the same protocol.

Because RoZO [R3] has no publicly available implementation, a fair head-to-head run under matched backbones, tuning budgets, and checkpoint rules cannot be executed without reconstructing several underspecified components (e.g., factor gauge after retraction, trust-region scheduling, and preconditioner details). Analytically, part (B) isolates what ambient Gaussian projection and normalization change relative to factor-induced tangent directions. For Δ=B ΔA+ΔB A\boldsymbol\Delta=\boldsymbol B\,\Delta\boldsymbol A+\Delta\boldsymbol B\,\boldsymbol A with standard Gaussian (ΔA,ΔB)(\Delta\boldsymbol A,\Delta\boldsymbol B), the leading-order unpreconditioned mean step BB⊤G+GA⊤A\boldsymbol B\boldsymbol B^\top\boldsymbol G+\boldsymbol G\boldsymbol A^\top\boldsymbol A varies under gauge rescaling (cB,A/c)(c\boldsymbol B,\boldsymbol A/c) and, even under balanced factors, weights tangent directions by the singular values of X\boldsymbol X (vanishing as Σ→0\boldsymbol\Sigma\to\boldsymbol 0). Our normalized estimator yields mean (mn/dr) PTX(G)(mn/d_r)\,P_{T_{\boldsymbol X}}(\boldsymbol G), which is invariant to factor parameterization and singular-value magnitudes, allowing near-zero initialization (σinit=10−6\sigma_{\mathrm{init}}=10^{-6}, Appendix A.2). Empirically, Table 5 (right) compares our manifold update only against Euclidean ZO on (B,A)(\boldsymbol B,\boldsymbol A) at rank 4, which does not represent full RoZO.

[R3] Song and Li, EACL, 2026.

Reviewer_NdoHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a forward-only, zeroth-order method for memory-efficient LLM fine-tuning. It optimizes persistent additive update matrices on a fixed-rank manifold while freezing pretrained weights. Each update block is kept at rank rr, and a Riemannian zeroth-order gradient is estimated via tangent-space projection, normalization, and finite differences using only two forward passes. It introduces ZO-RGD and a smoothed ZO-RGD-S (recursive smoothing without extra forward cost). The paper shows the estimator matches the retracted version to leading order. Experiments demonstrate competitive performance across model families and tasks, with improvements over strong forward-only baselines in several settings.

Critical strengths
  • The paper optimizes persistent additive update matrices on a fixed-rank manifold while freezing pretrained weights.

  • Experiments demonstrate competitive performance across model families and tasks.

Critical weaknesses
  • Why is fixed rank necessary? In terms of performance, LoRA achieves comparable results to fixed rank, but it achieves this using two low-rank matrices, which significantly reduces GPU memory usage. In contrast, using fixed rank requires storing parameters in the original (full) parameter dimensionality, leading to higher memory cost.
  • The experimental results are relatively weak. In addition, the comparison baselines are not the most recent methods, and the average performance does not outperform LOZO.
  • Regarding memory usage, MeZO exploits a random-seed trick, so its fine-tuning memory footprint is consistent with inference-time memory. In Table 6, the proposed method reports even lower memory than MeZO. This part needs a much more detailed explanation, e.g., what exactly is counted, whether the same random-seed/inference-consistent setup is used, and how memory is measured.
  • It would also be better to include wall-clock time comparisons, not only memory and loss/accuracy. This would make the trade-offs clearer and improve the credibility of the efficiency claims.
Questions for authors

Please see the weaknesses listed above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer NdoH

We thank the Reviewer for reading our work and for the comments. Shared questions are also addressed in parts (A)–(C) of the general response.

Replies to weaknesses:

Why is fixed rank necessary, given that LoRA uses two low-rank matrices and less memory?

To clarify a key implementation point, our method does not store trainable parameters in the full m×nm\times n dimension. Each adapted block stores only its thin-SVD triple (U,Σ,V)(\boldsymbol U,\boldsymbol\Sigma,\boldsymbol V) with r(m+n+1)r(m+n+1) scalars versus r(m+n)r(m+n) for LoRA (Section 4.3 and Appendix A.4), totaling 7.5 MB across OPT-1.3B at r=4r=4 (Appendix D.17), and its measured peak training memory matches MeZO and LOZO (Table 6). Optimizing on the fixed-rank manifold instead of perturbing (B,A)(\boldsymbol B,\boldsymbol A) directly avoids factor-scaling ambiguity: as shown in part (B), X=BA\boldsymbol X=\boldsymbol B\boldsymbol A is invariant to rescaling (cB,A/c)(c\boldsymbol B,\boldsymbol A/c), whereas Euclidean ZO perturbations in factor coordinates change direction and norm under rescaling.

The results are weak, the baselines are not the most recent, and the average does not exceed LOZO.

Our comparison includes the recent ICLR 2025 baselines LOZO [R2] and HiZOO [R4] alongside MeZO and MeZO-LoRA [R1], while RoZO [R3] (EACL 2026) has no public code. As shown in part (C), the average scores are on par with LOZO (70.34% vs. 70.62% across 11 tasks on OPT-13B; 73.71% vs. 72.93% across 6 tasks on OPT-30B). This matches our stated claim: competitive forward-only accuracy while guaranteeing an exact rank-rr deployable adapter throughout training, with a consistent cross-seed gain of +3.2+3.2 F1 over LOZO on OPT-13B SQuAD across all five paired seeds (Table 16).

Table 6 reports lower memory than MeZO; what is counted and how is it measured?

We treat the four forward-only methods as having equivalent peak memory: 39.4 GB versus 39.6 GB is a 0.5% allocator difference, and we do not claim lower memory than MeZO. Table 6 reports PyTorch peak reserved GPU memory across the full training loop (torch.cuda.max_memory_reserved), covering frozen FP16 model weights, forward-pass activations at batch size 16, and the CUDA caching allocator. All four methods use the identical seed-regeneration protocol across both forward passes and the update without retaining dense perturbation or optimizer-state buffers, profiled back to back for 1,000 steps on the same A100-80G GPU and replicated on a second card (Appendix D.16, Table 23). Following the parameter count in Appendix D.17, the thin-SVD state adds at most 30 MB on OPT-13B (at r=4r=4, and 15 MB at r=2r=2).

Wall-clock comparisons should be included.

Table 6 reports per-step wall-clock latency (3.51 s and 3.54 s for ZO-RGD and ZO-RGD-S vs. 1.93 s for LOZO and 2.03 s for MeZO), and part (C) evaluates convergence at equal wall-clock time. On OPT-13B SQuAD at the shared learning rate 10−710^{-7}, both variants drop below LOZO's 20,000-step final development loss within 73% of LOZO's total training time (at step 8,000), whereas on RTE reaching LOZO's final loss takes 1.1 to 1.5 times LOZO's wall-clock time.

[R1] Malladi et al., NeurIPS, 2023. [R2] Chen et al., ICLR, 2025. [R3] Song and Li, EACL, 2026. [R4] Zhao et al., ICLR, 2025.

Reviewer_2qFFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies memory-efficient fine-tuning using zeroth-order optimization, which avoids storing activations for backpropagation. The author proposes a method(ZO-RGD) in which the updates are constrained to a fixed-rank matrix manifold. It is a three-step process: drawing a random direction(Z), projecting it to the tangent space of the rank-constrained manifold, normalizing it, using the projected direction in the two-point estimator, and finally retracting the update to rank(r) using truncated SVD. The paper also introduces a smoothed version, ZO-RGD-S, and provides convergence guarantees for ZO-RGD.

Critical strengths

The paper is well written and easy to follow. The motivation for treating ΔW\Delta W as a variable on an r-rank manifold is principled, and it does not depend on the factorization ambiguity of ΔW\Delta W. The authors recognize the change in finite-difference scale mismatch after projection and thus perform a normalization operation. The paper provides theoretical convergence results and also empirical experiments.

Critical weaknesses
  • I am not convinced that factor-based methods(like RoZO) or low-rank-perturbation methods (e.g., LOZO) actually do much worse in practice. First, I request that the authors also provide the average over all tasks in Table 1 and Table 2 as a separate column. Then, based on the experimental results, the improvement is marginal relative to the baseline; the gains are small and noisy, and no standard deviation across seeds is reported.
  • Comparison with RoZO, which the paper mentions as the closest related work, is missing.
  • Can the authors provide some experiments; a toy example will work too, where the paper's motivation for not using BA-factor vs. using manifold constraints is clear?
  • As far as I understand, these results come from the proposed method using more compute; as in Table 6, we can see ZO-RGD-S takes almost 1.8 times more time. Can the authors run an experiment with the same budget (a wall-clock-matched comparison)?
  • The gap to LoRA+AdamW is still very large (Table 3). If we match the memory and compute budget, can the zero-order method reach close to first-order methods?
  • I would appreciate it if, in the conclusion section, the authors also mention the limitations of the proposed method.
Questions for authors

See Weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2qFF

We thank the Reviewer for the constructive feedback and suggestions. Shared points on RoZO, task averages, and runtime are detailed in parts (B) and (C) of the general response.

Replies to weaknesses:

Factor-based and low-rank-perturbation methods may not do much worse; averages and standard deviations are missing.

Contribution 4 claims competitive accuracy with gains in several settings, matching the averages in part (C): across tasks, our variants are on par with LOZO on average (70.34 vs. 70.62 on OPT-13B; 73.71 and 73.00 vs. 72.93 on OPT-30B). Tables 1–2 follow the single-run format of LOZO [R2] with cited baseline rows and carry no standard deviations. Standard deviations are reported over three seeds in Tables 3–5 and five paired seeds in Table 16. The cross-seed gain is on OPT-13B SQuAD: 84.9±1.384.9\pm1.3 (ZO-RGD-S) and 83.3±1.083.3\pm1.0 (ZO-RGD) vs. 82.2±0.782.2\pm0.7 for LOZO (Table 3), and +3.2+3.2 F1 for ZO-RGD-S over LOZO on 5/5 paired seeds at a shared learning rate (p=0.031p=0.031, Table 16).

A comparison with RoZO is missing.

As noted in part (B), RoZO [R3] has no public code, so a matched empirical comparison cannot be run. Analytically, both methods query in TXMrT_{\boldsymbol X}\mathcal M_r, whereas unpreconditioned factor perturbations yield a mean step that depends on factor scaling and singular values while ours depends only on the column and row spaces of X\boldsymbol X.

A toy example showing the benefit of manifold constraints over BA factors.

In closed form, (B,A)(\boldsymbol B,\boldsymbol A) and (cB,A/c)(c\boldsymbol B,\boldsymbol A/c) represent the same update X=BA\boldsymbol X=\boldsymbol B\boldsymbol A. With standard Gaussian factor perturbations, the leading-order mean ZO step for gradient G\boldsymbol G is BB⊤G+GA⊤A\boldsymbol B\boldsymbol B^\top\boldsymbol G+\boldsymbol G\boldsymbol A^\top\boldsymbol A at c=1c=1 and c2BB⊤G+c−2GA⊤Ac^2\boldsymbol B\boldsymbol B^\top\boldsymbol G+c^{-2}\boldsymbol G\boldsymbol A^\top\boldsymbol A after rescaling. For a 40×3040\times30 rank-3 matrix with unit singular values and balanced factors, setting c=10c=10 changes the root-mean-square norm of this mean step by 63×63\times and drops its cosine with PTX(G)P_{T_{\boldsymbol X}}(\boldsymbol G) from 0.98 to 0.67 over Gaussian G\boldsymbol G. Our normalized estimator gives mean (mn/dr) PTX(G)(mn/d_r)\,P_{T_{\boldsymbol X}}(\boldsymbol G) for both pairs. Empirically, Table 5 (right) compares the manifold update with Euclidean ZO on (B,A)(\boldsymbol B,\boldsymbol A) on OPT-1.3B (r=4r=4, separately tuned rates): 87.7±0.687.7\pm0.6 vs. 78.5±10.078.5\pm10.0 on SST-2, 56.0±1.456.0\pm1.4 vs. 58.4±4.058.4\pm4.0 on RTE, and 71.3±0.571.3\pm0.5 vs. 67.7±1.367.7\pm1.3 on SQuAD.

The method takes 1.8 times the time per step; a wall-clock-matched comparison is needed.

Both variants use two forward passes per step, as MeZO and LOZO do; the 1.8×1.8\times step time (Table 6) comes from generating and projecting dense Gaussian matrices (~60% of a step, Appendix D.11). At equal wall-clock time under the shared rate 10−710^{-7} on OPT-13B SQuAD, ZO-RGD and ZO-RGD-S reach mean development losses of 0.466 (3/5 completed seeds) and 0.464 (5/5 seeds) at step 8,000, using 73% of the wall-clock time LOZO takes over 20,000 steps to reach 0.490. On RTE, reaching LOZO's final loss takes 1.1–1.5$\times$ LOZO's wall-clock time, giving no wall-clock advantage there.

The gap to LoRA+AdamW is large; can ZO get close at matched memory and compute?

At the memory budget of forward-only ZO, LoRA+AdamW cannot run in this setup: on OPT-13B (batch size 16, no activation checkpointing), it uses 126.2 GB on RTE (H200) and runs out of memory on SQuAD on an 80 GB A100, whereas all four forward-only methods use 39.4–39.6 GB (Table 6). When backpropagation fits in memory, first-order LoRA is substantially more accurate than every forward-only method (83.4 vs. 60.4–69.4 on RTE, Table 3). Forward-only ZO targets the regime where backpropagation exceeds GPU memory.

The conclusion should state the limitations.

We agree and summarize the main limitations here: (i) per-step latency is 1.8×1.8\times that of LOZO, retaining a wall-clock gain on SQuAD while offering none on RTE; (ii) accuracy gains are task-dependent, and first-order LoRA remains stronger whenever backpropagation fits in memory; (iii) Theorems 1–2 cover base ZO-RGD under trajectory regularity (Assumptions 1–3), whereas ZO-RGD-S is empirical and base ZO-RGD diverges in FP16 on some SQuAD and ReCoRD runs (Appendix D.18); and (iv) a matched empirical comparison with RoZO is unavailable without public code.

[R2] Chen et al., ICLR, 2025. [R3] Song and Li, EACL, 2026.

Reviewer_5PqUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed to constrain LLM fine-tuning into a persistent low-rank space update, and derive a manifold optimization method to train it.

Critical strengths

This paper proposes a new fine-tuning formulation and corresponding optimization method to train it.

Critical weaknesses

I don't get why we want the overall update to stay in a low-rank space consistently.

Questions for authors

I understand the paper's logic, and the convergence. But what's wrong with per-step low-rank update? Why we want the accumulated update step to stay low-rank?

Theory-wise, the convergence is standard convergence result, I didn't see any motivation why we want such consistent low-rank update

Compute-wise, storing LOZO’s accumulated change directly is not compact since it's high-rank, but we can do seed replay, which is a known MeZO trick

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5PqU

We thank the Reviewer for reading our work and for the comments. Our core motivation for persistent fixed-rank updates is also summarized in part (A) of the general response.

Replies to weaknesses:

It is unclear why the overall update should stay low-rank.

Keeping the persistent update on Mr\mathcal M_r ensures that the trained result itself is a deployable rank-rr adapter. Each adapted block stores only its thin-SVD triple with r(m+n+1)r(m+n+1) scalars, totaling 7.5 MB across all 146 blocks on OPT-1.3B at r=4r=4 (Appendix D.17). Task-specific adapters can therefore be saved, exchanged, and attached alongside the frozen pretrained weights as in LoRA, while fine-tuning retains the inference-level peak memory of MeZO and LOZO (Table 6).

Replies to questions:

What is wrong with a per-step low-rank update?

Applying a low-rank perturbation at each step does not keep the accumulated weight update low-rank. In Appendix D.11, we measured the singular-value spectrum of LOZO's accumulated update WT−W0\boldsymbol W_T-\boldsymbol W_0: its median effective rank (at 90% spectral energy) reaches 1,019 out of 2,048 on OPT-1.3B and 2,466 out of 5,120 on OPT-13B within the first 4,000 steps. Consequently, per-step low-rank perturbations still produce a dense weight matrix per block, whereas our trajectory maintains exact rank rr at every checkpoint.

The convergence result is standard and does not motivate the constraint.

While the O(T−1/2)\mathcal O(T^{-1/2}) rate is standard for nonconvex two-point ZO optimization, the fixed-rank geometry reduces the dimension factor in the leading stochastic term. In the formal stationarity bound of Theorem 2 (Appendix C.3), the weighted norm scales each 2D block by cℓ=mℓnℓ/drℓc_\ell=m_\ell n_\ell/d_{r_\ell} with drℓ=rℓ(mℓ+nℓ−rℓ)d_{r_\ell}=r_\ell(m_\ell+n_\ell-r_\ell). Dividing Theorem 2 by cmin⁡=min⁡ℓcℓc_{\min}=\min_\ell c_\ell replaces the ambient dimension ∑ℓmℓnℓ\sum_\ell m_\ell n_\ell under the square root in the 2D block bound with the manifold dimension dM=∑ℓdrℓd_{\mathcal M}=\sum_\ell d_{r_\ell}, up to an outer heterogeneity factor κ=max⁡ℓcℓ/min⁡ℓcℓ\kappa=\max_\ell c_\ell/\min_\ell c_\ell (κ=1\kappa=1 for identical block shapes and κ≈3.2\kappa\approx3.2 across all OPT-13B blocks; for a 5120×51205120\times5120 block at r=4r=4, drd_r is 640×640\times smaller than mnmn). In addition, our proof analyzes the implemented off-manifold tangent-line queries directly without requiring two extra SVD retractions per step (Proposition 1 and Lemma 2).

Seed replay already gives compact storage.

Seed replay compresses checkpoint files on disk while still reconstructing a dense weight matrix at load time. Serving a model then requires replaying all TT full-parameter perturbations (e.g., 20,000 steps) to materialize WT\boldsymbol W_T or keeping a separate dense model copy per task, whereas a rank-rr adapter attaches directly to the shared frozen weights with r(m+n+1)r(m+n+1) scalars per block. Both mechanisms are complementary: our implementation likewise regenerates per-step random matrices from seeds during training to avoid perturbation buffers (Section 4.3), while outputting an explicit rank-rr adapter.

[R2] Chen et al., ICLR, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all four reviewers for their careful reading and constructive comments. Three concerns are shared: (A) why the accumulated update is kept at a fixed rank, (B) how our method relates to RoZO and factor-space ZO, and (C) accuracy versus computational cost.

(A) Why the accumulated update is kept at a fixed rank. A low-rank per-step perturbation does not keep the accumulated update low-rank. For LOZO [R2], WT−W0\boldsymbol W_T-\boldsymbol W_0 reaches a median effective rank (90% spectral energy) of 1,019/2,048 on OPT-1.3B and 2,466/5,120 on OPT-13B (Appendix D.11), returning a dense matrix per block. Under our fixed-rank constraint, each block stays at exact rank rr and stores r(m+n+1)r(m+n+1) scalars (7.5 MB total on OPT-1.3B at r=4r=4, Appendix D.17). Each update can thus be saved and swapped per task alongside the frozen backbone, like LoRA, while keeping the peak training memory of MeZO and LOZO (Table 6). In Theorem 2 (Appendix C.3), this constraint also replaces the ambient dimension ∑ℓmℓnℓ\sum_\ell m_\ell n_\ell under the square root in the leading O(T−1/2)\mathcal O(T^{-1/2}) 2D block bound with the manifold dimension ∑ℓrℓ(mℓ+nℓ−rℓ)\sum_\ell r_\ell(m_\ell+n_\ell-r_\ell), up to κ≈3.2\kappa\approx3.2 on OPT-13B (640×640\times smaller on a 5120×51205120\times5120 block at r=4r=4; see reply to 5PqU).

(B) Relation to RoZO and factor-space ZO. RoZO [R3] also maintains a rank-rr update X=BA\boldsymbol X=\boldsymbol B\boldsymbol A on the fixed-rank manifold and queries along tangent directions, so the fixed-rank formulation is shared. Whereas RoZO forms tangent perturbations from the two factors (with factor-space preconditioning, momentum transport, and trust-region rules), we project an ambient Gaussian matrix orthogonally onto TXMrT_{\boldsymbol X}\mathcal M_r and normalize its norm per block without extra optimizer state, analyzing the implemented off-manifold finite difference directly (Propositions 1–2, Theorems 1–2).

Both methods query in TXMrT_{\boldsymbol X}\mathcal M_r under different distributions. With standard Gaussian factor perturbations, the unpreconditioned direction Δ=B ΔA+ΔB A\boldsymbol\Delta=\boldsymbol B\,\Delta\boldsymbol A+\Delta\boldsymbol B\,\boldsymbol A gives the leading-order mean step E[⟨G,Δ⟩Δ]=BB⊤G+GA⊤A\mathbb E[\langle\boldsymbol G,\boldsymbol\Delta\rangle\boldsymbol\Delta]=\boldsymbol B\boldsymbol B^\top\boldsymbol G+\boldsymbol G\boldsymbol A^\top\boldsymbol A for gradient G\boldsymbol G. Rescaling to (cB,A/c)(c\boldsymbol B,\boldsymbol A/c) preserves X\boldsymbol X while changing this mean step to c2BB⊤G+c−2GA⊤Ac^2\boldsymbol B\boldsymbol B^\top\boldsymbol G+c^{-2}\boldsymbol G\boldsymbol A^\top\boldsymbol A, and balanced factors still weight tangent components by the singular values of X\boldsymbol X. Our normalized estimator gives mean (mn/dr) PTX(G)(mn/d_r)\,P_{T_{\boldsymbol X}}(\boldsymbol G) (Proposition 2), depending only on the column and row spaces of X\boldsymbol X. RoZO has no public code, so a matched empirical comparison cannot be run at present.

(C) Accuracy and cost. Contribution 4 claims competitive accuracy with gains in several settings, and the task averages from Tables 1–2 requested by Reviewers 2qFF and NdoH agree:

Average score (%)MeZOMeZO-LoRALOZOZO-RGDZO-RGD-S
OPT-13B, 11 tasks (Table 1)68.7669.3470.62–70.34
OPT-13B, 10 tasks excl. ReCoRD67.5168.1269.5568.6369.17
OPT-30B, 6 tasks (Table 2)71.52–72.9373.0073.71

The second row accounts for the diverged ReCoRD run of ZO-RGD (Table 1); on average, both variants are on par with LOZO. Across multi-seed controlled runs, the consistent gain is on OPT-13B SQuAD: 84.9±1.384.9\pm1.3 for ZO-RGD-S vs. 82.2±0.782.2\pm0.7 for LOZO (Table 3), and +3.2+3.2 F1 over LOZO on 5/5 paired seeds at a shared learning rate (Table 16).

Both variants use two forward passes per step, as MeZO and LOZO do, and take 1.8×1.8\times the step time of LOZO (Table 6) to generate and project dense Gaussian matrices. At the shared rate 10−710^{-7} on OPT-13B SQuAD, ZO-RGD and ZO-RGD-S reach mean development losses of 0.466 (3/5 completed seeds) and 0.464 (5/5 seeds) at step 8,000, using 73% of the wall-clock time LOZO takes over 20,000 steps to reach 0.490. On RTE, reaching LOZO's final loss takes 1.1–1.5$\times$ LOZO's wall-clock time, giving no wall-clock speedup there.

[R1] Malladi et al., NeurIPS, 2023. [R2] Chen et al., ICLR, 2025. [R3] Song and Li, EACL, 2026. [R4] Zhao et al., ICLR, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.