- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper formulates forward-only LLM adaptation as zeroth-order optimization of persistent additive updates constrained to fixed-rank matrix manifolds. Rather than optimizing redundant Euclidean factors , each learned update matrix is maintained directly on a rank- manifold. The proposed estimator projects Gaussian perturbations onto the tangent space and normalizes them to correct the rank- and shape-dependent perturbation scale. Based on this estimator, the paper introduces ZO-RGD and the scalar-smoothed ZO-RGD-S. The theoretical analysis shows first-order agreement between ambient and retracted finite differences, expected alignment of the normalized estimator with the Riemannian gradient up to a known scaling and bias, and a conditional non-asymptotic convergence result. Experiments cover OPT-1.3B/13B/30B, Llama-3.1-8B, and Qwen2.5-7B, together with controlled comparisons, geometry ablations, rank and precision diagnostics, and runtime/memory measurements.
Critical strengths
The paper does more than report endpoint benchmark numbers. It includes multiple model families, controlled self-run baselines, geometry/parameterization ablations, rank and numerical-stability analyses, and explicit runtime/memory profiling. In the controlled OPT comparison, ZO-RGD-S substantially improves RTE over MeZO/LOZO and also improves SQuAD, while the paper clearly separates the stronger first-order LoRA result from the forward-only ranking.
Critical weaknesses
The paper motivates its formulation by contrasting direct optimization of a fixed-rank update matrix with methods that operate through low-rank adapter factors, and describes RoZO as using geometry induced by low-rank adapter factors while the present work “takes the matrix update itself as the optimization variable on the fixed-rank manifold.” However, this distinction appears substantially narrower than stated. RoZO already formulates the low-rank update as a point on a fixed-rank manifold and performs tangent-space zeroth-order optimization together with rank-preserving geometric updates. Thus, both approaches appear to share the central formulation of maintaining a persistent low-rank additive update on a fixed-rank manifold, rather than merely imposing low rank on individual perturbations. The more specific novelty of the present paper seems to lie in its particular tangent-direction construction, ambient Gaussian sampling followed by orthogonal tangent projection and explicit norm normalization, and in the associated finite-difference analysis, rather than in the broader fixed-rank-manifold formulation itself. I would like the paper to sharpen this distinction and avoid presenting the latter as a major conceptual departure if the optimization object is already shared by prior/concurrent work.
Questions for authors
Could the authors provide a more direct head-to-head comparison with RoZO under the same backbone, tasks, rank, number of function evaluations, learning-rate tuning budget, and checkpoint-selection protocol? Given the conceptual proximity of the two methods, such a comparison would help establish whether the proposed ambient-Gaussian projection and normalization provide an advantage beyond an alternative construction of tangent-space ZO directions.
Response to Reviewer 4gXg
We thank the Reviewer for the careful reading and for the precise characterization of the relationship to RoZO, which is also discussed in part (B) of the general response.
Replies to weaknesses:
The distinction from RoZO is narrower than stated.
We agree with the Reviewer's assessment. RoZO [R3] likewise maintains a persistent rank-r additive update X=BA on the fixed-rank manifold using tangent-space ZO directions and SVD retraction, so our phrasing in Section 2 overstated the formulation-level distinction. Both works share the persistent fixed-rank manifold state; our specific contribution lies in the estimator and analysis identified by the Reviewer: constructing isotropic tangent directions via orthogonal projection of ambient Gaussian matrices and per-block norm calibration (requiring no factor-space preconditioning, momentum transport, or trust-region heuristics), proving leading-order agreement and gradient alignment for off-manifold ambient tangent-line queries (Propositions 1–2), and establishing the resulting conditional stationarity bound (Theorems 1–2).
Replies to questions:
A head-to-head comparison with RoZO under the same protocol.
Because RoZO [R3] has no publicly available implementation, a fair head-to-head run under matched backbones, tuning budgets, and checkpoint rules cannot be executed without reconstructing several underspecified components (e.g., factor gauge after retraction, trust-region scheduling, and preconditioner details). Analytically, part (B) isolates what ambient Gaussian projection and normalization change relative to factor-induced tangent directions. For Δ=BΔA+ΔBA with standard Gaussian (ΔA,ΔB), the leading-order unpreconditioned mean step BB⊤G+GA⊤A varies under gauge rescaling (cB,A/c) and, even under balanced factors, weights tangent directions by the singular values of X (vanishing as Σ→0). Our normalized estimator yields mean (mn/dr)PTX(G), which is invariant to factor parameterization and singular-value magnitudes, allowing near-zero initialization (σinit=10−6, Appendix A.2). Empirically, Table 5 (right) compares our manifold update only against Euclidean ZO on (B,A) at rank 4, which does not represent full RoZO.
[R3] Song and Li, EACL, 2026.