acceptodds
Withdrawn from ICLR 2027

AGOZO: Activation-Guided Orthogonalized Zeroth-Order Optimization for LLM Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zeroth-order (ZO) fine-tuning reduces memory use by replacing backpropagation with forward evaluations, but limited queries produce noisy gradient estimates. We propose Activation-Guided Orthogonalized ZO (AGOZO), which combines activation-guided queries with gradient orthogonalization. AGOZO uses input activations to select a low-rank query subspace, estimates the gradient within that subspace, and replaces the estimate's nonzero singular values by one before mapping it back to the full weight matrix. Our theory quantifies how much gradient information this operation preserves. Assuming exact directional derivatives along Gaussian perturbations, we derive upper and lower bounds on the update's average alignment with the true gradient. These bounds agree up to constant factors when the subspace rank and query budget are small relative to the layer's output dimension. We also characterize when orthogonalization provides a stronger one-step loss-reduction guarantee than raw activation-guided ZO under a common smoothness assumption. This comparison considers growing matrix dimensions while keeping the number of queries fixed. We further quantify how sharing queries across layers weakens each layer's alignment. Across three language-model backbones and nine tasks, experiments with three seeds and a matched budget of 40,000 forward evaluations show that AGOZO achieves the highest reported mean validation score or a rounded tie on 16 of 27 model–task pairs.

Primary area:
optimization
Keywords:
Zeroth-Order Optimization, Activation guided Optimizatoin, Orthogonalized Updates

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KgQCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AGOZO (Activation-Guided Orthogonalized Zeroth-Order Optimization) to fine-tune language models without backpropagation, using only forward evaluations. It first uses layer activations to find a low-rank subspace for informative update directions, then performs zeroth-order gradient estimation in that subspace and applies orthogonalization to remove noisy singular-value magnitudes while preserving key singular directions. Experiments show AGOZO is best or tied on some model–task pairs.

Critical strengths

It sounds reasonable to construct a subspace using activations and then apply Muon within that subspace, instead of projecting to the full parameter space and applying Muon afterward. However, both components of this idea—(i) using activations to build a subspace and (ii) applying Muon in a subspace—have already been proposed in prior work. As such, the paper largely presents a combination of these two existing ideas rather than introducing a new, fundamentally distinct contribution.

[1] Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization, NeurIPS, 2026.
[2] AGZO Activation-Guided Zeroth-Order Optimization for LLM Fine-Tuning. ICML, 2026.

Critical weaknesses
  • The paper largely presents a combination of these two existing ideas rather than introducing a new, fundamentally distinct contribution. As a result, the innovation appears limited.
  • The theoretical justification feels somewhat heavy. It is hard to build intuition or extract clear insights.
  • In the main text, please include a clear algorithm that explains the entire workflow step by step. In particular, the paper contributions should be stated more explicitly in the main body (not only in the abstract/introduction), and the algorithm should help the readers understand how each component works together.
  • Experiments are not strong enough. The experimental section is currently too weak. Please provide more comprehensive experiments and stronger evidence to support the claims.
  • Baseline results in Table 2 look unusually poor. The baseline results reported in Table 2 seem significantly worse than expected. In particular, for SST-2 and SQuAD, the baselines are much lower than those typically reported in prior work.
Questions for authors

Please see the weaknesses listed above.

Reviewer_iMfNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes AGOZO, basically AGZO + ZO-Muon. The authors provide theoretical analyses of the gradient alignment and one-step descent guarantees of the resulting updates. Experiments on three language models and nine downstream tasks demonstrate competitive performance against several ZO baselines.

Critical strengths
  1. Improving the query efficiency of zeroth-order LLM fine-tuning is an important problem.
  2. The paper develops several theoretical results, including spectrum-sensitive alignment bounds for noisy matrix-sign estimates and a fixed-query comparison of descent certificates.
Critical weaknesses
  1. My primary concern is the extremely limited algorithmic novelty. AGOZO appears to be a straightforward combination of two existing methods: AGZO, which selects an activation-guided query subspace, and ZO-Muon, which applies matrix-sign orthogonalization to a ZO gradient estimate. The paper does not clearly explain why these two techniques should be combined beyond the intuition that each is individually useful. In particular, it is unclear whether their combination introduces any new mechanism or complementary benefit that cannot be obtained from either method alone. The experimental evidence does not fully resolve this concern. Although AGOZO outperforms AGZO, its improvement over ZO-Muon is relatively modest and inconsistent across models.

  2. Although the theoretical analysis is technically substantial, I find its connection to the central algorithmic contribution rather weak. In particular, Theorem 4.1 studies the alignment between a generic gradient matrix and the matrix sign of its noisy Gaussian ZO estimate. Similarly, Theorem 4.4 compares the descent certificates of raw and orthogonalized ZO estimates. Neither result fundamentally relies on the subspace being selected using activations. The same analyses appear applicable to ZO-Muon with an arbitrary orthonormal projection. The activation coverage ratio introduced in the paper quantifies how much gradient information is retained by the selected subspace, but this is largely independent of the matrix-sign analysis. The theory effectively combines two separate properties rather than explaining a specific interaction between activation-guided projection and orthogonalization. As a result, the main theoretical contribution seems closer to a general analysis of ZO-Muon than an explanation of why AGOZO, as a combination of AGZO and ZO-Muon, should be particularly effective. This creates a gap between the proposed method and the theoretical results used to justify it.

  3. The manuscript is unnecessarily lengthy and repetitive. Several sections repeatedly explain similar intuitions, theoretical implications, and limitations without significantly advancing the discussion. The main scientific contributions are consequently difficult to distinguish from the extensive supporting analysis. The writing is also frequently formulaic and overly elaborate, giving the impression of an AI-assisted draft that has not received sufficient editorial refinement. My concern is not the use of AI tools itself, but the lack of a concise and carefully organized scientific narrative. The paper would benefit from substantial rewriting to clarify its core motivation, emphasize its genuinely novel insights, and remove redundant discussions.

Questions for authors

I am quite familiar with ZO and ZO for LLM fine-tuning. To me, this submission has nothing interesting or exciting. I don't have any questions for the authors either.

Reviewer_XHCBOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents AGOZO, which combines activation-guided subspace sampling with matrix orthogonalization for zeroth-order LLM fine-tuning. It estimates the gradient within an activation subspace, orthogonalizes the estimate, and maps the update back to the parameter space. The authors analyze the descent signal retained with few queries and evaluate the method on three models and nine tasks. However, I still have questions about how well the theory explains the practical gains, why the gains over ZO-Muon vary across models, and whether the ablations isolate the contribution of orthogonalization.

Critical strengths
  1. The motivation is clear. Activation guidance determines which subspace to sample in, while matrix orthogonalization changes how the resulting gradient estimate is used. The paper distinguishes these two questions. Since gradient estimates are noisy with few queries, understanding whether orthogonalization preserves a useful descent signal is a worthwhile problem.

  2. The experiments cover nine tasks on OPT-1.3B, OPT-13B, and Llama-3-8B. The main table includes directly relevant baselines such as AGZO and ZO-Muon, with results over three seeds under a budget of approximately 40,000 forward evaluations. The appendix also reports training time, peak GPU memory, and a breakdown of operation timings, providing a fairly complete account of the experimental costs.

  3. The authors explain the scope of the theoretical results clearly. They distinguish one-step descent guarantees from actual training performance and discuss limitations arising from activation projection, queries shared across layers, and numerical approximations. This helps readers understand which conclusions the theory supports.

Critical weaknesses
  1. I would like to better understand how well the new theory explains the practical gains from combining the two components. Activation-guided subspaces and orthogonalization within subspaces have already been studied in AGZO and ZO-Muon, respectively. The theory here mainly concerns ideal directional derivatives and exact polar factors, whereas training uses finite differences and a Newton–Schulz approximation. The authors provide a conditional error analysis and numerical checks under idealized conditions, but have not quantified the approximation error in the actual implementation. It therefore remains unclear how much these approximations affect the mechanism described by the theory.

  2. AGOZO's gains over ZO-Muon vary noticeably across models. Appendix O reports an average task-score improvement of 0.6 points across all 27 configurations, but an average decrease of 1.4 points on Llama-3-8B. The authors acknowledge these differences and note that some gaps remain unresolved with three seeds. I think this deserves further discussion: why does AGOZO perform differently relative to ZO-Muon across models? The current results do not make it easy to identify when this combination is more advantageous.

  3. The "2×2" ablation in Appendix O changes several settings at once. The methods without orthogonalization sample one direction per step, while those with orthogonalization sample four; the finite-difference schemes and numbers of updates also differ. Although the total forward-evaluation budgets are approximately matched, it is still difficult to separate the gains due to orthogonalization itself. The random and activation subspaces also differ in projection side and rank. The authors acknowledge these limitations and suggest further controls in Appendix A, but a controlled training experiment isolating the contribution of orthogonalization is still needed.

Questions for authors

Could the authors compare exact polar updates with Newton–Schulz updates in actual fine-tuning, or measure the approximation error discussed in Appendix F.5? I would also appreciate further analysis of the performance differences between OPT and Llama to help identify when AGOZO is more suitable. For the component ablation, I think the most useful addition would be to fix the activation subspace, rank, query count, and finite-difference scheme, give the raw, per-layer scalar-normalized, and orthogonalized updates comparable hyperparameter-tuning budgets, and compare their final performance. This would more directly test whether the gains come from orthogonalization itself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.