- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes AGOZO (Activation-Guided Orthogonalized Zeroth-Order Optimization) to fine-tune language models without backpropagation, using only forward evaluations. It first uses layer activations to find a low-rank subspace for informative update directions, then performs zeroth-order gradient estimation in that subspace and applies orthogonalization to remove noisy singular-value magnitudes while preserving key singular directions. Experiments show AGOZO is best or tied on some model–task pairs.
Critical strengths
It sounds reasonable to construct a subspace using activations and then apply Muon within that subspace, instead of projecting to the full parameter space and applying Muon afterward. However, both components of this idea—(i) using activations to build a subspace and (ii) applying Muon in a subspace—have already been proposed in prior work. As such, the paper largely presents a combination of these two existing ideas rather than introducing a new, fundamentally distinct contribution.
[1] Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization, NeurIPS, 2026.
[2] AGZO Activation-Guided Zeroth-Order Optimization for LLM Fine-Tuning. ICML, 2026.
Critical weaknesses
- The paper largely presents a combination of these two existing ideas rather than introducing a new, fundamentally distinct contribution. As a result, the innovation appears limited.
- The theoretical justification feels somewhat heavy. It is hard to build intuition or extract clear insights.
- In the main text, please include a clear algorithm that explains the entire workflow step by step. In particular, the paper contributions should be stated more explicitly in the main body (not only in the abstract/introduction), and the algorithm should help the readers understand how each component works together.
- Experiments are not strong enough. The experimental section is currently too weak. Please provide more comprehensive experiments and stronger evidence to support the claims.
- Baseline results in Table 2 look unusually poor. The baseline results reported in Table 2 seem significantly worse than expected. In particular, for SST-2 and SQuAD, the baselines are much lower than those typically reported in prior work.
Questions for authors
Please see the weaknesses listed above.