- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes AIM-ZO, a zeroth-order (ZO) fine-tuning method for LLMs that improves the usefulness of forward evaluations without using backpropagation. Unlike MeZO and activation-based AGZO, AIM-ZO accumulates forward-activation information during training to maintain a broad evolving subspace of width , while each perturbation uses a smaller active subspace of width . On five LLMs (0.6B–30B) across 11 tasks, AIM-ZO achieves higher average performance under matched forward-evaluation budgets.
Critical strengths
-
Using activation to construct subspaces has been explored in prior work. This paper instead maintains a decomposition of subspace into shared and varying parts: one component is fully included (always present), while the other component can be randomly sampled. This idea seems plausible.
-
Under the full fine-tuning setting, the paper validates the effectiveness of the proposed algorithm.
Critical weaknesses
-
Subspace construction is unclear. I don’t quite understand how the proposed structured subspace is constructed. In particular, the role of Oja’s rule is not explained clearly to me—how it works, why it’s needed, and what its practical effect is.
-
Hyperparameter setting () is missing or unclear. It would help to explain how should be set, and what factors determine its value.
-
Theory is too complex for intuition. The theoretical proof is very complicated and difficult to follow, and it doesn’t really help readers build intuition about the underlying ideas behind the algorithm.
-
Experiments don’t cover LoRA, and AGZO results are concerning. All experiments are conducted only under full fine-tuning, with no results under LoRA, which makes it harder to assess the method in parameter-efficient scenarios. Also, the AGZO results are unexpectedly poor—compared with MeZO, the performance drops much more than I would expect.
Questions for authors
Please see the weaknesses listed above.
5 comments
Response to Reviewer rQu6 (Part 2/2)
Weakness 4. LoRA Evaluation and AGZO Results
Experiments don’t cover LoRA, and AGZO results are concerning. All experiments are conducted only under full fine-tuning, with no results under LoRA, which makes it harder to assess the method in parameter-efficient scenarios. Also, the AGZO results are unexpectedly poor—compared with MeZO, the performance drops much more than I would expect.
AIM-ZO currently perturbs original weight matrices and has not been adapted to LoRA's low-rank parameterization. We therefore compare MeZO+LoRA with AIM-ZO tuning the layers targeted by LoRA. Corresponding results are:
| Task / metric | MeZO+LoRA | AIM-ZO, selected layers |
|---|---|---|
| RTE / accuracy | 77.86 | 78.10 |
| WiC / accuracy | 56.43 | 57.11 |
| CB / accuracy | 84.52 | 84.52 |
| MultiRC / F1a | 73.30 | 73.52 |
| DROP / F1 | 33.69 | 35.37 |
The advantage narrows in this setting. This comparison directly applies AIM-ZO to the selected layers, without adaptation to LoRA's low-rank parameterization.
We performed implementation-consistency checks for AGZO, LoZO, and CurvZO against their authors' implementations using matched objectives, batches, and seeds. Taking AGZO as an example:
| Check | Scope | Result |
|---|---|---|
| Real-model paired run | 20 steps: losses, finite differences, batches, sampled parameters and displacements | Exact match |
| Full-model parameter hashes | All parameters at initialization and completion | Identical SHA256 hashes |
| Direct author-function CPU tests | FP32/FP16/BF16, 3 steps each; all parameters at every step | All pass torch.equal |
The original AGZO Qwen experiments use FP32; our comparisons use BF16. AGZO remains competitive on other datasets, suggesting that the SQuAD anomaly may reflect instability under BF16. In these runs, about 64% of steps give identical positive- and negative-perturbation losses, eliminating the finite-difference signal. This is consistent with precision-related loss of the perturbation signal. The implementation checks and training diagnostics above are documented.
We sincerely hope our responses have addressed your concerns, and that you could reconsider your rating toward a positive score. If you have any further questions, please feel free to follow up, and we are glad to address them.
Dear Reviewer rQu6,
We would appreciate your feedback on our earlier responses. We have addressed your concerns by:
- Explaining how Oja’s rule accumulates activation information across training steps to maintain an evolving perturbation subspace.
- Clarifying the setting of and its role in balancing adaptation speed and stochastic tracking error.
- Explaining the theoretical intuition and how the analysis supports the two core designs of AIM-ZO.
- Providing a selected-layer comparison between MeZO+LoRA and AIM-ZO applied to the corresponding original Q/V weights.
- Presenting implementation-consistency checks against the authors’ reference implementations, with detailed numerical evidence for AGZO.
- Explaining the anomalous AGZO results using the observed loss of finite-difference signals under BF16.
We sincerely hope that these clarifications and additional results address your concerns and may encourage a more favorable reconsideration of our paper. Please let us know if any questions remain; we would be happy to discuss them further.
Best regards, Submission 1869 Authors
Response to Reviewer rQu6 (Part 1/2)
Thank you for the comments. We address the concerns below.
Oja continuously tracks the activation principal subspace as training evolves. For each linear layer, St=Ht⊤Ht/nt,Qt=qf((I+ηqSt)Qt−1). The previous basis carries historical information, current activations adjust its directions, and QR restores orthonormality. AGZO reconstructs a space from the current batch; AIM-ZO recursively accumulates information across steps, maintaining a wider space from which smaller active spaces are selected. Section 3.2 gives the update and algorithm.
Appendix D.1 compares Oja with AGZO's current-batch PI-5 construction: Oja achieves higher mean gradient capture in five of six examined layers on independent evaluation batches. A five-seed OPT-2.7B/RTE ablation also supports continual updating:
Appendix C specifies ηq=0.3 for the main experiments. The tracking theorem in Section 4.1 gives Eεtrk,t2≲e−ηqγt/2εburn2+ηq2γ2ωˉ2+γKηqΣ2+failure terms. Within the theorem's admissible range, increasing the step size accelerates initialization decay and reduces tracking lag, but increases stochastic error. Its choice balances adaptation against stability and depends on target drift, observation second moments, eigengap, and maintained width. Appendix A states the sufficient conditions. Appendix D.1 compares 0.03, 0.1, 0.3, and 0.6 through frozen-checkpoint replay, reporting projector error and gradient capture.
The assessment that a theoretical analysis is insufficiently intuitive is partly subjective. We clarify the intuition and proof structure below and welcome feedback on the specific steps that would benefit from further explanation.
Section 4 follows the two core designs: maintaining gradient-relevant information across training and separating maintained from active width.
Section 4.1 rewrites QR–Oja as a projector update, obtains contraction from the eigengap, and controls drift and stochastic observations. The capture proposition connects the maintained space to gradient energy through structural, activation-alignment, and tracking residuals.
Section 4.2 computes expected retained energy from direction-inclusion probabilities, then uses a Gaussian cosine identity to obtain an alignment lower bound. With sufficient energy retention, it exceeds expected alignment under full activation. Section 4.3 examines RLOO in high-capture spaces through linear controls, real-loss diagnostics, and online ablations.
So, could the reviewer identify the specific propositions, assumptions, or derivation steps that are difficult to follow? We can provide detailed derivations for those steps.