- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors provides an activation-informed ZO fine-tuning method that uses input activations to define a fixed low-rank parameterization. Each adapted weight update is represented by a frozen activation-informed basis and a small trainable coefficient matrix. A dimension–alignment trade-off was analyzed as well.
Critical strengths
- Relative to AGZO, the paper uses a fixed low-rank basis throughout fine-tuning. This is a simple design choice, and the reported experiments show strong performance compared with the other zeroth-order baselines evaluated.
Critical weaknesses
- The proposed method appears closely related to AGZO (ICML 2026): both use activation-informed subspaces for zeroth-order fine-tuning, while the main distinction here appears to be that the basis is computed once and held fixed. Section 3 also substantially overlaps with AGZO’s methodological discussion. The paper should identify its distinct technical contributions more explicitly and clarify which ideas are inherited from AGZO.
- Because the relevant activation subspace may change during fine-tuning, it is unclear why a basis computed before training should remain preferable to AGZO’s dynamically updated basis. The authors should explain the expected benefit of freezing it and quantify the computation and memory saved relative to updating it during training.
- Section 5.5 shows that the fixed basis remains informative relative to random and weight-based subspaces. However, comparison with AGZO’s dynamically computed basis would more directly test whether freezing the basis sacrifices meaningful gradient alignment.
- The basis is computed from a single calibration batch, but the paper does not explain how that batch is selected. An ablation across different calibration batches would help establish whether the reported fine-tuning results are robust to this choice.
- The methodology should clarify how the proposed procedure handles nonlinear components of the model
Questions for authors
- What is the technical contribution beyond AGZO, apart from computing the basis once and freezing it? This distinction is central to assessing novelty.
- Please distinguish the parts of Section 3 that are new from those adapted from AGZO.
- How is the single calibration batch selected? How sensitive are the reported results to the choice of calibration batch? Some ablation study would help determine whether the method’s performance depends on a favorable calibration batch.
3 comments
Thanks for adding the comparison with AGZO. The proposed method performs better on the evaluated dataset, but I still find the additional novelty limited.
Thanks for explaining the results for the fixed basis. However, the finding that it captures less instantaneous gradient energy seems at odds with the main motivation. I recognize that the empirical results are good, but it is still unclear why the fixed basis works better.
Thanks for checking the robustness of the basis selection.
I appreciate the authors’ efforts to address my questions, but I will keep my original score.
Responses to Reviewer Tct7's further comments
We thank the reviewer for the follow-up and for acknowledging the AGZO comparison and the robustness analysis of our basis selection.
Novelty and key difference from AGZO. The low-rank structure of layer input activations is used in methods such as EVA, CorDA, and AGZO. The distinction lies in how this shared property is used. AGZO uses it to select perturbation directions in weight space at each step. ZO-Act uses it to define the trainable variables of the zeroth-order optimization problem. Specifically, ZO-Act represents the weight update as and applies both perturbations and optimizer updates directly to . With fixed, standard Adam accumulates moment estimates in the same coefficient space throughout training. The same parameterization enables zeroth-order fine-tuning of the quantized models using much less memory.
Gradient energy of the fixed basis. AGZO’s higher instantaneous gradient-energy capture is expected. At each step, AGZO constructs its basis from the current mini-batch activations, which also enter the computation of the weight gradient for that batch. This gives AGZO’s basis a natural advantage in capturing the gradient at that step. The goal of ZO-Act is to retain useful gradient directions at very low rank. The fixed activation-informed basis captures more gradient energy than a random basis, and the ablation in Table 14 shows that periodically refreshing it did not improve task accuracy. These results suggest that the initial basis retains useful gradient directions throughout fine-tuning.
Benefits of the fixed coefficient space. The fixed coefficient space gives Adam consistent coordinates in which to accumulate moment estimates across steps. As shown in Table 1, switching from SGD to Adam improves ZO-Act’s average performance on Llama-3-8B. ZO-Act also shows lower standard deviations than AGZO across four seeds on RTE, BoolQ, and GSM8K in Table 19. Computing the basis takes 1.5 seconds at initialization, and Table 7 reports runtimes of 42 minutes for ZO-Act and 47 minutes for AGZO. The learned update is an explicit low-rank adapter, so we can save only and for each task and reuse the same frozen backbone. The same parameterization enables direct zeroth-order fine-tuning of quantized models.
Comparison on AGZO's Qwen3-4B benchmarks. We further evaluate ZO-Act on all seven Qwen3-4B benchmarks reported by AGZO, with the AGZO results taken from the AGZO paper. ZO-Act scores higher on five of the seven tasks and ties on BoolQ. The difference on CB corresponds to a single example of the 56-example evaluation set. These results are included in Table 20 of Appendix A.11.
| Qwen3-4B | SST-2 | RTE | CB | BoolQ | WiC | MultiRC | SQuAD |
|---|---|---|---|---|---|---|---|
| AGZO | 89.2 | 84.8 | 87.5 | 82.0 | 67.8 | 85.3 | 87.6 |
| ZO-Act | 92.7 | 85.6 | 85.7 | 82.0 | 68.7 | 86.6 | 89.1 |
Response to Reviewer Tct7 for ZO-Act
We thank the reviewer for the careful evaluation and constructive feedback. We have clarified the methodological distinctions between ZO-Act and AGZO and added further experiments on gradient alignment, basis refreshing, and calibration robustness to address the concerns raised. We respond to each point below.
W1 / Q1 / Q2. Contribution beyond AGZO.
We added a dedicated comparison with AGZO in Appendix A.11. Both methods build upon the low-rank structure of neural network activations and gradients, which has been extensively studied in prior work, but exploit this property differently. AGZO constructs activation-guided perturbations from each mini-batch and directly updates the full weights by ZO-SGD. ZO-Act instead uses a one-shot activation basis to define a fixed low-dimensional parameterization Weff=W0+V0B, and performs both perturbation and optimization only in the explicit coefficient space B.
Regarding Section 3, the activation–gradient relationship used to motivate subspace selection follows directly from the chain rule and standard SVD properties and is not specific to AGZO. ZO-Act was developed independently, and we became aware of AGZO only after developing our method. The fixed coefficient-space parameterization and the corresponding forward-only optimization procedure are our methodological contributions, rather than adaptations of AGZO.
The key distinction lies in the explicit coefficient-space parameterization, which provides consistent low-dimensional coordinates for optimization and enables direct use of Adam with lightweight moment states of size r×n per layer. By keeping the backbone frozen, ZO-Act also naturally supports INT4 quantization, requiring only 5.8 GB of memory and achieving 83.4 on RTE compared with 69.7 for QZO in Table 4. The effectiveness of the activation-informed subspace is further supported by matched-setting experiments: ZO-Act achieves 87.0 on RTE compared with 52.0 for a random frozen basis in Table 10. Together with the dimension–alignment analysis in Section 4, these results highlight the advantages of the proposed coefficient-space optimization framework.
ZO-Act achieves higher mean accuracy than AGZO on all four tasks over four seeds, as shown in Table 19.
W2 / W3. Why a fixed basis and comparison with the dynamic basis of AGZO.
The basis of AGZO is computed in every mini-batch and captures somewhat more gradient energy than the fixed basis, as shown in Table 13.
However, greater instantaneous gradient capture does not necessarily lead to better downstream accuracy. In our periodic-refresh ablation, updating the ZO-Act basis every 100 steps does not improve performance on RTE, CB, or BoolQ, as shown in Table 14.
Importantly, keeping the basis fixed provides consistent low-dimensional coordinates for optimization, enabling the direct use of Adam with lightweight optimizer states and naturally supporting frozen INT4 backbones. Despite capturing less instantaneous gradient energy, ZO-Act achieves higher mean accuracy than AGZO across RTE, CB, BoolQ, and GSM8K over four seeds, as shown in Table 19. Moreover, ZO-Act requires only 1.5 seconds for one-shot subspace initialization, while AGZO has slightly higher runtime with comparable memory usage, as shown in Table 7.
W5. Nonlinear components.
ZO-Act updates only the coefficient matrices associated with linear projections. Embeddings, normalization parameters, biases, and output heads remain frozen, as described in Appendix B, while nonlinear operations are preserved in the forward computation.
W4 / Q3. Calibration batch.
ZO-Act uses the first training mini-batch without any special selection. Downstream accuracy is stable across three calibration batches over four seeds, as shown in Table 6.
The bases constructed from four disjoint calibration batches are also highly consistent, as shown in Table 15.
These results suggest that ZO-Act is robust to the choice of calibration batch and does not rely on a particularly favorable initialization in the evaluated settings.