acceptodds
Under review as a conference paper at ICLR 2027

ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces

Abstract

Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods typically perturb the full parameter space or rely on structured low-dimensional perturbations, still leading to noisy gradient estimates and limited optimization flexibility. We propose ZO-Act, an activation-informed ZO fine-tuning method that uses input activations to define a fixed low-rank parameterization. For each linear layer, ZO-Act computes a small activation basis once at initialization and optimizes only lightweight coefficient matrices using forward-only loss evaluations. This explicit coefficient-space parameterization reduces the effective perturbation dimension, enables direct use of standard momentum-based optimizers such as Adam, and naturally supports quantized LLM fine-tuning by keeping low-bit weights frozen. Our analysis characterizes a dimension-alignment trade-off: reducing the perturbation dimension improves ZO gradient estimation, while effective subspace optimization requires preserving sufficient gradient energy. We further show empirically that activation-informed subspaces capture substantially more gradient energy than random and weight-based alternatives. Across Llama-3-8B, OPT-13B, Qwen2.5-7B, Qwen2.5-32B, and INT4 Llama-3-8B, ZO-Act consistently achieves competitive performance against strong ZO baselines across language understanding, question answering, commonsense reasoning, and mathematical reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Fine-tuning, Zeroth-order optimization, Activation-informed subspaces, Low-rank adaptation, Forward-only training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Tct7OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors provides an activation-informed ZO fine-tuning method that uses input activations to define a fixed low-rank parameterization. Each adapted weight update is represented by a frozen activation-informed basis and a small trainable coefficient matrix. A dimension–alignment trade-off was analyzed as well.

Critical strengths
  • Relative to AGZO, the paper uses a fixed low-rank basis throughout fine-tuning. This is a simple design choice, and the reported experiments show strong performance compared with the other zeroth-order baselines evaluated.
Critical weaknesses
  • The proposed method appears closely related to AGZO (ICML 2026): both use activation-informed subspaces for zeroth-order fine-tuning, while the main distinction here appears to be that the basis is computed once and held fixed. Section 3 also substantially overlaps with AGZO’s methodological discussion. The paper should identify its distinct technical contributions more explicitly and clarify which ideas are inherited from AGZO.
  • Because the relevant activation subspace may change during fine-tuning, it is unclear why a basis computed before training should remain preferable to AGZO’s dynamically updated basis. The authors should explain the expected benefit of freezing it and quantify the computation and memory saved relative to updating it during training.
  • Section 5.5 shows that the fixed basis remains informative relative to random and weight-based subspaces. However, comparison with AGZO’s dynamically computed basis would more directly test whether freezing the basis sacrifices meaningful gradient alignment.
  • The basis is computed from a single calibration batch, but the paper does not explain how that batch is selected. An ablation across different calibration batches would help establish whether the reported fine-tuning results are robust to this choice.
  • The methodology should clarify how the proposed procedure handles nonlinear components of the model
Questions for authors
  • What is the technical contribution beyond AGZO, apart from computing the basis once and freezing it? This distinction is central to assessing novelty.
  • Please distinguish the parts of Section 3 that are new from those adapted from AGZO.
  • How is the single calibration batch selected? How sensitive are the reported results to the choice of calibration batch? Some ablation study would help determine whether the method’s performance depends on a favorable calibration batch.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Tct7 for ZO-Act

We thank the reviewer for the careful evaluation and constructive feedback. We have clarified the methodological distinctions between ZO-Act and AGZO and added further experiments on gradient alignment, basis refreshing, and calibration robustness to address the concerns raised. We respond to each point below.

W1 / Q1 / Q2. Contribution beyond AGZO.

We added a dedicated comparison with AGZO in Appendix A.11. Both methods build upon the low-rank structure of neural network activations and gradients, which has been extensively studied in prior work, but exploit this property differently. AGZO constructs activation-guided perturbations from each mini-batch and directly updates the full weights by ZO-SGD. ZO-Act instead uses a one-shot activation basis to define a fixed low-dimensional parameterization Weff=W0+V0BW_{\mathrm{eff}} = W_0 + V_0 B, and performs both perturbation and optimization only in the explicit coefficient space BB.

Regarding Section 3, the activation–gradient relationship used to motivate subspace selection follows directly from the chain rule and standard SVD properties and is not specific to AGZO. ZO-Act was developed independently, and we became aware of AGZO only after developing our method. The fixed coefficient-space parameterization and the corresponding forward-only optimization procedure are our methodological contributions, rather than adaptations of AGZO.

The key distinction lies in the explicit coefficient-space parameterization, which provides consistent low-dimensional coordinates for optimization and enables direct use of Adam with lightweight moment states of size r×nr\times n per layer. By keeping the backbone frozen, ZO-Act also naturally supports INT4 quantization, requiring only 5.8 GB of memory and achieving 83.4 on RTE compared with 69.7 for QZO in Table 4. The effectiveness of the activation-informed subspace is further supported by matched-setting experiments: ZO-Act achieves 87.0 on RTE compared with 52.0 for a random frozen basis in Table 10. Together with the dimension–alignment analysis in Section 4, these results highlight the advantages of the proposed coefficient-space optimization framework.

ZO-Act achieves higher mean accuracy than AGZO on all four tasks over four seeds, as shown in Table 19.

TaskAGZOZO-ActGain
RTE75.0 ± 5.583.3 ± 3.3+8.3
CB70.1 ± 1.780.4 ± 3.3+10.3
BoolQ82.9 ± 1.185.1 ± 0.8+2.2
GSM8K41.3 ± 2.545.3 ± 0.8+4.0

W2 / W3. Why a fixed basis and comparison with the dynamic basis of AGZO.

The basis of AGZO is computed in every mini-batch and captures somewhat more gradient energy than the fixed basis, as shown in Table 13.

BasisCaptured energy on RTECaptured energy on BoolQCaptured energy on GSM8K
ZO-Act fixed basis12.58%9.95%11.20%
AGZO dynamic basis17.96%13.20%12.15%

However, greater instantaneous gradient capture does not necessarily lead to better downstream accuracy. In our periodic-refresh ablation, updating the ZO-Act basis every 100 steps does not improve performance on RTE, CB, or BoolQ, as shown in Table 14.

BasisRTECBBoolQ
Fixed87.082.185.1
Refreshed every 100 steps84.880.484.9

Importantly, keeping the basis fixed provides consistent low-dimensional coordinates for optimization, enabling the direct use of Adam with lightweight optimizer states and naturally supporting frozen INT4 backbones. Despite capturing less instantaneous gradient energy, ZO-Act achieves higher mean accuracy than AGZO across RTE, CB, BoolQ, and GSM8K over four seeds, as shown in Table 19. Moreover, ZO-Act requires only 1.5 seconds for one-shot subspace initialization, while AGZO has slightly higher runtime with comparable memory usage, as shown in Table 7.

W5. Nonlinear components.

ZO-Act updates only the coefficient matrices associated with linear projections. Embeddings, normalization parameters, biases, and output heads remain frozen, as described in Appendix B, while nonlinear operations are preserved in the forward computation.

W4 / Q3. Calibration batch.

ZO-Act uses the first training mini-batch without any special selection. Downstream accuracy is stable across three calibration batches over four seeds, as shown in Table 6.

CalibrationRTEBoolQ
Default83.3 ± 3.385.1 ± 0.8
Batch A84.1 ± 1.384.4 ± 1.1
Batch B82.4 ± 2.085.0 ± 0.8

The bases constructed from four disjoint calibration batches are also highly consistent, as shown in Table 15.

TaskRTECBBoolQSST-2WiCAverage
Cosine similarity0.9280.9680.8650.9540.9940.942

These results suggest that ZO-Act is robust to the choice of calibration batch and does not rely on a particularly favorable initialization in the evaluated settings.

Thanks for adding the comparison with AGZO. The proposed method performs better on the evaluated dataset, but I still find the additional novelty limited.

Thanks for explaining the results for the fixed basis. However, the finding that it captures less instantaneous gradient energy seems at odds with the main motivation. I recognize that the empirical results are good, but it is still unclear why the fixed basis works better.

Thanks for checking the robustness of the basis selection.

I appreciate the authors’ efforts to address my questions, but I will keep my original score.

AuthorsOpenReview ↗

Responses to Reviewer Tct7's further comments

We thank the reviewer for the follow-up and for acknowledging the AGZO comparison and the robustness analysis of our basis selection.

Novelty and key difference from AGZO. The low-rank structure of layer input activations is used in methods such as EVA, CorDA, and AGZO. The distinction lies in how this shared property is used. AGZO uses it to select perturbation directions in weight space at each step. ZO-Act uses it to define the trainable variables of the zeroth-order optimization problem. Specifically, ZO-Act represents the weight update as V0BV_0B and applies both perturbations and optimizer updates directly to BB. With V0V_0 fixed, standard Adam accumulates moment estimates in the same r×nr\times n coefficient space throughout training. The same parameterization enables zeroth-order fine-tuning of the quantized models using much less memory.

Gradient energy of the fixed basis. AGZO’s higher instantaneous gradient-energy capture is expected. At each step, AGZO constructs its basis from the current mini-batch activations, which also enter the computation of the weight gradient for that batch. This gives AGZO’s basis a natural advantage in capturing the gradient at that step. The goal of ZO-Act is to retain useful gradient directions at very low rank. The fixed activation-informed basis captures more gradient energy than a random basis, and the ablation in Table 14 shows that periodically refreshing it did not improve task accuracy. These results suggest that the initial basis retains useful gradient directions throughout fine-tuning.

Benefits of the fixed coefficient space. The fixed coefficient space gives Adam consistent coordinates in which to accumulate moment estimates across steps. As shown in Table 1, switching from SGD to Adam improves ZO-Act’s average performance on Llama-3-8B. ZO-Act also shows lower standard deviations than AGZO across four seeds on RTE, BoolQ, and GSM8K in Table 19. Computing the basis takes 1.5 seconds at initialization, and Table 7 reports runtimes of 42 minutes for ZO-Act and 47 minutes for AGZO. The learned update V0BV_0B is an explicit low-rank adapter, so we can save only V0V_0 and BB for each task and reuse the same frozen backbone. The same parameterization enables direct zeroth-order fine-tuning of quantized models.

Comparison on AGZO's Qwen3-4B benchmarks. We further evaluate ZO-Act on all seven Qwen3-4B benchmarks reported by AGZO, with the AGZO results taken from the AGZO paper. ZO-Act scores higher on five of the seven tasks and ties on BoolQ. The difference on CB corresponds to a single example of the 56-example evaluation set. These results are included in Table 20 of Appendix A.11.

Qwen3-4BSST-2RTECBBoolQWiCMultiRCSQuAD
AGZO89.284.887.582.067.885.387.6
ZO-Act92.785.685.782.068.786.689.1
Reviewer_3nxdOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is well developed overall, with clear research motivation and a simple yet effective method. It demonstrates consistent empirical improvements across multiple models, diverse downstream tasks, and quantized settings. I believe its main contribution lies in transforming activation-informed subspaces into a fixed low-dimensional parameterization, enabling zeroth-order fine-tuning to directly leverage standard optimizers such as Adam while maintaining low memory overhead.

Critical strengths
  1. The paper is well organized, with clear motivation behind the proposed method. The authors explain the subspace selection strategy by examining the relationship between activation subspaces and weight gradients. The theoretical analysis and experimental design are also well connected.

2.Constructing a low-dimensional optimization space from the leading singular directions of input activations is supported by clear optimization intuition. By fixing the activation-informed basis and directly optimizing low-dimensional coefficient matrices, the proposed method effectively reduces the perturbation dimension.

3.The authors conduct comprehensive evaluations on Llama, OPT, and Qwen model families, scaling up to 32B parameters and including INT4 quantization experiments. ZO-Act achieves consistent performance improvements across various language understanding and reasoning tasks. Furthermore, ablation studies on subspace initialization, experiments with different ranks, and analyses of gradient capture throughout optimization provide strong empirical support for the effectiveness of activation-informed subspaces.

Critical weaknesses

I have not identified any major concerns so far.

Questions for authors

Could the authors further clarify whether the activation subspace constructed through one-shot calibration can maintain good gradient alignment across different calibration batches, data distributions, and longer optimization trajectories? For example, comparing fixed subspaces constructed from different calibration datasets with periodically updated activation subspaces.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3nxd for ZO-Act

We thank the reviewer for the constructive suggestions. We have added further experiments to evaluate the robustness of the one-shot activation-informed basis across calibration batches, tasks, and optimization trajectories.

Q1. Robustness of the one-shot basis.

Bases constructed from four disjoint calibration batches are highly consistent, as shown in Table 15.

TaskRTECBBoolQSST-2WiCAverage
Cosine similarity0.9280.9680.8650.9540.9940.942

Downstream accuracy is stable across calibration batches over four seeds, as shown in Table 6.

CalibrationRTEBoolQ
Default83.3 ± 3.385.1 ± 0.8
Batch A84.1 ± 1.384.4 ± 1.1
Batch B82.4 ± 2.085.0 ± 0.8

Refreshing the basis every 100 steps does not improve accuracy, as shown in Table 14.

BasisRTECBBoolQ
Fixed87.082.185.1
Refreshed every 100 steps84.880.484.9

The captured gradient energy stabilizes after about 100 steps during full fine-tuning in Figure 1, and the same qualitative ordering holds along the actual ZO-Act trajectory in Figure 2. Table 13 shows that the fixed basis retains nontrivial gradient alignment at the final checkpoint across RTE, BoolQ, and GSM8K, providing additional evidence of its effectiveness across different tasks.

Reviewer_GGmSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ZO-Act is a forward-only (zeroth-order) fine-tuning method for LLMs.

For each linear layer, it runs one calibration forward pass, takes the top-r right singular vectors V_r of the input activations, and freezes them. It then trains only the coefficient matrix B in ΔW = V_r B. The gradient estimates use Gaussian perturbations averaged over q directions (default r = 1, q = 8) and are fed to Adam. The frozen backbone makes INT4 fine-tuning straightforward.

The paper gives a non-convex ZO-GD bound in the coefficient space and evaluates on Llama-3-8B, OPT-13B, Qwen2.5-7B/32B, and INT4 Llama-3-8B against nine ZO baselines.

Critical strengths

A simple, practical method with a nice insight. ZO-Act needs one SVD per layer up front (1.5 s), adds no inference cost, and makes INT4 fine-tuning easy (5.8 GB). Moving ZO estimation into an explicit low-dimensional space, so that standard Adam becomes affordable, is a good idea. The forward-pass accounting is also transparent.

Some genuinely strong results.

  • Commonsense: 70.8 vs. 68.2 for the best baseline, best on seven of eight tasks (Table 2).
  • GSM8K: ZO-Act leads every baseline on each of four seeds, with a lower spread (45.3 ± 0.8 vs. at most 41.3 ± 2.5) at an equal or smaller budget (Table 3).

The subspace effect is now isolated (after rebuttal). In the matched Table 10 comparison, ZO-Act clearly beats MeZO-LoRA + Adam and the random-frozen-basis ZO-LoRA-FA, which differs only in the basis. The new INT4 ZO baselines point the same way.

Critical weaknesses

Most of my original concerns were addressed in the rebuttal:

  • Optimizer and query budget: addressed by the matched comparison and the random-basis ablation in Table 10, plus ZO baselines on the same INT4 backbone.
  • Seeds and claims: the difference between Table 1 (single seed, following ZO-Muon) and the four-seed results is now explained. The average column has been added (OPT-13B: 75.87 vs. 75.90 for ZO-Muon). The contributions and conclusion now say "competitive," which matches the evidence.
  • Rank-one behaviour: X v_1 varies across tokens (Table 16), r = 1 beats ZO bias-only tuning over four seeds (Table 17), and v_1 stays aligned after masking the highest-norm channels (Table 18).

My remaining reservation is that first-order LoRA on all linear layers still outperforms ZO-Act on most tasks (e.g., 88.5 on RTE and 92.9 on CB in Table 8). The earlier "ZO-Act beats first-order LoRA" comparison in Table 1 applied LoRA only to the query and value projections. It would help to state this clearly in the main text so readers can place ZO-Act against first-order fine-tuning.

Questions for authors

Thank you for the thorough response. My main questions have been answered, and I have a few small suggestions for the final version:

  1. Please state in the main text that the first-order LoRA row in Table 1 adapts only q/v projections, and point to the all-linear-layer comparison in Table 8.
  2. Please keep the four-seed results (mean ± std) close to Table 1, so readers do not compare single-seed and multi-seed numbers by mistake.
  3. A short note on why AGZO + Adam is not included (the full m × n moment states, as explained in the rebuttal) would help readers interpret Table 10.
3 comments
AuthorsOpenReview ↗

Response to Reviewer GGmS for ZO-Act

We thank the reviewer for the detailed and constructive feedback. We have added matched-optimizer comparisons, multi-seed results, rank-one diagnostics, and additional INT4 baselines to directly address the concerns raised. We respond to each point below.

W1(a). Baseline sources and learning-rate tuning.

Following common practice, most baseline results in Tables 1 and 12 are taken from previously published work (ZO-Muon and LOZO+, respectively). The AGZO and LOZO+ baselines in Table 1 were run by us with separately tuned learning rates. For our other experiments, including multi-seed evaluations, GSM8K, and commonsense reasoning, we also tuned the learning rates separately for each method and task.

W1(b) / Q2. Matched optimizer and query budget.

We added a matched comparison in Table 10, where all methods use r=1r=1, Adam, q=8q=8, and the same forward budget on Llama-3-8B.

MethodRTECBBoolQ
MeZO-LoRA with Adam67.971.478.6
ZO-LoRA-FA with random frozen basis and Adam52.057.165.4
ZO-Act with Adam87.082.185.1

ZO-LoRA-FA has the same parameterization as ZO-Act and differs only in the choice of basis, so the gap isolates the effect of the activation-informed subspace. AGZO recomputes its basis at every step and updates the weights directly, so applying Adam to AGZO would require moment states over the full m×nm\times n weight space of every layer, which removes the memory advantage of ZO fine-tuning. Enabling Adam at low memory cost is a direct benefit of the fixed coefficient space of ZO-Act, as discussed in Appendix A.11.

The first-order LoRA in Table 1 is from ZO-Muon and adapts only the query and value projections. With LoRA on all linear layers in Table 8, first-order LoRA reaches 88.5 on RTE and 92.9 on CB and outperforms ZO-Act on most tasks.

For INT4, we added ZO baselines on the same INT4 Llama-3-8B backbone in Table 4.

Method on INT4 Llama-3-8BRTECBBoolQ
QZO69.771.478.2
MeZO-LoRA61.067.973.6
ZO-LoRA-FA49.860.766.4
ZO-Act83.485.783.8

W2 / Q1. Seed of Table 1, multi-seed results, and claims.

Table 1 uses data-sampling seed 0 and PyTorch random seed 42, consistent with the single-seed setting of ZO-Muon. In contrast, Table 11 reports results averaged over four seeds {0,13,42,100}\{0,13,42,100\}, with both the data-sampling seed and PyTorch random seed varied together across runs. These differences in random seeds and data sampling explain the discrepancies between Table 1 and Table 11. Since most published baselines report only single-seed results, we retain the single-seed comparison in Table 1 for consistency and report separate four-seed evaluations in Table 11.

We added an average column to Table 1. On OPT-13B, ZO-Act achieves an average score of 75.87, nearly identical to ZO-Muon's 75.90, while outperforming it on four of the six tasks. We have also revised the contributions and conclusion to reflect the overall competitive performance of ZO-Act.

Q3. Rank-one behavior.

Xv1Xv_1 varies substantially across tokens, as shown in Table 16 on 64 held-out RTE examples from layer 16. A rank-one update is therefore input-dependent and not equivalent to a bias.

ProjectionMeanStdMinMax
q_proj11.7482.520−1.66318.170
o_proj1.3960.713−0.6443.989
gate_proj6.0561.6140.10510.675
down_proj1.5350.400−0.0922.635

Rank-one ZO-Act outperforms ZO bias-only tuning over four seeds, as shown in Table 17.

MethodRTECBBoolQ
ZO bias-only78.2 ± 3.371.0 ± 4.782.6 ± 2.2
ZO-Act with r = 183.3 ± 3.380.4 ± 3.385.1 ± 0.8

We also mask the highest-norm activation channels, recompute the leading direction, and measure its absolute cosine similarity to the original v1v_1, as shown in Table 18. The leading direction remains well aligned after masking, so v1v_1 is not solely determined by the highest-norm channels.

Layer inputMask top 1 channelMask top 5 channels
Attention input0.9580.857
Output projection0.9520.875
MLP input0.9640.890
MLP down projection0.8550.805

Q4. Related work. Section 2 now discusses EVA, CorDA, LoRA-FA, SensZOQ, and QZO.

Reviewer_GGmSOpenReview ↗

Revised comment after rebuttal

Revised comment after rebuttal

I thank the authors for the thorough response.

Main concern addressed. The matched comparison in Table 10 (r=1, Adam, q=8, same forward budget) directly addresses my main concern:

  • ZO-Act still clearly outperforms MeZO-LoRA.
  • The random-frozen-basis ZO-LoRA-FA ablation isolates the benefit of the activation-informed subspace from the optimizer and query budget.

Other points. The seed discrepancy between Table 1 and Table 11 is now explained. The revised claims ("competitive" rather than state-of-the-art, with OPT-13B average 75.87 vs. 75.90 for ZO-Muon) better match the evidence.

My remaining reservation is that first-order LoRA on all linear layers still outperforms ZO-Act on most tasks.

Overall, my main concerns are addressed, and I raise my score from 2 to 3.

Thank you so much for taking the time to read our response and update your assessment! We are glad that our response addressed your main concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.