acceptodds
Under review as a conference paper at ICLR 2027

AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning

Abstract

Zeroth-order (ZO) optimization offers a memory-efficient alternative for LLM fine-tuning by estimating updates only from forward evaluations of perturbed parameters, without backpropagation or activation storage. However, in billion-parameter LLMs, isotropic perturbations often waste many forward evaluations on weakly informative directions. To make these evaluations more informative, existing ZO methods restrict perturbations to low-dimensional subspaces. Yet the quality of these subspaces is critical: overly compressed or poorly maintained spaces can miss useful update directions. To obtain a high-quality subspace for ZO updates, this paper proposes AIM-ZO, a ZO fine-tuning method based on Activation-Informed Subspace Maintenance. AIM-ZO uses forward activations as local directional information and continuously integrates them into a broad, evolving subspace over training. To access broader gradient-relevant structure while keeping individual perturbations low-dimensional, AIM-ZO activates only a smaller set of shared and sampled directions, decoupling the maintained width from the active width. We evaluate AIM-ZO across 5 LLMs and 11 downstream tasks under matched forward-evaluation budgets; its six-task average exceeds the strongest fully evaluated ZO baseline by 1.26 percentage points on OPT-2.7B and MeZO by 2.85 percentage points on OPT-30B. Our code is available at https://anonymous.4open.science/r/AIM-ZO-0155/

Primary area:
foundation or frontier models, including LLMs
Keywords:
Zeroth-Order Optimization, Large Language Models, Memory-Efficient Fine-Tuning, Online Subspace Maintenance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rQu6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AIM-ZO, a zeroth-order (ZO) fine-tuning method for LLMs that improves the usefulness of forward evaluations without using backpropagation. Unlike MeZO and activation-based AGZO, AIM-ZO accumulates forward-activation information during training to maintain a broad evolving subspace of width KK, while each perturbation uses a smaller active subspace of width k<Kk<K. On five LLMs (0.6B–30B) across 11 tasks, AIM-ZO achieves higher average performance under matched forward-evaluation budgets.

Critical strengths
  • Using activation to construct subspaces has been explored in prior work. This paper instead maintains a decomposition of subspace into shared and varying parts: one component is fully included (always present), while the other component can be randomly sampled. This idea seems plausible.

  • Under the full fine-tuning setting, the paper validates the effectiveness of the proposed algorithm.

Critical weaknesses
  1. Subspace construction is unclear. I don’t quite understand how the proposed structured subspace is constructed. In particular, the role of Oja’s rule is not explained clearly to me—how it works, why it’s needed, and what its practical effect is.

  2. Hyperparameter setting (ηq\eta_q) is missing or unclear. It would help to explain how ηq\eta_q should be set, and what factors determine its value.

  3. Theory is too complex for intuition. The theoretical proof is very complicated and difficult to follow, and it doesn’t really help readers build intuition about the underlying ideas behind the algorithm.

  4. Experiments don’t cover LoRA, and AGZO results are concerning. All experiments are conducted only under full fine-tuning, with no results under LoRA, which makes it harder to assess the method in parameter-efficient scenarios. Also, the AGZO results are unexpectedly poor—compared with MeZO, the performance drops much more than I would expect.

Questions for authors

Please see the weaknesses listed above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer rQu6 (Part 1/2)

Thank you for the comments. We address the concerns below.

Weakness 1. Oja: Construction, Motivation, and Effect

Subspace construction is unclear. I don’t quite understand how the proposed structured subspace is constructed. In particular, the role of Oja’s rule is not explained clearly to me—how it works, why it’s needed, and what its practical effect is.

Oja continuously tracks the activation principal subspace as training evolves. For each linear layer, St=Ht⊤Ht/nt,Qt=qf⁡((I+ηqSt)Qt−1).S_t=H_t^\top H_t/n_t,\qquad Q_t=\operatorname{qf}((I+\eta_qS_t)Q_{t-1}). The previous basis carries historical information, current activations adjust its directions, and QR restores orthonormality. AGZO reconstructs a space from the current batch; AIM-ZO recursively accumulates information across steps, maintaining a wider space from which smaller active spaces are selected. Section 3.2 gives the update and algorithm.

Appendix D.1 compares Oja with AGZO's current-batch PI-5 construction: Oja achieves higher mean gradient capture in five of six examined layers on independent evaluation batches. A five-seed OPT-2.7B/RTE ablation also supports continual updating:

MaintenanceFinal dev. accuracy (%)
Fixed space68.40 ± 2.04
Oja every 10 steps68.92 ± 3.38
Oja every step70.20 ± 1.95

Weakness 2. Setting the Oja Step Size

Hyperparameter setting () is missing or unclear. It would help to explain how should be set, and what factors determine its value.

Appendix C specifies ηq=0.3\eta_q=0.3 for the main experiments. The tracking theorem in Section 4.1 gives Eεtrk,t2≲e−ηqγt/2εburn2+ωˉ2ηq2γ2+KηqΣ2γ+failure terms.\mathbb E\varepsilon_{\mathrm{trk},t}^2\lesssim e^{-\eta_q\gamma t/2}\varepsilon_{\mathrm{burn}}^2+\frac{\bar\omega^2}{\eta_q^2\gamma^2}+\frac{K\eta_q\Sigma^2}{\gamma}+\text{failure terms}. Within the theorem's admissible range, increasing the step size accelerates initialization decay and reduces tracking lag, but increases stochastic error. Its choice balances adaptation against stability and depends on target drift, observation second moments, eigengap, and maintained width. Appendix A states the sufficient conditions. Appendix D.1 compares 0.03, 0.1, 0.3, and 0.6 through frozen-checkpoint replay, reporting projector error and gradient capture.


Weakness 3. Theoretical Intuition

Theory is too complex for intuition. The theoretical proof is very complicated and difficult to follow, and it doesn’t really help readers build intuition about the underlying ideas behind the algorithm.

The assessment that a theoretical analysis is insufficiently intuitive is partly subjective. We clarify the intuition and proof structure below and welcome feedback on the specific steps that would benefit from further explanation.

  1. Section 4 follows the two core designs: maintaining gradient-relevant information across training and separating maintained from active width.

  2. Section 4.1 rewrites QR–Oja as a projector update, obtains contraction from the eigengap, and controls drift and stochastic observations. The capture proposition connects the maintained space to gradient energy through structural, activation-alignment, and tracking residuals.

  3. Section 4.2 computes expected retained energy from direction-inclusion probabilities, then uses a Gaussian cosine identity to obtain an alignment lower bound. With sufficient energy retention, it exceeds expected alignment under full activation. Section 4.3 examines RLOO in high-capture spaces through linear controls, real-loss diagnostics, and online ablations.

So, could the reviewer identify the specific propositions, assumptions, or derivation steps that are difficult to follow? We can provide detailed derivations for those steps.

AuthorsOpenReview ↗

Response to Reviewer rQu6 (Part 2/2)

Weakness 4. LoRA Evaluation and AGZO Results

Experiments don’t cover LoRA, and AGZO results are concerning. All experiments are conducted only under full fine-tuning, with no results under LoRA, which makes it harder to assess the method in parameter-efficient scenarios. Also, the AGZO results are unexpectedly poor—compared with MeZO, the performance drops much more than I would expect.

AIM-ZO currently perturbs original weight matrices and has not been adapted to LoRA's low-rank parameterization. We therefore compare MeZO+LoRA with AIM-ZO tuning the layers targeted by LoRA. Corresponding results are:

Task / metricMeZO+LoRAAIM-ZO, selected layers
RTE / accuracy77.8678.10
WiC / accuracy56.4357.11
CB / accuracy84.5284.52
MultiRC / F1a73.3073.52
DROP / F133.6935.37

The advantage narrows in this setting. This comparison directly applies AIM-ZO to the selected layers, without adaptation to LoRA's low-rank parameterization.

We performed implementation-consistency checks for AGZO, LoZO, and CurvZO against their authors' implementations using matched objectives, batches, and seeds. Taking AGZO as an example:

CheckScopeResult
Real-model paired run20 steps: losses, finite differences, batches, sampled parameters and displacementsExact match
Full-model parameter hashesAll parameters at initialization and completionIdentical SHA256 hashes
Direct author-function CPU testsFP32/FP16/BF16, 3 steps each; all parameters at every stepAll pass torch.equal

The original AGZO Qwen experiments use FP32; our comparisons use BF16. AGZO remains competitive on other datasets, suggesting that the SQuAD anomaly may reflect instability under BF16. In these runs, about 64% of steps give identical positive- and negative-perturbation losses, eliminating the finite-difference signal. This is consistent with precision-related loss of the perturbation signal. The implementation checks and training diagnostics above are documented.

We sincerely hope our responses have addressed your concerns, and that you could reconsider your rating toward a positive score. If you have any further questions, please feel free to follow up, and we are glad to address them.

Dear Reviewer rQu6,

We would appreciate your feedback on our earlier responses. We have addressed your concerns by:

  1. Explaining how Oja’s rule accumulates activation information across training steps to maintain an evolving perturbation subspace.
  2. Clarifying the setting of ηq\eta_q and its role in balancing adaptation speed and stochastic tracking error.
  3. Explaining the theoretical intuition and how the analysis supports the two core designs of AIM-ZO.
  4. Providing a selected-layer comparison between MeZO+LoRA and AIM-ZO applied to the corresponding original Q/V weights.
  5. Presenting implementation-consistency checks against the authors’ reference implementations, with detailed numerical evidence for AGZO.
  6. Explaining the anomalous AGZO results using the observed loss of finite-difference signals under BF16.

We sincerely hope that these clarifications and additional results address your concerns and may encourage a more favorable reconsideration of our paper. Please let us know if any questions remain; we would be happy to discuss them further.

Best regards, Submission 1869 Authors

Reviewer_VQsYOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

AIM-ZO maintains an activation-informed subspace across training steps using Oja's algorithm and selects smaller active subspaces for perturbation-based gradient estimation. The authors provide theoretical analyses of subspace tracking, gradient capture, and directional alignment. Experiments across multiple LLMs and downstream tasks demonstrate improvements over several ZO baselines.

Critical strengths
  1. Improving the efficiency of zeroth-order LLM fine-tuning is an important problem, particularly for memory-constrained settings.
  2. The paper provides several theoretical results concerning subspace tracking, gradient capture, and directional alignment.
  3. The experiments cover multiple model scales and downstream tasks. The authors also provide extensive ablation studies and supplementary analyses, including gradient capture, memory consumption, and runtime measurements.
Critical weaknesses
  1. My primary concern is the limited novelty of the proposed method. AIM-ZO appears to combine several existing techniques: activation-guided perturbation subspaces from AGZO, online subspace maintenance using the classical Oja algorithm, and subspace sampling with a population-centered one-sided estimator. The paper does not clearly articulate what fundamentally new insight or algorithmic principle emerges from this combination, beyond the potential benefits of the individual techniques.
  2. Although the paper identifies limitations of existing subspace-based ZO methods, the motivation for the particular design of AIM-ZO is not sufficiently developed. For example, it is unclear why maintaining a wider subspace and randomly sampling a subset of its directions should be preferable to simply maintaining and using a compact, high-quality subspace. The paper also does not convincingly establish that the computational cost and stochastic variation of AGZO's current-batch subspace construction are major bottlenecks that justify the proposed online maintenance mechanism. More generally, the motivations for the different components are presented separately, without a clear explanation of why their combination is necessary.
  3. The paper is unnecessarily complicated in its presentation. The explanations of the method and theoretical results are lengthy, and the central intuition is sometimes obscured by extensive notation and technical details. Several important experimental comparisons and implementation details are scattered across the appendices, making it difficult to identify the evidence supporting the main claims.
  4. An important limitation is that the theoretical results do not fully characterize the actual optimization procedure. In particular, Proposition 2 considers perturbations constructed using full Gaussian coefficient matrices, whereas AIM-ZO uses normalized rank-one perturbations in practice. Moreover, the analysis primarily concerns individual gradient estimates, while the implemented algorithm combines multiple perturbations with different active subspaces through a population-centered RLOO estimator. Consequently, the theoretical alignment results do not directly establish the claimed advantages of the implemented updates. The paper also lacks an end-to-end convergence analysis connecting the proposed components to optimization performance.
  5. Several experimental observations weaken the overall conclusions:
    • The improvements over the strongest baselines are relatively modest on some models, particularly Qwen3-0.6B and Qwen3-8B.
    • The ablation in Table 19 shows that using all 64 shared directions without random tail sampling achieves higher mean accuracy than the shared-48/sampled-16 configuration under that experimental setting. This raises doubts about the necessity of one of the central design choices.
    • The runtime results in Table 25 indicate that AIM-ZO can be substantially slower than MeZO under the same forward-evaluation budget. For example, the training time is approximately doubled on OPT-2.7B. The practical efficiency gains are therefore less convincing than the query-budget comparisons alone suggest.
    • The large-scale OPT-30B experiments compare AIM-ZO only with MeZO, without including other competitive subspace-based ZO methods.
Questions for authors
  1. Some baseline results appear substantially weaker than expected. For example, on OPT-2.7B SQuAD, AGZO achieves only 29.82 F1 and LoZO achieves 43.52 F1, compared with 80.99 F1 for MeZO. Similarly, CurvZO and LoZO perform close to random guessing on OPT-13B SST-2. Could the authors explain these unusually poor results? Were the baseline implementations and hyperparameters validated against their originally reported performance?
  2. In Section 3.3, the first h columns of the maintained basis are selected as shared directions. However, the Oja update followed by QR orthogonalization does not explicitly sort the basis vectors according to their importance. How is the ordering of these directions determined and maintained during training? Is there any mechanism ensuring that the first h directions capture more gradient-relevant information than the remaining directions?
  3. In Appendix D.1, Table 7 shows that the current-batch exact-SVD subspaces consistently achieve higher gradient capture than the maintained Oja subspaces, whereas Table 8 reports advantages for Oja over the current-batch PI-5 construction in most tested layers. Could the authors clarify how these observations should be interpreted together, particularly regarding the benefit of online maintenance over current-batch subspace construction?
  4. Table 4 reports different hyperparameter search ranges for the baseline methods, while Appendix C describes some differences in checkpoint selection and evaluation schedules. Could the authors clarify whether comparable hyperparameter tuning budgets and checkpoint-selection procedures were used across methods?
7 comments
AuthorsOpenReview ↗

Response to Reviewer VQsY (Part 1/4)

Thank you for the comments. We address each concern below.

Weakness 1. Contributions

My primary concern is the limited novelty of the proposed method. AIM-ZO appears to combine several existing techniques: activation-guided perturbation subspaces from AGZO, online subspace maintenance using the classical Oja algorithm, and subspace sampling with a population-centered one-sided estimator. The paper does not clearly articulate what fundamentally new insight or algorithmic principle emerges from this combination, beyond the potential benefits of the individual techniques.

AIM-ZO starts from the research question of obtaining a high-quality subspace for low-dimensional ZO updates. Our two contributions address this question: accumulating forward activation information across training to maintain a broad, evolving perturbation subspace; and decoupling maintained width K from active width k to preserve directional coverage while controlling estimation dimension and error.

The two contributions work together: persistent maintenance retains information beyond the current batch, and partial activation makes that broader information usable without increasing each estimate's dimension. We evaluate the resulting ZO method against AGZO and other ZO baselines, while maintenance, active selection, and estimator ablations isolate the benefits of the individual choices.


Weakness 2. Design Motivation

Although the paper identifies limitations of existing subspace-based ZO methods, the motivation for the particular design of AIM-ZO is not sufficiently developed. For example, it is unclear why maintaining a wider subspace and randomly sampling a subset of its directions should be preferable to simply maintaining and using a compact, high-quality subspace. The paper also does not convincingly establish that the computational cost and stochastic variation of AGZO's current-batch subspace construction are major bottlenecks that justify the proposed online maintenance mechanism. More generally, the motivations for the different components are presented separately, without a clear explanation of why their combination is necessary.

Current-batch reconstruction omits history, while maintaining only a compact space discards other candidate directions. Oja recursively reuses the previous basis to integrate new observations. A broad maintained space preserves candidates; smaller active spaces control estimation difficulty. Section 4.1 gives tracking/capture guarantees, and Section 4.2 gives conditions for partial activation to improve alignment.

The experiments examine the two widths separately. On OPT-2.7B/RTE, the maintained-width scan fixes k=64 and h=48, using BF16, three seeds, 40,000 forward evaluations, and Oja every ten steps:

Maintained width KAccuracy (%)
6465.70 ± 1.25
12865.46 ± 3.62
25667.15 ± 0.72

The K=256 result shows that information retained beyond a compact width-64 space can improve training at the same active width. Conversely, at fixed K=128, the new three-seed comparison gives 65.46% with k=64 versus 63.66% with k=128. Offline active-width diagnostics also show increasing capture but decreasing single-direction alignment as more directions are activated.

Maintenance-frequency ablations give 68.40±2.04 for a fixed space versus 70.20±1.95 for per-step Oja updates over five seeds. Appendix D.1 reports basis-maintenance costs of 0.124 s for Oja versus 0.229 s for current-batch PI construction. On OPT-2.7B/SQuAD in BF16, the available final F1 results also show lower across-seed variability for AIM-ZO:

MethodSeedsF1 (mean ± sample SD)
AGZO329.82 ± 5.54
AIM-ZO581.19 ± 0.49
AuthorsOpenReview ↗

Response to Reviewer VQsY (Part 2/4)

Weakness 3. Presentation and Theoretical Intuition

The paper is unnecessarily complicated in its presentation. The explanations of the method and theoretical results are lengthy, and the central intuition is sometimes obscured by extensive notation and technical details. Several important experimental comparisons and implementation details are scattered across the appendices, making it difficult to identify the evidence supporting the main claims.

Section 3 provides concrete solutions for the two contributions: recursively maintaining a broad activation-informed space and selecting smaller shared/sampled active spaces. Sections 4.1 and 4.2 then provide their respective theoretical support. Section 4.1 bounds tracking error and connects it to retained gradient energy under the structural/alignment conditions. Section 4.2 quantifies active-space energy retention and gives a sufficient condition for improved directional alignment.

Other components, including low-rank perturbations, normalization, and RLOO, make the method practical. Their implementation benefits are examined through component ablations. Section 4.3 further examines estimator choice through linear controls, real-loss diagnostics, and online comparisons. This separates the two core contributions from the supporting implementation choices.


Weakness 4. Analysis versus Implementation

An important limitation is that the theoretical results do not fully characterize the actual optimization procedure. In particular, Proposition 2 considers perturbations constructed using full Gaussian coefficient matrices, whereas AIM-ZO uses normalized rank-one perturbations in practice. Moreover, the analysis primarily concerns individual gradient estimates, while the implemented algorithm combines multiple perturbations with different active subspaces through a population-centered RLOO estimator. Consequently, the theoretical alignment results do not directly establish the claimed advantages of the implemented updates. The paper also lacks an end-to-end convergence analysis connecting the proposed components to optimization performance.

We use Gaussian probes in the directional-alignment analysis because rotational symmetry and tractable moment/cosine identities make the relationship between retained gradient energy and estimation dimension explicit.

In practice, normalized rank-one probes reduce perturbation construction and application costs. The three-seed BF16 OPT-2.7B/RTE coefficient-rank ablation uses 40,000 training forward evaluations: rank-one and full Gaussian probes achieve 65.46±3.62 and 65.58±2.21 accuracy, with respective mean per-run median non-evaluation step times of 10.35 s and 12.24 s. Section 4.3 and the estimator ablations separately evaluate population-centered RLOO with actual losses and online training.

AuthorsOpenReview ↗

Response to Reviewer VQsY (Part 3/4)

Question 2. Ordering of Shared Directions

In Section 3.3, the first h columns of the maintained basis are selected as shared directions. However, the Oja update followed by QR orthogonalization does not explicitly sort the basis vectors according to their importance. How is the ordering of these directions determined and maintained during training? Is there any mechanism ensuring that the first h directions capture more gradient-relevant information than the remaining directions?

No separate importance sorting is needed. Let At=I+ηqStA_t=I+\eta_qS_t. Unpivoted QR–Oja gives AtQt−1=QtRtA_tQ_{t-1}=Q_tR_t, hence AtQt−1[:,1:h]=Qt[:,1:h]Rt[1:h,1:h].A_tQ_{t-1}[:,1:h]=Q_t[:,1:h]R_t[1:h,1:h]. The triangular block is invertible, so the first h columns evolve as a width-h subspace iteration automatically; no separate prefix update or sorting is required.


Question 3. Exact SVD versus PI-5

In Appendix D.1, Table 7 shows that the current-batch exact-SVD subspaces consistently achieve higher gradient capture than the maintained Oja subspaces, whereas Table 8 reports advantages for Oja over the current-batch PI-5 construction in most tested layers. Could the authors clarify how these observations should be interpreted together, particularly regarding the benefit of online maintenance over current-batch subspace construction?

Both comparisons and their protocols are already reported in Appendix D.1. Table 7 uses finite-training-set mean gradients over 16 steps, with exact current-batch SVD as a diagnostic reference. Table 8 compares Oja with AGZO's practical PI-5 construction using gradients from independent development batches. The results are:

Model / layerTable 7: Oja / exact SVD capture (%)Table 8: Oja minus PI-5, K=128 (pp)Table 8: Oja minus PI-5, k=64 (pp)
Qwen3-4B / early52.2 / 69.5-11.06-10.31
Qwen3-4B / middle86.2 / 89.0+4.71+4.06
Qwen3-4B / late87.8 / 92.6+1.02+2.38
OPT-2.7B / early63.5 / 71.7+8.95+6.29
OPT-2.7B / middle95.1 / 96.4+1.54+2.19
OPT-2.7B / late65.7 / 77.0+4.82+4.68

Exact SVD has higher capture in Table 7. Against the PI-5 construction used in training, Oja has higher mean independent-batch capture in five of six layers, for both maintained and active spaces. Table 8 uses 256 paired batch/checkpoint comparisons per layer; these are diagnostic pairs, not independent training runs. Thus, the practical maintenance comparison is against PI-5, while exact SVD provides the diagnostic reference.


Weakness 5. Shared versus Sampled Directions

The ablation in Table 19 shows that using all 64 shared directions without random tail sampling achieves higher mean accuracy than the shared-48/sampled-16 configuration under that experimental setting. This raises doubts about the necessity of one of the central design choices.

Additional runs at fixed K=128, k=64 give:

Active directionsAccuracy (%)
48 shared + 16 sampled67.51 ± 3.18
64 shared67.00 ± 2.16

The shared/sampled configuration achieves higher mean accuracy, supporting exploration of the remaining maintained directions.

Reviewer_7seQOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AIM-ZO for zeroth-order LLM fine-tuning. The method uses forward activations to maintain a perturbation subspace and selects a smaller set of directions for each perturbation, separating the maintained width from the active width. This design aims to retain useful gradient information while making direction estimation easier with a limited query budget. The authors provide theoretical analysis and evaluate the method on several models and downstream tasks.

Critical strengths
  1. The paper identifies a useful trade-off in subspace-based ZO optimization: a narrow subspace can miss important gradient components, while a wide one makes estimation harder with few queries. I find the distinction between maintained and active width well motivated, and the proposed design follows naturally from this problem.

  2. The experiments cover several model sizes and a range of downstream tasks. In particular, the OPT-2.7B and OPT-13B results include a broad set of ZO baselines under matched forward-evaluation budgets, giving useful evidence of the method's effectiveness.

  3. The gradient-capture and directional-alignment analyses help explain the design choices. The ablations on subspace maintenance, resampling, and estimator choices also make it easier to understand how the different parts of AIM-ZO affect performance.

Critical weaknesses
  1. It is still unclear how broadly the activation–gradient alignment assumption holds. The diagnostics in Appendix D.1 provide supporting evidence on RTE, but examine selected q_proj layers and relatively short training windows. Since the method relies on activations to identify useful search directions, evidence from other tasks, parameter modules, and more widely separated training stages would help establish the scope of this assumption.

  2. The larger-model comparisons are less complete than those on OPT-2.7B and OPT-13B. OPT-30B is compared mainly with MeZO, while the Qwen3-8B results include only MeZO, AGZO, and AIM-ZO among the trained methods. The results are encouraging, but leave open whether AIM-ZO remains competitive with the stronger baselines used elsewhere in the paper.

  3. The benefit of partial activation has not been directly established in downstream training at a fixed maintained width. The paper includes theoretical analysis and offline active-width diagnostics, but Tables 17 and 18 change the maintained width while keeping the active width fixed. These experiments do not answer whether activating only part of a given maintained space leads to better final performance than activating all of it.

Questions for authors

Could the authors clarify when the activation–gradient alignment assumption is expected to hold, and extend the diagnostics to a few additional tasks or modules, such as FFN projections? For the larger-model results, including one of the stronger baselines already evaluated on the smaller OPT models would make the comparison more informative. I would also like to see a direct test of partial versus full activation with the maintained width fixed, for example k = 64 versus k = 128 at K = 128. Matching the forward-evaluation budget and other training settings would help determine whether the estimated-direction advantage translates into a downstream performance gain.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7seQ

Thank you for the comments. We address the concerns below.

Weakness 1. Scope of Activation–Gradient Alignment

It is still unclear how broadly the activation–gradient alignment assumption holds. The diagnostics in Appendix D.1 provide supporting evidence on RTE, but examine selected q_proj layers and relatively short training windows. Since the method relies on activations to identify useful search directions, evidence from other tasks, parameter modules, and more widely separated training stages would help establish the scope of this assumption.

The assumption concerns whether the leading activation subspace retains sufficient gradient energy. A linear layer's weight gradient depends on both input activations and backpropagated signals. When substantial gradient energy lies along high-activation-energy directions, activations identify a useful search space. Our capture analysis characterizes the retained gradient information under this condition.

We added same-task diagnostics along both RTE- and SST-2-trained OPT-2.7B trajectories (seed42), at steps 500, 1500, and 2500. RTE covers all 32 layers and six projections (q/k/v/out_proj, fc1/fc2); SST-2 covers layers 0, 15, and 31 and q_proj, v_proj, fc1, and fc2.

We measure gradient-energy coverage, ∥GQ∥F2/∥G∥F2\|GQ\|_F^2/\|G\|_F^2. At each checkpoint, 32 samples construct a rank-64 uncentered activation SVD basis and 32 disjoint samples estimate the gradient. Sample lists are fixed across stages within each task. Equal-weight means across examined modules are:

TaskStep500 (%)Step1500 (%)Step2500 (%)
RTE83.5086.8986.79
SST-285.7587.5587.25

All 576 RTE and 36 SST-2 combinations exceed the mean of five same-rank random subspaces. Coverage varies by layer and module: SST-2 layer-0 q_proj decreases from 59.78% to 56.32%, while middle/late-layer v_proj stays near 100%. The evidence thus extends across tasks, attention/FFN modules, and separated training stages. These fresh SVD bases diagnose the activation–gradient relationship; Appendix D.1 separately evaluates the historical Oja state.


Weakness 2. Stronger Baselines on Larger Models

The larger-model comparisons are less complete than those on OPT-2.7B and OPT-13B. OPT-30B is compared mainly with MeZO, while the Qwen3-8B results include only MeZO, AGZO, and AIM-ZO among the trained methods. The results are encouraging, but leave open whether AIM-ZO remains competitive with the stronger baselines used elsewhere in the paper.

Additional stronger-baseline comparisons on larger models are underway. Full training runs take considerable time; we will update the comparisons when complete results are available.


Weakness 3. Partial versus Full Activation at Fixed Maintained Width

The benefit of partial activation has not been directly established in downstream training at a fixed maintained width. The paper includes theoretical analysis and offline active-width diagnostics, but Tables 17 and 18 change the maintained width while keeping the active width fixed. These experiments do not answer whether activating only part of a given maintained space leads to better final performance than activating all of it.

I would also like to see a direct test of partial versus full activation with the maintained width fixed, for example k = 64 versus k = 128 at K = 128. Matching the forward-evaluation budget and other training settings would help determine whether the estimated-direction advantage translates into a downstream performance gain.

We compared k=64 (48 shared + 16 sampled directions) with full activation, k=128, at fixed K=128 on OPT-2.7B/RTE, following the paper's experimental protocol across three seeds.

The seeds are 42, 142, and 242. Each new full-activation run completed 2,500 steps and 40,000 training objective calls. We select the checkpoint with the highest development accuracy, as for the historical partial-activation baseline.

Seedk=64 (48+16)k=128
4266.79%65.70%
14268.23%62.82%
24261.37%62.45%
Mean65.46%63.66%

These results support partial activation at fixed maintained width: under this query budget, including every maintained direction does not improve final performance.

The observation is consistent with the analyzed trade-off: a wider maintained space retains gradient information, while a smaller active space helps estimate directions with limited queries. It supports retaining 48+16 as the default.


We sincerely hope our responses have addressed your concerns, and that you could reconsider your rating toward a positive score. If you have any further questions, please feel free to follow up, and we are glad to address them.

Dear Reviewer 7seQ,

We would appreciate your feedback on our earlier responses. In particular, we have:

  1. Extended the activation–gradient diagnostics to both RTE- and SST-2-trained trajectories, covering additional attention and FFN modules at three training stages.
  2. Added the requested partial-versus-full activation comparison at fixed K=128K=128, obtaining mean accuracy of 65.46% with k=64k=64 versus 63.66% with k=128k=128 across three seeds. We sincerely hope that these additional results address your concerns regarding activation–gradient alignment and partial activation, and may encourage a more favorable reconsideration of our paper. Please let us know if any questions remain; we would be happy to discuss them further.

Best regards, Submission 1869 Authors

Dear Reviewer 7seQ,

We would appreciate your feedback on our earlier responses. In particular, we have:

  1. Extended the activation–gradient diagnostics to both RTE- and SST-2-trained trajectories, covering additional attention and FFN modules at three training stages.
  2. Added the requested partial-versus-full activation comparison at fixed K=128K=128, obtaining mean accuracy of 65.46% with k=64k=64 versus 63.66% with k=128k=128 across three seeds. We sincerely hope that these additional results address your concerns regarding activation–gradient alignment and partial activation, and may encourage a more favorable reconsideration of our paper. Please let us know if any questions remain; we would be happy to discuss them further.

Best regards, Submission 1869 Authors

Dear Reviewer 7seQ,

We would appreciate your feedback on our earlier responses. In particular, we have:

  1. Extended the activation–gradient diagnostics to both RTE- and SST-2-trained trajectories, covering additional attention and FFN modules at three training stages.
  2. Added the requested partial-versus-full activation comparison at fixed K=128K=128, obtaining mean accuracy of 65.46% with k=64k=64 versus 63.66% with k=128k=128 across three seeds. We sincerely hope that these additional results address your concerns regarding activation–gradient alignment and partial activation, and may encourage a more favorable reconsideration of our paper. Please let us know if any questions remain; we would be happy to discuss them further.

Best regards, Submission 1869 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.