acceptodds
Under review as a conference paper at ICLR 2027

AXIOM: Activation-Conditioned Momentum as Associative Memory

Abstract

Classical momentum-based optimizers, such as Stochastic Gradient Descent (SGD) with momentum and Adam accumulate gradient statistics isotropically, treating every direction in parameter space identically and remaining blind to the geometry of the activations that produced each gradient. Motivated by the Nested Learning view of optimization as a system of associative memories, we in this study, reformulate momentum as the solution of a regularized least squares objective that maps an input activation to its local surprise signal, rather than an exponential moving average, yielding a closed form optimizer we call AXIOM (Activation-guided eXpressive Iterative Optimization with associative Memory). We derive the resulting update independently through a Sherman–Morrison expansion and through Fenchel–Rockafellar duality, showing that both routes converge on the same rank-one, activation-conditioned correction. We characterize AXIOM as a discrete-time linear time-varying system, prove global uniform exponential stability of its unforced dynamics, and establish an O(1/T) + O(σ 2 g ) nonconvex convergence rate for the forced stochastic system, matching classical firstorder guarantees. To test whether activation-aware momentum provides practical benefit under heterogeneous and non-stationary feature geometries, we introduce Wavelet2VecNet-a framework that couples pretrained Self-Supervised Learning (SSL)-based contextual embeddings (Wav2Vec2) with a two-dimensional wavelet packet transform decomposition for compact utterance-level speech representation learning, and evaluate it across five speech processing tasks namely, dysarthric speech severity-level assessment, infant cry pathology classification, environmental acoustic scene and sound classification, and deepfake speech source attribution. AXIOM matches or exceeds Adam and SGD on four of five tasks, with the largest gains (+7 to +14 accuracy points) on the most acoustically heterogeneous tasks, and no advantage on a near-saturated verification task – a pattern consistent with our theoretical account of when activation-conditioned momentum should help. We report ablations across wavelet bases and feature backbones, including a case where AXIOM’s advantage does not persist, and discuss the limitations of the present activation-only, linear-layer analysis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Wav2vec2.0, Wavelet, Associate memory, Speech

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HFqoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AXIOM, an activation-conditioned momentum update derived from a regularized least-squares associative-memory objective. It replaces the first moment of an Adam-style optimizer while retaining the second moment and adapting internal hyperparameters online. The paper analyzes the single-activation closed-form update, unforced stability, and a simplified stochastic optimization system, and evaluates a wavelet-based speech/audio framework on five tasks.

Critical strengths
  1. The single-activation closed-form derivation in Section 3.2 is concrete and implementable. For fixed positive alpha and eta, the uniform contraction of the unforced memory recursion follows directly from the operator spectrum. These provide a useful starting point for studying activation-conditioned momentum.

  2. The default configuration improves accuracy on four audio tasks, with particularly large reported gains on environmental sound. The paper also reports settings where the advantage does not persist, including alternative backbones or wavelets. These empirical signals warrant further investigation even though the current evidence does not overcome the soundness concerns below.

Critical weaknesses
  1. There are concrete errors and unresolved assumptions in the theoretical account. In Appendix O (pp. 56–57, Eqs. 243–249), bar(M_t)=rho M_t with rho<1, but the fixed-point argument effectively identifies bar(M_t) with M_t. For a repeated pair (x,u), the actual fixed point is M*=u x^T/(alpha+||x||^2), hence M*x=||x||^2 u/(alpha+||x||^2), generally not u. For alpha=eta=x=u=1, the update is M_(t+1)=0.2 M_t+0.4 and converges to 0.5, not 1. The coefficient of u in Eq. 247 is also not 1-lambda_parallel. This invalidates the claimed exact repeated-pattern memory, although it does not invalidate the closed-form update or unforced stability. More centrally, Appendix L takes activation-dependent A_t/B_t outside conditional expectations (Eqs. 186, 203), without establishing the needed measurability or conditional centering when activation and gradient use the same minibatch. Unbiased raw gradients alone do not imply unbiased filtered gradients. Lemma 10 also imposes an additional spectral condition omitted from the informal main statement; writing k=mu_B/C_B=mu_P, that condition requires rho<k/(1+k)<=1/2 for some admissible epsilon. This is a restriction of the current proof, not a claim that higher-rho runs necessarily diverge.

  2. The analyzed algorithm and empirical evidence do not yet establish the headline practical claim. Appendix L studies W_(t+1)=W_t-eta M_(t+1), without Algorithm 1's Adam denominator, while practical hyperparameters vary online. The single-sample rank-one update is not a complete definition for minibatches or unfolded convolutional patch matrices: a matrix least-squares objective instead involves (gamma I+2XX^T)^(-1), unless a specific approximation or sequential rule is used. The dataset identities, subject-level splits, UrbanSound8K fold protocol, tuning budgets and multi-seed variability are insufficiently specified. Finally, the claim that benefits scale with activation heterogeneity is not supported by a quantitative heterogeneity measure or controlled comparison. Seen-source accuracy near 99.9% does not establish saturation of unseen verification with EER around 13–16%. These issues collectively require more than a wording correction.

Questions for authors
  1. Please confirm or correct the repeated-pattern fixed point in Appendix O. Please explain whether the one-dimensional example above requires revising the exact-retrieval and memory interpretation.

  2. Please state the exact algorithm covered by Theorem 2. Specify the filtration and conditional-centering assumption for activation-dependent operators and the additional Lemma 10 spectral condition. State whether the experimental parameters satisfy these conditions.

  3. Please provide the actual minibatch and convolutional-patch update. Clarify any averaging or sequential approximation and its complexity. Also specify dataset identities and subject-level splits and the UrbanSound8K fold protocol and independent tuning budgets.

  4. Within the existing tasks please isolate activation conditioning from the retained Adam second moment and hypergradient adaptation. Report seed variability and any quantitative evidence connecting activation heterogeneity to gains.

My recommendation is rejection because the concrete mathematical error and central proof-to-algorithm gaps are not resolved by the current empirical evidence. These concerns do not imply that the reported measurements are fabricated or that activation-conditioned momentum is unpromising.

Reviewer_Liw6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AXIOM, an activation-conditioned replacement for the first-moment mechanism in an Adam-style optimizer. It formulates the momentum matrix as a regularized associative memory mapping layer inputs to downstream gradient signals. The single-sample objective yields a closed-form, error-driven rank-one correction, while the practical optimizer retains Adam's second-moment estimator and adapts its hyperparameters online. Experiments show that AXIOM improves over Adam and SGD on four tasks in the main setting, with the largest accuracy gains on the two environmental-sound tasks.

Critical strengths

The activation-conditioned formulation provides a concrete alternative to scalar-decay momentum. The regularized objective makes the proposed memory update explicit, and its closed-form solution corrects the discrepancy between the stored activation-to-gradient association and the current observation. This is an interesting way to incorporate input geometry into first-moment dynamics.

The reported improvements motivate further investigation of the method. AXIOM improves accuracy over Adam by 2.35, 7.48, 2.42, and 14.43 percentage points on four selected tasks. The backbone and wavelet comparisons broaden the evaluation and include settings where the advantage disappears, rather than suggesting universal superiority.

Critical weaknesses

The central heterogeneity–benefit claim is not directly tested. The paper repeatedly argues that AXIOM's advantage increases with activation heterogeneity, describing the cross-task results as a monotonic relationship and a dose-response pattern. However, it does not define an operational measure of heterogeneity or report activation statistics establishing the claimed ordering of the tasks. Calling environmental-sound tasks more heterogeneous is not sufficient to demonstrate that this property explains their larger gains.

Questions for authors

Clarification and validation of the central claim. See also weakness. What precisely constitutes activation heterogeneity in this work, and can you quantify it for the layers and tasks used in the experiments?

Reviewer_cCLWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper replaces an isotropic momentum accumulator with an activation-conditioned associative memory. A regularized least-squares objective yields a closed-form rank-one update, which is combined with Adam's second moment and online hyperparameter adaptation. The authors analyze contraction and convergence and evaluate the method with a frozen Wav2Vec2/wavelet/CNN pipeline on five audio tasks. Reported gains are substantial on several heterogeneous tasks, while speaker verification remains mixed.

Critical strengths
  • The regularized objective gives an explicit, interpretable update with activation-dependent forgetting. Its closed form and unforced contraction result provide a useful starting point for studying structured momentum.

  • The shared audio pipeline covers five tasks and reports both substantial gains and a mixed verification result, suggesting a potentially useful optimizer effect beyond a single benchmark.

Critical weaknesses
  • The theoretical claims need substantive correction. Appendix O drops the forgetting factor: with alpha=0.5, eta=1 and x=u=1, the stated update is m_next=0.25m+0.5 and converges to 2/3, contradicting exact retrieval and zero error in Eqs. 245–249. Separately, Eq. 203 factors activation-dependent operators out of a conditional expectation without establishing the required conditional unbiasedness; the convergence analysis also omits the implemented Adam preconditioner and joint hyperparameter dynamics.

  • The empirical evidence is difficult to assess or reproduce. Dataset identities/protocols for the clinical tasks, speaker/subject-disjoint splits, UrbanSound8K fold handling, actual training budgets and tuning ranges are insufficiently specified. A single reported seed gives no uncertainty for the large gains. These omissions prevent ruling out protocol effects or assessing robustness.

Questions for authors
  • How do Eqs. 245–249 change when the factor rho is retained? Does the corrected biased fixed point change the associative-memory claims?

  • What filtration or conditional-unbiasedness assumption justifies Eq. 203 when B_t depends on the same minibatch as g_t?

  • Do the experimental settings satisfy Lemma 10's additional feasibility condition? Which guarantees cover Adam preconditioning and online alpha/eta updates?

  • How is the rank-one update implemented for minibatches and convolutional patches? Is aggregation exact or approximate?

  • Please specify dataset versions, subject/speaker-disjoint splits, UrbanSound8K folds, training budgets and tuning ranges, and provide multi-seed uncertainty.

  • How is activation heterogeneity measured? What controlled comparison supports the claimed relationship between heterogeneity and optimizer gain?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Area Chairs

  1. Regarding concerns on theoretical updates and clarification (Reviewer cCLW, Liw6, HFqo): Revised the repeated-pattern fixed-point derivation in Appendix O, incorporating the forgetting factor (rho) and revising the exact-retrieval claim. Added the predictability and conditional-unbiasedness assumptions in Appendix L, and clarified the scope of Theorems 1 and 2, including trade-off between Lemma 10's sufficient feasibility condition and experimental settings.

Thus, the theoretical claims and experimental evidence are now well connected.

  1. Regarding the concerns about reproducibility (Reviewer cCLW, Liw6, HFqo): Expanded Appendix Q.5 with the actual minibatch and convolutional update formulations, clarifying that outer-product aggregation and patch extraction are exact, while the batch inverse is approximated using a trace-scaled expression.

Thus, reproducibility of results presented is enhanced now.

  1. Regarding the concerns about experimental Setup (Reviewer cCLW, HFqo): Expanded Appendix Q with experimental protocols, dataset splits, pre-processing, and training configurations. Following the earliest ICLR study, we added Appendix R reporting mean and standard deviation across three random seeds (42, 123, 2025) for all three optimizers across all five tasks yielding consistent results (as in table 3).

  2. Regarding the concerns about the claims (Reviewer Liw6): Since the original experiments did not experimentally define activation-heterogeneity relationships, the inadvertent use of this claim is avoided. Further, clarification related to experiments related to proposed AXIOM optimizer and its link with individual contribution of activation condition, Adams second-moment pre-condition, and online hypergradient adaptation is presented. Hence the claims have been revised and updated.

  3. On the whole, the revised manuscript binds well the theoretical reasoning along with experimental evidence and reproducibility of our work presented in this paper. (Reviewer cCLW, Liw6, HFqo)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.