- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AXIOM, an activation-conditioned momentum update derived from a regularized least-squares associative-memory objective. It replaces the first moment of an Adam-style optimizer while retaining the second moment and adapting internal hyperparameters online. The paper analyzes the single-activation closed-form update, unforced stability, and a simplified stochastic optimization system, and evaluates a wavelet-based speech/audio framework on five tasks.
Critical strengths
-
The single-activation closed-form derivation in Section 3.2 is concrete and implementable. For fixed positive alpha and eta, the uniform contraction of the unforced memory recursion follows directly from the operator spectrum. These provide a useful starting point for studying activation-conditioned momentum.
-
The default configuration improves accuracy on four audio tasks, with particularly large reported gains on environmental sound. The paper also reports settings where the advantage does not persist, including alternative backbones or wavelets. These empirical signals warrant further investigation even though the current evidence does not overcome the soundness concerns below.
Critical weaknesses
-
There are concrete errors and unresolved assumptions in the theoretical account. In Appendix O (pp. 56–57, Eqs. 243–249), bar(M_t)=rho M_t with rho<1, but the fixed-point argument effectively identifies bar(M_t) with M_t. For a repeated pair (x,u), the actual fixed point is M*=u x^T/(alpha+||x||^2), hence M*x=||x||^2 u/(alpha+||x||^2), generally not u. For alpha=eta=x=u=1, the update is M_(t+1)=0.2 M_t+0.4 and converges to 0.5, not 1. The coefficient of u in Eq. 247 is also not 1-lambda_parallel. This invalidates the claimed exact repeated-pattern memory, although it does not invalidate the closed-form update or unforced stability. More centrally, Appendix L takes activation-dependent A_t/B_t outside conditional expectations (Eqs. 186, 203), without establishing the needed measurability or conditional centering when activation and gradient use the same minibatch. Unbiased raw gradients alone do not imply unbiased filtered gradients. Lemma 10 also imposes an additional spectral condition omitted from the informal main statement; writing k=mu_B/C_B=mu_P, that condition requires rho<k/(1+k)<=1/2 for some admissible epsilon. This is a restriction of the current proof, not a claim that higher-rho runs necessarily diverge.
-
The analyzed algorithm and empirical evidence do not yet establish the headline practical claim. Appendix L studies W_(t+1)=W_t-eta M_(t+1), without Algorithm 1's Adam denominator, while practical hyperparameters vary online. The single-sample rank-one update is not a complete definition for minibatches or unfolded convolutional patch matrices: a matrix least-squares objective instead involves (gamma I+2XX^T)^(-1), unless a specific approximation or sequential rule is used. The dataset identities, subject-level splits, UrbanSound8K fold protocol, tuning budgets and multi-seed variability are insufficiently specified. Finally, the claim that benefits scale with activation heterogeneity is not supported by a quantitative heterogeneity measure or controlled comparison. Seen-source accuracy near 99.9% does not establish saturation of unseen verification with EER around 13–16%. These issues collectively require more than a wording correction.
Questions for authors
-
Please confirm or correct the repeated-pattern fixed point in Appendix O. Please explain whether the one-dimensional example above requires revising the exact-retrieval and memory interpretation.
-
Please state the exact algorithm covered by Theorem 2. Specify the filtration and conditional-centering assumption for activation-dependent operators and the additional Lemma 10 spectral condition. State whether the experimental parameters satisfy these conditions.
-
Please provide the actual minibatch and convolutional-patch update. Clarify any averaging or sequential approximation and its complexity. Also specify dataset identities and subject-level splits and the UrbanSound8K fold protocol and independent tuning budgets.
-
Within the existing tasks please isolate activation conditioning from the retained Adam second moment and hypergradient adaptation. Report seed variability and any quantitative evidence connecting activation heterogeneity to gains.
My recommendation is rejection because the concrete mathematical error and central proof-to-algorithm gaps are not resolved by the current empirical evidence. These concerns do not imply that the reported measurements are fabricated or that activation-conditioned momentum is unpromising.
Response to Area Chairs
Thus, the theoretical claims and experimental evidence are now well connected.
Thus, reproducibility of results presented is enhanced now.
Regarding the concerns about experimental Setup (Reviewer cCLW, HFqo): Expanded Appendix Q with experimental protocols, dataset splits, pre-processing, and training configurations. Following the earliest ICLR study, we added Appendix R reporting mean and standard deviation across three random seeds (42, 123, 2025) for all three optimizers across all five tasks yielding consistent results (as in table 3).
Regarding the concerns about the claims (Reviewer Liw6): Since the original experiments did not experimentally define activation-heterogeneity relationships, the inadvertent use of this claim is avoided. Further, clarification related to experiments related to proposed AXIOM optimizer and its link with individual contribution of activation condition, Adams second-moment pre-condition, and online hypergradient adaptation is presented. Hence the claims have been revised and updated.
On the whole, the revised manuscript binds well the theoretical reasoning along with experimental evidence and reproducibility of our work presented in this paper. (Reviewer cCLW, Liw6, HFqo)