acceptodds
Under review as a conference paper at ICLR 2027

Ano: Adaptive Optimization for Tracking Under Noise and Drift

Abstract

Deep reinforcement learning asks optimizers to track moving objectives from persistently noisy gradients. We show that, in this regime, Adam's momentum improves the reliability of its update direction but not its progress. Because Adam's update is linear in the momentum, the variance reduction it buys is spent on shorter steps rather than on progress along the signal: at a fixed learning rate, Adam's signal-aligned progress per step matches that of RMSprop, which uses no momentum at all. This trade-off is sound when the objective is fixed and more steps can compensate, but costly when the objective moves. We introduce Ano, which uses momentum only to choose the direction of each coordinate, not the size of the step: the update takes the sign of the momentum and the magnitude of the current normalized gradient. The variance reduction then goes into direction rather than into shorter steps, roughly tripling signal-aligned progress per step in the high-noise limit. Applying the same logic to the second-moment estimate yields an Active Contraction Accumulator, which returns to the post-burst gradient scale in finite time where Adam's estimate returns only asymptotically. Ano keeps Adam's state and hyperparameter set. On real SAC gradient streams, it produces larger signal-aligned updates than Adam. Across 10 MuJoCo and Atari environments with 25 seeds, it improves aggregate return over Adam and four other adaptive and sign-based optimizers, while remaining competitive rather than superior on stationary supervised benchmarks, where shorter steps act as useful annealing.

Primary area:
optimization
Keywords:
Adaptive optimization, Stochastic optimization, Deep reinforcement learning, Non-stationary optimization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_v6cvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Ano to improve tracking under noise and target drift in reinforcement learning. Ano uses the sign of momentum to determine update direction, the current gradient to determine magnitude, and an actively contracting second-moment accumulator. The motivation is practically relevant, and results across ten tasks and multiple seeds support its potential as a default optimizer replacement. However, more direct evidence is needed to establish whether the gains extend beyond changes in effective step size.

Critical strengths
  1. Sections 2-3 clearly distinguish directional reliability, update magnitude, and recovery of the second-moment estimate. The algorithm changes only how the two existing state tensors are used, making the contribution clear and inexpensive to implement.

  2. Table 1 compares default configurations on ten tasks with 25 seeds and reports both normalized effect sizes and probabilities of improvement. The latter are invariant to positive linear normalization within each task, providing substantive support for the main empirical result.

  3. The conditional sign-coupling argument in Appendix B and the bound on accumulated momentum error in Appendix C support their stated conclusions within the specified assumptions. Table 3 also clearly distinguishes empirical results for practical Ano from theoretical guarantees for Scheduled Capped Ano.

Critical weaknesses
  1. Sections 2.1 and 4.1 and Appendix F.3 primarily compare signal projections at the same learning rate, although Ano also increases the mean-squared update magnitude. The fixed-learning-rate result remains valid, but the evidence does not rule out obtaining similar tracking gains by adjusting Adam's step size.

  2. Appendix G transfers configurations selected on a 100k-step HalfCheetah/SAC proxy to longer SAC runs and Atari/PPO. In Table 8, this tuning reduces Adam's Phoenix score from approximately 3377 to 409. The protocol therefore does not adequately test whether the default advantage persists after fair tuning within each algorithm family.

  3. Table 2 evaluates the accumulator's incremental benefit in reinforcement learning only on HalfCheetah, and the authors explicitly acknowledge that the intervals for Ano and Ano-EMA overlap. The evidence supports the optimizer as a whole but does not establish the accumulator's independent contribution to the aggregate advantage across ten tasks.

Questions for authors
  1. When replaying recorded gradients and optimizing noisy, drifting quadratic objectives? How do Ano, Ano-EMA, and Adam compare in tracking error and the cost of noise after matching mean-squared update norms or independently tuning each method's learning rate?

  2. Does the default-configuration advantage persist after tuning on longer-horizon SAC runs and separate Atari/PPO validation tasks, with a finer learning-rate search between the current values spaced by factors of ten?

  3. Can you compare Ano-EMA, AnoYogi, and Ano across all ten tasks with other settings fixed, and report the paired gain in aggregate return attributable to the accumulator, together with uncertainty intervals?

2 comments

Thank you for your constructive assessment. We address your three questions below.


Tracking after matching update magnitude

Following your suggestion, we added the diagnostic “Update-Magnitude Matching Across Noise Regimes.” Adam’s learning rate is calibrated on three validation seeds to match Ano’s mean-squared update magnitude, then frozen and evaluated on five held-out seeds.

The held-out magnitude ratios are 1.013 under low noise and 1.021 under high noise. Despite this match, Adam’s tracking MSE is 0.02303 versus 0.01505 for Ano under low noise, and 0.90153 versus 0.76392 under high noise. Under low noise, Ano also first reaches MSE 0.01 in a median of 25 updates versus 46–47 for Adam, with all configurations reaching the threshold in every post-switch interval. These are descriptive first-passage measurements over seven correlated switches per seed.

The result is not uniform dominance: common-learning-rate Adam achieves lower average MSE under high noise (0.72535 versus 0.76392). The control therefore shows that matching average update magnitude does not reproduce Ano’s tracking behavior, rather than ruling out competitive Adam learning rates. The appendix provides the full results, cross-regime calibration checks, and Ano-EMA as a common-learning-rate—not amplitude-matched—control.


A controlled tuning comparison under a shared budget

Our tuning study aims to compare optimizers under a shared selection protocol, rather than find globally optimal configurations for each method. Adam and Ano are evaluated exhaustively over the same specified learning-rate and momentum grid, with five seeds per configuration and the same selection criterion. This gives both methods the same search opportunity, without optimizer-specific adjustments to the selection procedure.

We also report default, proxy-tuned, and best-of-default-and-tuned results separately, so an unfavorable transfer does not replace the default baseline in the main comparison. The Phoenix example illustrates why this distinction matters: a configuration selected on short-horizon HalfCheetah/SAC need not transfer well to Atari/PPO.

The shared budget therefore supports a controlled comparison, while leaving the adequacy of the proxy for each target family as a separate question. Longer-horizon, family-specific searches with finer learning-rate resolution would extend this evaluation; our current results establish the advantage under the reported default configurations, complemented by the shared-grid sensitivity study.


Clarifying the scope of the accumulator evidence

We agree that establishing the ACA's independent contribution across all ten tasks would require the broader paired ablation you suggest. Our current component ablation is limited to HalfCheetah, and the manuscript explicitly notes that the overlapping Ano/Ano-EMA intervals do not establish that contribution. The reported +6.7% is a descriptive difference in mean return, not a claim of statistical significance.

We have softened the ablation headings to avoid implying an independently established accumulator benefit. We have also made this distinction explicit in the Limitations section: the ten-task results evaluate Ano as a complete optimizer, while the accumulator's independent contribution to the aggregate gains remains to be established.

Thank you for the response and for clarifying the scope of your claims. While I appreciate the added synthetic magnitude-matching check and the honest Limitations updates, my core empirical concerns remain unresolved: step-size matching was not tested on real SAC gradients (and common-LR Adam still achieves lower high-noise tracking MSE in the toy check), no fair within-family RL tuning was added, and ACA's independent contribution across the 10 tasks remains unverified. I will therefore maintain my score of 2

Reviewer_aosSOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Ano, a new optimizer based on Adam where the numerator of the step is changed from the momentum in Adam to the signe of the momentum times the absolute value of the gradient (sign(mi)⋅∣gi∣\text{sign}(m_i)\cdot |g_i|) and the second moment damping factor is changed to revert back to baseline in finite time rather than asymptotically after a burst in gradient magnitude : vk+1=β2vk+(1−β2)sign(gk+12−vk)⋅gk+12v_{k+1} = \beta_2 v_k + (1-\beta_2)\text{sign}(g_{k+1}^2 - v_k) \cdot g_{k+1}^2 (what the paper calls ACA). The paper shows with a mix of theoretical and empirical evidence that in the regime of constant average gradient, the dot product of Ano's step size with the true gradient is larger than Adam's. The paper goes on to give empirical evidence in RL (Mujoco+SAC, Atari+PPO) that with Ano the agents learn faster. They also give empirical evidence that for supervised learning it is not worse than Adam. Finally the paper provides a proof of convergence for a modification of Ano with decaying learning rate, β1→1\beta_1\rightarrow 1 , and capped vkv_k, in an L-smooth setting.

Critical strengths
  • The paper identifies two key weaknesses of Adam and finds fixes for them: 1) noise levels and step size are coupled, 2) slow recovery of damping after large gradients.
  • Provides empirical and theoretical evidence for faster optimization using those fixes. The use of 25 seeds per environment with stratified-bootstrap confidence intervals is a clear strength. Given the high variance of RL results, this strengthens confidence that the reported improvements are not artifacts of noise.
Critical weaknesses
  1. Unsubstantiated claim: "Under a fixed objective, this trade-off is sensible: shortening steps as noise grows acts as annealing, and additional steps compensate. Under a moving objective, the number of steps available before the target changes is limited, so per-step progress becomes the relevant quantity."

No theory or experiment are done to substantiate this specific claim. The only experiments done are showing that for RL, Ano generally learns faster (fig. 3), but seeing as the RL experiments do stop before plateauing, and where it does plateau (Half Cheetah and Hopper), Ano and Adam reach the same asymptotic performance. We can expect exactly the same thing to happen with supervised learning. Seeing as the authors do not provide their supervised learning training plots, this claim about there being a substantive difference between RL and supervised learning has no backing.

  1. Missing literature review of optimizers in RL: The paper aims to provide a new optimizer for RL but does not talk about existing work on the subject and does not put the current work in the context of that existing work. Missing papers include Adam on Local Time: Relative Adam Timesteps (Adam-Rel) (Ellis et al., 2024), Resetting the Optimizer in Deep RL (Asadi et al., 2023)). One of those optimizers designed specifically for RL should also have been included in the baselines.

I am very open to increasing my score if these two critical weaknesses are addressed. This paper can be an excellent paper.

Questions for authors

Clarification needed:

  1. The following sentence in the introduction seems unjustified, please justify it better or say where in the paper it will be justified (i.e. section 2.1 after Thm 2.3): "But Adam’s update is linear in the momentum, and averaging does not change the mean gradient: the ==variance reduction therefore shows up as a shorter step== rather than as more progress along the signal."
  2. Eqn 12-15 : X, A, Q all need a dependence on kk. There is not a single X, but many XkX_ks! Furthermore, what does ξ−6\xi_{-6} mean? ξ\xi's with negative indices need to be defined! Are they all set to 00?
  3. τ\tau in section 2.1 vs appendix B : τ\tau in 2.1 before Thm 2.3 seams to mean something different than in appendix B. This is confusing. Please clarify. Also in section 2.1 after Thm. 2.3 which definition does τ\tau use? I assume the one of appendix B, but that is not clear. Needs clarification.
  4. "Both forms of damping are sensible for a fixed objective and costly for a moving one: the first shortens steps as noise grows, the second keeps them short after the noise has receded." -> What both forms of damping? Only one form of damping is talked about here.
  5. Table 11: the average should also include uncertainty interval.
  6. Appendix C.1: Fk\mathcal{F}_k is not defined. Please define. (It is defined later in C.2 but should be defined when introduced.)
  7. What is the relationship of this work to more modern optimizer like Muon.
  8. How was the hyperparameter grid chosen and why is it different by task? Some of the learning rates appear unusual.
  9. Could you specify the RL hyperparameters in Section D.8, which is currently empty?
  10. Consistency and clarity of figures: The set of baselines changes from one figure to another; the same baselines should be used throughout the paper. Captions should state clearly what each panel shows (e.g., SAC on the left, PPO on the right). Some figures could also be made clearer; for instance, Figure 1c could include a horizontal dotted line indicating the pre-burst level.

Suggestions for future work:

  1. Due to the claim that Ano takes larger signal-aligned step sizes than Adam, one would expect that for supervised learning, Ano should learn faster than Adam. This should be super easy to show using supervised training curves. I would like and expect to see supervised training curves and see that while Ano and Adam might reach similar asymptotic performance, Ano should reach it faster.
  2. How close can Adam get to Ano when using a larger learning rate? Since a larger learning rate allows larger steps, it could partially compensate for the step-size shrinkage the paper identifies.
  3. Does adding the ACA to Adam substantially improve Adam’s performance? This would help isolate the contribution of each component.
  4. CIFAR-100/10 are very "small" datasets. It would have been nice to have results also for something more along the lines of ImageNet. Nowadays, such an experiment is not hard to do.
  5. Ablation scope: The ablation is performed on only one RL environment. It should cover all RL environments, and the results should be aggregated in the ablation table as performance changes relative to the full method.
6 comments

Thank you for identifying these two points and for the detailed clarification requests. We address the main concerns first, followed by the specific clarifications and suggestions.


Fixed versus moving objectives and supervised learning trajectories

You are right that faster learning followed by similar final performance can also occur in supervised learning. We have added the trajectories in Appendix E.1. Ano reduces training loss faster on CIFAR-100 and several GLUE tasks, while final predictive performance remains comparable at the evaluated training budgets. We have revised the fixed–moving discussion accordingly: the distinction motivates our study of optimizer responsiveness, rather than a speed advantage exclusive to RL.


Positioning relative to Adam-Rel and optimizer resetting

We agree that these methods are relevant to the paper’s positioning and that a direct comparison would be informative. Adam-Rel (Ellis et al., 2024) is already discussed in Section 7: it modifies Adam’s bias-correction timestep while retaining its moment estimates. We have expanded this discussion to include “Resetting the Optimizer in Deep RL” (Asadi et al., 2023), and explained more explicitly how these approaches relate to Ano.

Our baseline suite was designed to compare per-step update rules under the reported training protocols, rather than optimizer-specific reset interventions. Ano changes the update magnitude and accumulator dynamics; reset-based methods intervene in the optimizer’s state or timestep during training. Such interventions could also be investigated with Ano, although their implementation and benefits need not transfer unchanged.

This distinction explains our experimental scope, but does not remove the value of comparing against reset-augmented Adam. We therefore acknowledge this missing comparison as a limitation. A crossed evaluation of update rules and reset interventions—with explicit reset conventions and comparable tuning budgets—would distinguish their individual benefits from their interactions. We have identified both this comparison and the transferability of reset interventions to Ano as open questions in the Limitations section.


Clarifications

Analysis and notation (questions 1–4 and 6). We added a pointer to the analysis following Theorem 2.3 in Section 2.1 and explicitly named noise-dependent shrinkage and post-burst damping. We corrected the time indices of XX, AA, and QQ, defined negative-index noise, clarified τ\tau, and defined Fk\mathcal F_k at its first use in Appendix C.1.

Uncertainty and figures (questions 5 and 10). Table 11 now includes aggregate uncertainty. Captions identify settings and baseline selections; Figure 1c includes a pre-burst reference.

Muon (question 7). Section 7 now distinguishes Muon's matrix-level orthogonalization of momentum updates from Ano's coordinatewise momentum sign, current-gradient magnitude, and actively contracting accumulator.

Grids and RL settings (questions 8–9). We corrected the CV grid to {10−4,10−3,10−2}\{10^{-4},10^{-3},10^{-2}\}; this was a reporting error, not a change to the runs. We explain the domain-specific reference scales and GLUE's preliminary runs. Section D.8 now explicitly points to the SAC/PPO settings in Tables 9–10.


Additional suggestions

Adam with a larger learning rate. Appendix F.4 adds “Update-Magnitude Matching Across Noise Regimes.” On five held-out seeds, calibrated Adam/Ano mean-squared magnitude ratios are 1.013 and 1.021 under low and high noise, yet tracking MSEs remain 0.02303 versus 0.01505 and 0.90153 versus 0.76392. Under low noise, Ano first reaches MSE 0.01 in a median of 25 versus 46–47 updates, with all methods attaining it in all 35 intervals. These are descriptive first-passage results over seven correlated switches per seed, not sustained-accuracy guarantees. Common-learning-rate Adam has lower high-noise MSE (0.72535 versus 0.76392): matching average magnitude does not reproduce Ano's tracking behavior, but this is not uniform superiority or task-specific RL tuning.

Further evaluations. Supervised trajectories are included as discussed above. Adam + ACA, ImageNet, and ablations across the full RL suite are explicitly identified as future work in the Limitations, not completed experiments. The current results do not establish ACA's independent contribution across all RL tasks.

Reviewer_aosSOpenReview ↗

Optimizer's for RL discussion

I am happy with the updates to the related works section now including optimizers for RL.

Reviewer_aosSOpenReview ↗

Adam with a larger learning rate

Adding section F.4 addresses my concern.

Reviewer_DnjnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Ano, a drop-in replacement for Adam aimed at deep reinforcement learning, where each critic update is a noisy regression step toward a bootstrapped target that keeps moving. The authors argue that Adam’s momentum only buys more reliable update directions at the cost of shorter steps, so at a fixed learning rate it makes no more progress along the signal than RMSprop, which matters when a moving target leaves few steps to track it.

Theory is focused on a stationary noise model with a constant gradient mean, a variance-burst recovery calculation, and a convergence result for a capped variant, none of which model TD targets, target networks, or the sequential structure of RL. Empirically, Ano shows larger improvement over Adam, RMSprop, Adan, Lion and Grams across 5 MuJoCo/SAC and 5 Atari/PPO environments, while staying competitive on GLUE and CIFAR-100.

Critical strengths
  • Not an expert in optimization here, but the approach seems pretty easy to implement, and introduces no new hyper parameter relative to Adam.
Critical weaknesses

the entire framing of the paper is around RL, but the theory is basically agnostic to the fact that RL is the primary target. The main peculiarity in RL is the fact that, not only the target of the update is moving, but also that this target is set by taking the latest learned solution (the online network) from the previous inner step of optimization. In this sense, RL with TD like updates could best be viewed as, not just a single noisy problem, but actually a sequence of optimization problems. Similarly, one can view TD like updates as approximating a bilevel optimization problem, where in the inner loop the Bellman error is minimized while in the outer step the gap between the two networks is optimized (either exactly or through Polyak averaging)

This is all to say that to really claim a new more performant optimizer in RL, the specific structure of the RL problem needs to be taken into account, but currently the paper falls completely flat in this sense. Nothing in the optimizer makes it specifically promising for TD/critic learning.

Moving to the empirical side, no experiment varies the target-update period, the Polyak averaging parameter (0.005 in their SAC), or the number of gradient steps per target, even though these control the very drift the paper invokes. PPO in the Atari actually has no target network at all, so if Ano helps there it points to noise or step size rather than bootstrapped-target structure.

The optimizer may very well be useful, but the claim that it is specifically an RL/TD optimizer is unsupported, so the paper should either add some kind of analysis respecting the structure of the RL problem, and showing gains for Ano in light of that, or reframe as a generic noise-robust optimizer evaluated evaluated on different problems including on RL.

Questions for authors
  • Can we think of selling points for Ano specifically in the context of RL? for example, TD learning is know to diverge in the off-policy setting (even with linear function approximation). Is there any reason to believe Ano will help with that?

  • In general I am open to changing my mind if authors can show that Ano empowers existing RL algorithms to do better, for instance in enlarging the scope of convergence, or making convergence faster when such RL algorithms are already known to converge in these setups. The current experimental results are too generic, and provide minimal support for the overall framing of the paper.

3 comments

Thank you for your careful assessment. We address your concerns below


Why these optimizer-level mechanisms are relevant to deep RL

Ano targets the combination of persistent gradient noise and evolving learning problems. Its relevance to RL comes from these conditions, rather than from an update rule specific to Bellman targets.


We quantify persistent gradient noise and describe the sources of non-stationarity in the evaluated RL systems.

Appendix D.1 reports persistent gradient noise during training, particularly for the SAC critic, alongside changing value estimates, replay-buffer composition, and policy-dependent data distributions. Sections 2–3 study two optimizer responses to this setting: noise-dependent update shrinkage and prolonged damping after gradient-scale bursts. Ano changes how momentum sets update magnitude and how the second-moment estimate recovers from those bursts.


The experiments connect these mechanisms to practical RL performance.

Section 4.1 compares optimizers on identical recorded SAC critic gradients, while Section 4.2 evaluates them in the full learning loop across ten SAC/PPO environments with 25 seeds. Table 1 reports positive aggregate differences against all five baselines, with confidence intervals excluding zero. The replay diagnostic tests update-level behavior on real RL gradients; the end-to-end experiments establish the practical benefit within the evaluated systems.

The PPO results broaden this evidence beyond algorithms with explicit target networks: bootstrapping is one source of evolving learning problems, not a prerequisite for the proposed mechanisms. Varying target-update periods or Polyak coefficients would isolate target-network effects more specifically, but an optimizer-level improvement need not exploit Bellman structure to benefit RL training.

Whether Ano can prevent off-policy TD divergence remains an open question. Our experiments demonstrate a complementary benefit: replacing the optimizer improves aggregate return across the evaluated SAC and PPO tasks, without changing the underlying RL algorithms.

I don't really feel like authors engaged with my review. Presenting the optimizer with RL as its main customer, while not bringing to bear insights about the structure of the RL problem and Bellman error, seems disappointing, and therefore I am maintaining my score.

Thank you for the follow-up. We understand your concern that presenting RL as the main application calls for insights into its specific structure, particularly Bellman-error optimization.

Our response to this concern was to narrow the theoretical claims: the revised introduction describes a local persistent-noise analysis, and the Limitations explicitly state that these mechanisms are not complete models of deep RL training. We do not claim that Ano exploits Bellman structure or addresses off-policy TD divergence.

RL remains our main empirical application, rather than the source of an RL-specific theoretical guarantee. The contribution we aim to establish is an analysis of generic optimizer mechanisms under noise, together with evidence that the resulting update rule improves aggregate performance in the evaluated SAC/PPO systems. We recognize that this differs from the Bellman-specific insight you were seeking, but hope this clarifies how we addressed your concern and the distinction we make between the scope of the analysis and its application.

Reviewer_3dU5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new optimizer called Ano. The idea of Ano is to decompose the per-batch gradient gkg_{k} (at step kk) into ∣gk∣sign(gk)\lvert g_{k}\rvert \mathrm{sign}(g_{k}), and rewrite the update rules of RMSProp and Adam as: xk+1=xk−ηνk∣gk∣sign(gk)\mbox(RMSProp)x_{k+1}=x_{k}-\frac{\eta}{\sqrt{\nu_{k}}}\lvert g_{k}\rvert \mathrm{sign}(g_{k}) \quad\mbox{(RMSProp)} and xk+1=xk−ηνk∣mk∣sign(mk)\mbox(Adam)x_{k+1}=x_{k}-\frac{\eta}{\sqrt{\nu_{k}}}\lvert m_{k}\rvert \mathrm{sign}(m_{k}) \quad\mbox{(Adam)} where mkm_{k} and νk\nu_{k} are running averages of gkg_{k} and gk2g_{k}^2, respectively. Then, the authors propose xk+1=xk−ηνk∣gk∣sign(mk)\mbox(Ano−EMA),x_{k+1}=x_{k}-\frac{\eta}{\sqrt{\nu_{k}}}\lvert g_{k}\rvert \mathrm{sign}(m_{k}) \quad\mbox{(Ano-EMA)}, and further in Ano-ACA they replace νk\nu_{k} by νk^\hat{\nu_{k}} such that νk^=β2νk−1^+(1−β2)sign(gk2−νk−1^)gk2\hat{\nu_{k}}=\beta_2\hat{\nu_{k-1}}+(1-\beta_2)\mathrm{sign}(g_{k}^2-\hat{\nu_{k-1}})g_{k}^2, instead of the running average.

The theoretical motivation of this proposal is based on discussion of the "signal-aligned progress" metric, which is also proposed by the authors (to be discussed in detail below).

Empirical evaluation of the optimizer is conducted upon several deep RL tasks, GLUE and CIFAR-100. The authors argue that Ano has its advantage on RL tasks because these tasks have "moving objectives" and Ano has improved "signal-aligned progress".

Critical strengths

Although I believe the theoretical motivation is fundamentally flawed (to be discussed below), the authors have done a significant amount of work to clearly convey their intuitions based on reasonable theoretical assumptions and arguments. The paper is well written and easy to read (except for several logical jumps in the arguments which I believe require more evidence, but are intuitively understandable with some efforts), and the empirical evaluation, although being rather small experiments focused on RL, should be substantial enough for a sanity-check or proof-of-concept.

Critical weaknesses

I believe the metric of "signal-aligned progress", given in Definition 2.2 of the paper and the whole theoretical argument is based upon, is not well-defined. It is defined as below:

�0�$$ where $\Delta x_k^{\mathrm{alg}}$ is the optimizer update at step $k$, and $\nabla F(x)$ is the average gradient. The problem is that one cannot really distinguish the "learning-rate" $\eta$ from the optimizer update $\Delta x_k^{\mathrm{alg}}$. In the definition above, by dividing $\Delta x_k^{\mathrm{alg}}$ by $\eta$, the authors want to measure the "true size" of the update; but there is no general way of doing that. For any update rule in an optimizer, written in the form $\Delta x_k^{\mathrm{alg}}:=\eta u_k$, what if I say $(2u_k)$ is actually the "primitive update" and $\eta/2$ is the "learning-rate"? Substituting $\eta/2$ into the definition of "signal-aligned progress" above, its value will get doubled! So one can arbitrarily hack the value of this metric. If this argument sounds overbearing, what if I say I defined a new optimizer, which applies a non-linear function $f$ to the original update, and make the update rule $\Delta x_k^{\mathrm{alg}}:=(\eta/2) f(u_k)$? Behind the hood, the non-linear function $f(u_k)$ will return $2 u_k$ for most of the time, but will occasionally flip its value. It still gives a larger "signal-aligned progress" if we run a Monte Carlo simulation the same as in this paper! Essentially, it seems to me that all the authors' proposals are just doing this hack on the "signal-aligned progress" metric: By replacing $\lvert m_{k}\rvert$ with $\lvert g_{k}\rvert$ in Adam, the authors increased the expected size of this part of the optimizer update, because $g_{k}$ has the gradient noise. Other than this trick, the authors did not provide any further justification for this specific design. The exact definition of categorizing RL as having a "moving objective", is also unclear. All the justification I can find in the paper is in Section D.1, where the authors give a vague discussion saying that "Bootstrapped value estimates make the target depend on changing network parameters" (Line 1951), without formally defining what are moving objectives. Isn't "the target depend[ing] on changing network parameters" true for any deep neural networks, not limited to RL? Even with a rigorous definition of "moving objective", I believe the main argument of the paper, that "Under a moving objective, the number of steps available before the target changes is limited, so per-step progress becomes the relevant quantity" (Line 125) is still not well supported: Say, if we simply decrease the step size, intuitively the speed of network parameters changing will become slower, and because the "moving target" depends on the "changing network parameters", the target will move slower; so we have more steps available before the target changes a certain amount, being it limited! Therefore, this argument does not explain why Adam would perform worse on RL tasks, even if I agree with all the authors' assumptions!
Questions for authors

Have you thoroughly tuned your Adam baseline on the RL tasks?

3 comments

Thank you for your careful assessment. We address your concerns below.


What signal-aligned progress establishes—and why we examine update magnitude alongside it

We agree that SS is scale-dependent: reparameterizing the learning rate and update can change SS without changing the displacement. Our comparisons fix this convention explicitly, making them well-defined rather than invariant to reparameterization. Under this convention, SS measures mean signal-directed displacement, normalized by the learning rate and squared signal norm—not a convergence rate. Incorrectly directed updates contribute negatively, but their effect on the mean does not fully describe fluctuations or trajectory stability. Your amplification-and-sign-flip example therefore illustrates a genuine limitation of using SS alone. We have clarified these limits in Definition 2.2. The smooth-descent lemma in Appendix C.3.1 separately illustrates the first-order projection and quadratic update cost in the analysis of our scheduled capped variant.

Our design motivation concerns the response across noise regimes under a fixed optimizer configuration, rather than maximizing SS or UU at one noise level. We first examine update magnitude UU: Adam reaches a lower high-noise plateau than RMSprop, although both use second-moment normalization. We then examine SS, because retaining magnitude is insufficient if the resulting motion carries little useful signal. Together, these observations motivate using the current gradient magnitude while retaining the momentum’s coordinatewise sign decisions. This is a design objective, not a claim that SS remains constant as noise increases.

We also added “Update-Magnitude Matching Across Noise Regimes” to test whether simply increasing Adam’s learning rate reproduces Ano’s behavior. Calibration on validation seeds yields held-out mean-squared magnitude ratios of 1.013 under low noise and 1.021 under high noise. The corresponding tracking MSEs are 0.02303 versus 0.01505 for Ano, and 0.90153 versus 0.76392. Under low noise, Ano first reaches MSE 0.01 in a median of 25 updates versus 46–47 for Adam, with all configurations reaching the threshold in all 35 post-switch intervals. These are descriptive first-passage results over seven correlated switches per independent seed. Common-learning-rate Adam nevertheless achieves lower average tracking MSE under high noise (0.72535 versus 0.76392). Thus, matching average update magnitude does not reproduce the same tracking behavior, but the result does not establish uniform superiority or make SS sufficient to assess optimization. Full results are provided in the appendix.

We have revised the manuscript to distinguish the update-level analysis from claims about tracking performance. As stated in our Limitations section, these diagnostics do not capture the closed-loop interactions of RL training. Practical performance is evaluated through the end-to-end benchmarks, complemented by Q3’s sensitivity analysis on the tested grid. The revised text describes how noise affects signal-directed motion and update magnitude, while leaving the practical benefit to be assessed through closed-loop experiments.


Moving objectives and the role of additional steps

By “moving objectives,” we refer to evolving training targets and policy-dependent changes in the training-data distribution—not merely changing network predictions under a fixed supervised objective. We have made this distinction explicit in the introduction and discuss these sources of non-stationarity in Appendix D.1.

We agree that “limited number of steps” was too strong: target evolution is coupled to learning, and reducing the learning rate can also slow the target. Under a fixed objective, additional steps continue to optimize the same problem. In coupled actor–critic learning, they also influence subsequent policies, data, and targets. This motivates studying per-update behavior, without implying that larger per-step progress necessarily improves tracking. We have revised the corresponding statements in the abstract, introduction, and Appendix D.1 accordingly.


Adam tuning under a shared selection protocol

Adam and Ano were evaluated over the same specified grid of learning rates and momentum coefficients, with five independent seeds per configuration and the same selection criterion. All optimizers followed a common proxy-based tuning protocol, with search spaces adapted to their respective parameterizations.

The goal was a controlled comparison under a shared protocol and an assessment of hyperparameter sensitivity—not globally optimal configurations. For RL, configurations were selected on the HalfCheetah/SAC proxy and transferred to the final evaluations. We report default and tuned results separately, so an unfavorable transfer does not replace the default baseline in the main comparison.

Reviewer_3dU5OpenReview ↗

Thanks for the response

So the authors agree with my assessment that SS is scale-dependent -- Then I feel the whole story of the paper, that Ano works because it reduces variance more efficiently along the signal-aligned direction, is just not well supported. This is not a claim that one can easily accept, even intuitively, from the definition of the optimizer. Some more rigorous quantitative metrics are required. I would like to keep my score.

Thank you for the follow-up. We would like to clarify that our argument is not that Ano reduces gradient variance more effectively than Adam. Given the same gradient history and momentum parameters, both use the same coordinatewise momentum signs; Ano changes the magnitudes attached to those signs.

We agree that SS changes under a reparameterization of the learning rate and update rule, even when the actual displacement is unchanged, and that a larger SS alone does not establish a better optimizer. Our argument instead concerns how signal-directed displacement SS and update magnitude UU respond jointly to increasing noise, with the signal and optimizer hyperparameters held fixed. RMSprop retains larger normalized steps but uses noisier instantaneous gradient signs. Adam uses more reliable momentum signs, but its normalized update magnitude reaches a lower high-noise plateau. Their mean signal-directed displacements (SS) remain close in the high-noise regime studied. This motivates combining RMSprop’s current normalized gradient magnitude, whose UU remains stable at high noise, with the momentum’s more reliable signs.

This argument is supported by quantitative analysis, not inferred from the update rule alone: the local analysis establishes the same 1/σ1/\sigma decay for Ano-EMA, and numerical evaluations of the explicit coefficients show larger values than Adam and RMSprop across the tested settings.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of discussion and revisions

We thank the reviewers for their detailed feedback. For the Area Chair, we summarize how the discussion has strengthened the paper and distinguish the concerns addressed from those that remain.

1. Generic optimizer mechanisms and their relevance to RL — Dnjn, aosS, 3dU5.

The reviewers questioned different aspects of the fixed-versus-moving-objective motivation. In response, we clarified the paper’s positioning: Ano addresses generic optimizer responses to persistent noise and gradient-scale bursts, rather than exploiting Bellman-specific structure. RL is a motivated application because these conditions arise during training and learning efficiently from collected experience is important. Our response to Dnjn connects this motivation to measured gradient noise, diagnostics on recorded SAC gradients, and end-to-end SAC/PPO results.

We also revised the argument about a limited number of steps before targets change, acknowledging that target evolution is coupled to learning. Following aosS’s request, we added supervised trajectories: early-learning gains also occur outside RL, so they do not establish an advantage exclusive to RL. Dnjn maintains that the RL-centered motivation requires more direct evidence tied to RL structure, including target-update dynamics. aosS retains a framing concern but interprets the evidence as supporting a more broadly useful optimizer and explicitly supports acceptance.

2. Signal-directed progress and the role of step size — 3dU5, v6cv, aosS.

We addressed two related questions: what the diagnostic SS establishes, and whether increasing Adam’s step size reproduces Ano’s behavior. We clarified that SS depends on the learning-rate convention and is not a standalone measure of optimizer quality. The design rationale instead comes from studying SS and update magnitude UU jointly as noise increases under a fixed configuration.

We added the magnitude-matched tracking control in Appendix F.4. Matching average update magnitude does not reproduce Ano’s tracking behavior in this diagnostic, although it does not establish uniform superiority over Adam. aosS confirmed that F.4 addressed their concern. v6cv acknowledged the addition but maintains the request for controls on real SAC gradients.

3dU5’s follow-up focuses on the justification based on SS. In response, we reiterated the paper’s existing design rationale and quantitative analysis: Ano combines momentum signs with current normalized gradient magnitudes; it does not claim to reduce gradient variance more effectively than Adam. Their follow-up does not explicitly reassess our separate responses on moving objectives or baseline tuning.

3. Baseline tuning and component attribution — v6cv, 3dU5, aosS.

We clarified the shared search protocol and the separate reporting of default and proxy-tuned configurations. This documents how all optimizers were tuned, but does not replace the longer-horizon, within-family searches requested by v6cv. We also distinguish the performance of Ano as a whole from the accumulator’s independent contribution: the current component ablation does not establish that contribution across the full suite.

v6cv recognizes the support provided by the default-configuration benchmark and the validity of specified theoretical arguments under their assumptions, while maintaining the need for stronger empirical controls, within-family tuning, and broader component ablations. Broader ablations and Adam + ACA were also among aosS’s suggestions for future work.

4. Related work, clarity, and reporting — aosS.

We expanded the discussion of Adam-Rel and optimizer resetting, clarified the relationship to Muon, added supervised learning curves, corrected mathematical notation and hyperparameter reporting, and improved uncertainty reporting and figures. aosS explicitly confirmed satisfaction with the related-work update and the clarifications. We also changed Figure 11 to log-log scales following their suggestion. Direct comparisons with reset-based interventions and larger-scale supervised evaluations remain open.

The contribution combines quantitative analysis of optimizer responses to noise, a simple optimizer requiring no additional state or hyperparameters, and aggregate improvements across the evaluated RL tasks. We hope this summary helps the Area Chair assess that contribution alongside the concrete additions and the distinct questions remaining in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.