- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Ano to improve tracking under noise and target drift in reinforcement learning. Ano uses the sign of momentum to determine update direction, the current gradient to determine magnitude, and an actively contracting second-moment accumulator. The motivation is practically relevant, and results across ten tasks and multiple seeds support its potential as a default optimizer replacement. However, more direct evidence is needed to establish whether the gains extend beyond changes in effective step size.
Critical strengths
-
Sections 2-3 clearly distinguish directional reliability, update magnitude, and recovery of the second-moment estimate. The algorithm changes only how the two existing state tensors are used, making the contribution clear and inexpensive to implement.
-
Table 1 compares default configurations on ten tasks with 25 seeds and reports both normalized effect sizes and probabilities of improvement. The latter are invariant to positive linear normalization within each task, providing substantive support for the main empirical result.
-
The conditional sign-coupling argument in Appendix B and the bound on accumulated momentum error in Appendix C support their stated conclusions within the specified assumptions. Table 3 also clearly distinguishes empirical results for practical Ano from theoretical guarantees for Scheduled Capped Ano.
Critical weaknesses
-
Sections 2.1 and 4.1 and Appendix F.3 primarily compare signal projections at the same learning rate, although Ano also increases the mean-squared update magnitude. The fixed-learning-rate result remains valid, but the evidence does not rule out obtaining similar tracking gains by adjusting Adam's step size.
-
Appendix G transfers configurations selected on a 100k-step HalfCheetah/SAC proxy to longer SAC runs and Atari/PPO. In Table 8, this tuning reduces Adam's Phoenix score from approximately 3377 to 409. The protocol therefore does not adequately test whether the default advantage persists after fair tuning within each algorithm family.
-
Table 2 evaluates the accumulator's incremental benefit in reinforcement learning only on HalfCheetah, and the authors explicitly acknowledge that the intervals for Ano and Ano-EMA overlap. The evidence supports the optimizer as a whole but does not establish the accumulator's independent contribution to the aggregate advantage across ten tasks.
Questions for authors
-
When replaying recorded gradients and optimizing noisy, drifting quadratic objectives? How do Ano, Ano-EMA, and Adam compare in tracking error and the cost of noise after matching mean-squared update norms or independently tuning each method's learning rate?
-
Does the default-configuration advantage persist after tuning on longer-horizon SAC runs and separate Atari/PPO validation tasks, with a finer learning-rate search between the current values spaced by factors of ten?
-
Can you compare Ano-EMA, AnoYogi, and Ano across all ten tasks with other settings fixed, and report the paired gain in aggregate return attributable to the accumulator, together with uncertainty intervals?
2 comments
Thank you for the response and for clarifying the scope of your claims. While I appreciate the added synthetic magnitude-matching check and the honest Limitations updates, my core empirical concerns remain unresolved: step-size matching was not tested on real SAC gradients (and common-LR Adam still achieves lower high-noise tracking MSE in the toy check), no fair within-family RL tuning was added, and ACA's independent contribution across the 10 tasks remains unverified. I will therefore maintain my score of 2
Thank you for your constructive assessment. We address your three questions below.
Tracking after matching update magnitude
Following your suggestion, we added the diagnostic “Update-Magnitude Matching Across Noise Regimes.” Adam’s learning rate is calibrated on three validation seeds to match Ano’s mean-squared update magnitude, then frozen and evaluated on five held-out seeds.
The held-out magnitude ratios are 1.013 under low noise and 1.021 under high noise. Despite this match, Adam’s tracking MSE is 0.02303 versus 0.01505 for Ano under low noise, and 0.90153 versus 0.76392 under high noise. Under low noise, Ano also first reaches MSE 0.01 in a median of 25 updates versus 46–47 for Adam, with all configurations reaching the threshold in every post-switch interval. These are descriptive first-passage measurements over seven correlated switches per seed.
The result is not uniform dominance: common-learning-rate Adam achieves lower average MSE under high noise (0.72535 versus 0.76392). The control therefore shows that matching average update magnitude does not reproduce Ano’s tracking behavior, rather than ruling out competitive Adam learning rates. The appendix provides the full results, cross-regime calibration checks, and Ano-EMA as a common-learning-rate—not amplitude-matched—control.
A controlled tuning comparison under a shared budget
Our tuning study aims to compare optimizers under a shared selection protocol, rather than find globally optimal configurations for each method. Adam and Ano are evaluated exhaustively over the same specified learning-rate and momentum grid, with five seeds per configuration and the same selection criterion. This gives both methods the same search opportunity, without optimizer-specific adjustments to the selection procedure.
We also report default, proxy-tuned, and best-of-default-and-tuned results separately, so an unfavorable transfer does not replace the default baseline in the main comparison. The Phoenix example illustrates why this distinction matters: a configuration selected on short-horizon HalfCheetah/SAC need not transfer well to Atari/PPO.
The shared budget therefore supports a controlled comparison, while leaving the adequacy of the proxy for each target family as a separate question. Longer-horizon, family-specific searches with finer learning-rate resolution would extend this evaluation; our current results establish the advantage under the reported default configurations, complemented by the shared-grid sensitivity study.
Clarifying the scope of the accumulator evidence
We agree that establishing the ACA's independent contribution across all ten tasks would require the broader paired ablation you suggest. Our current component ablation is limited to HalfCheetah, and the manuscript explicitly notes that the overlapping Ano/Ano-EMA intervals do not establish that contribution. The reported +6.7% is a descriptive difference in mean return, not a claim of statistical significance.
We have softened the ablation headings to avoid implying an independently established accumulator benefit. We have also made this distinction explicit in the Limitations section: the ten-task results evaluate Ano as a complete optimizer, while the accumulator's independent contribution to the aggregate gains remains to be established.