- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This manuscript studies whether changing Adam's memory parameters (β₁, β₂) can be compensated by adjusting a time-varying ε schedule. The authors derive theoretical conditions for preserving different properties of the training dynamics and provide some experiments to support their analysis.
Critical strengths
The manuscript provides a theoretical analysis of discrete Adam and derives conditions for preserving different properties when changing its memory parameters. The mathematical result is interesting.
Critical weaknesses
The manuscript assumes full-batch Adam. I am not sure how useful these results are for our real neural network training.
It also requires a sufficiently small learning rate but does not provide an explicit bound.
The CIFAR-10 experiment only uses linear heads on frozen representations. It reports a small change in a trajectory measure, rather than an improvement in loss or accuracy. I do not think this is enough to show that the method is useful in practice.
Questions for authors
-
Can you give an explicit bound on the learning rate, or explain how to check whether a given setting satisfies the assumptions?
-
Why is the effective clock in Eq. 5 chosen this way? Would the results change if the clock were shifted?
-
Can the method be extended to mini-batch Adam or the constant ε setting?
-
Does the method cover β₁ ≥ β₂ or β₁ = 0 (RMSProp-like)?
3 comments
3. Why Eq. 5, and what happens under a clock shift?
The clock accumulates the response of Adam's memory to the schedule. For the local log-decay rate is . Expanding the memory response around and integrating gives The first term captures leading progress; the second retains the cumulative slower correction. The discrete analysis controls the remaining terms.
The first-order local response is . Accumulating its constant part gives the linear term, and accumulating its part gives the logarithm. The two components of the clock therefore arise directly from the two time scales in the schedule.
Shifting leaves the leading and logarithmic coefficients unchanged and transforms the risk amplitude as For runs with matched leading and slower responses, a common shift preserves risk equivalence; separate shifts multiply the limiting risk ratio by . Comparisons at a fixed update count remain unchanged.
The amplitude transformation is visible directly in the exponential term: substituting gives . The constant factor is absorbed into the risk amplitude, while the leading and logarithmic coefficients retain their values.
4. Minibatch and constant-epsilon extensions.
For actual minibatch Adam in linear classification, Appendix E.3 gives a conditional risk bound computable from the observed history and combines these bounds into a time-uniform probability guarantee. On six fresh H3/H4 reshuffling trajectories at batch 256 and the original reference step, the numerical risk envelope after 704 updates is .001077–.001433 times the initial risk, at a 95% level simultaneous over the six trajectories and all finite times.
The derivation retains the actual Adam update formed jointly by the sampled gradient and historical moments. Its risk envelope can be updated as the training history becomes available.
Appendix E.4 further proves long-run velocity for an explicit exponential-loss family with coordinate sign conflicts, fixed minibatches and nonzero memory. Two specified instances satisfy the conditions at batch 256 and the original reference step.
The risk bound applies along the observed random trajectory; the velocity result describes the long-run behavior of the specified family with sign conflicts.
The practical prediction result also transfers to batch 32: on two fresh feature heads, schedule-only forecasts at 1.28M updates have mean absolute log-ratio error .006273, versus .021294 for the best of seven alternatives, a 70.5% reduction. All eight targets beat their own best alternative.
For constant epsilon, Appendix D.1 gives a separate result for separable mixed-sign exponential loss: any fixed admits computable sufficiently small fixed steps with explicit two-sided risk bounds and from the given initialization.
This gives an explicit constant-epsilon comparison for the annealed-schedule results, including bounds that start from the specified initialization.
5. Does the analysis include and ?
Yes. The original deterministic domain is with the corresponding step/schedule conditions. Figure 1 explicitly uses target and reference . With , the first moment becomes the current gradient, giving the RMSProp-like update used in our single-buffer experiment.
The general stochastic risk bound also permits either ordering. The specific stochastic velocity result in Appendix E.4 uses together with its stated stability conditions.
Thank you for your response. Could you show your edits in the revised manuscript?
From above comments, I think the authors have addressed most of my original questions. It is good. But the main concern for me to raise the score is the presentation.
In fact, I read it with a really hard work, like a technical report. Then, the interesting thing is when I read the last part, AI use statement:"ChatGPT and OpenAI Codex were used during mathematical formulation and derivation, proof development and checking, experimental design and implementation, numerical analysis, and manuscript preparation. The authors reviewed the generated material, verified the reported claims and experiments, and take full responsibility for the content of the paper."
I do not think using AI is a problem, but my concern is the current wiriting still hard to follow, espeically because it contains so many technical things, regardless of how the text generated. So much things. Fig. 1 is just one example, I could not understand why .5856264389167615 occurs. I expected RATE and TWO to be different (maybe it is why so many numbers) but they are exactly the same. So what is the point?
For these reasons, I do not think the current version is ready for ICLR, so now I would like to keep my score at weak reject.
Thank you for the questions about practical relevance and the reach of the analysis. We have added full-network results and concrete step-size, stochastic, and constant-epsilon analyses.
The revised Sections 5.4–5.5 bring finite-budget minibatch prediction and the CIFAR/SVHN network results into the main paper. Section 6 connects these uses with the initialization-checkable, stochastic and constant-epsilon results in Appendices D.2/D.4, E.3/E.4 and D.1. Appendix C provides the additional empirical protocols and complete comparisons.
1. Full-network training and practical value.
We now train all layers of ResNet-18 with minibatches and evaluate test loss and accuracy. CIFAR-10 uses the single-buffer (0,0.95) configuration; the CIFAR-100 (0.8,0.95) recipe is fixed from CIFAR-10.
Both the feature extractor and classifier are updated. Within each seed, methods share initialization and minibatch order, allowing paired comparisons of the configurations.
Across ten paired seeds per dataset:
RATE reduces CE relative to RAW by 4.77% and 3.20%, winning 9/10 and 10/10 seeds, and beats the fixed constants in all ten seeds on both datasets. Mean accuracy improves by 0.534 and 0.865 percentage points over RAW. Each method uses 2,000 updates, batch 256, no augmentation, and common parameter averaging/training-only BN. The constants are ϵ=10−8,η=.001 for CIFAR-10 and ϵ=.001,η=.003 for CIFAR-100.
The CIFAR-10 implementation also reduces optimizer-state storage from 85.251 to 42.625 MiB by actually removing the first-moment buffer. These experiments directly demonstrate calibration benefits in stochastic full-network training.
The accuracy gains show that calibration also improves the final class predictions, alongside the reduction in cross-entropy.
We further transferred the same CIFAR single-buffer recipe to SVHN street-number images without retuning, retaining full-network ResNet-18, 2,000 updates, batch 256, parameter averaging and training-only BN. Five fresh paired seeds give mean validation CE 0.222395 for RATE versus 0.235317 for RAW, a 5.49% reduction with 5/5 wins; accuracy increases from 94.424% to 94.612%. RATE beats each of six constant-epsilon/learning-rate combinations in every seed, with mean CE 10.32% below their best mean of 0.247978. Among all 15 configurations, including learning-rate schedules and projection controls, RATE has the lowest mean CE, 1.59% below the best control mean, REF's 0.225985. Optimizer-state storage remains half of REF's. This transfer to another image domain connects calibration directly to stochastic full-network training and prediction quality.
2. Explicit step-size conditions and checks.
Appendix D.2 supplies data- and initialization-computable sufficient conditions for the full exponential-loss vector expansion. They verify bounded normalized states, moment-filter stability, descent, and contraction, and allow mixed signs, noncollinear rows and degenerate geometry.
We additionally check all 34 original deterministic H3/H4 configurations at their original common step η=.004735687324502032, original weights and zero moments. Rigorous enclosures of 40,000 updates prove entry into a region invariant under the same step; Appendix D.4 connects this to the asymptotic conclusion. This makes the step-size requirement concrete for the experiments.
For a new exponential-loss configuration, the sufficient tests can be evaluated directly from the data and initialization. For the original H3/H4 runs, the completed prefix calculations establish entry into the region where subsequent updates are controlled at the same learning rate.