acceptodds
Under review as a conference paper at ICLR 2027

Memory Renormalizes Stabilizer Annealing in Adam

Abstract

Adam's memory parameters and denominator stabilizer both shape its dynamics, raising a natural question: when can a memory change be compensated by retuning the stabilizer, and what can such compensation preserve? For deterministic full-batch Adam on separable linear data, with sufficiently small fixed positive steps and an exponentially annealed outside-root stabilizer with polynomial modulation, the two moment filters renormalize the schedule into an effective rate and shape . These coordinates determine all divergent terms of the parameter trajectory, including iterated-logarithmic corrections on degenerate margin geometries. Matching is necessary and sufficient for the trajectory difference to converge to a finite vector under the prescribed effective clocks, while rate matching alone can leave logarithmic drift. Yet trajectory equivalence need not imply relative-risk equivalence: the residual finite offset can yield a nonunit asymptotic risk ratio. Preserving relative risk at equal effective time additionally requires an amplitude coordinate, whereas equal-update risk is governed by a distinct complete invariant; the two preservation targets can be incompatible. Experiments on actual discrete Adam iterates support these separating predictions across regular, degenerate, and native-softmax geometries. Together, these results give a clock- and observable-dependent classification of equivalence under Adam memory changes.

Primary area:
optimization
Keywords:
Adam, optimizer memory, stabilizer annealing, trajectory equivalence, hyperparameter calibration

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_5sjvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript studies whether changing Adam's memory parameters (β₁, β₂) can be compensated by adjusting a time-varying ε schedule. The authors derive theoretical conditions for preserving different properties of the training dynamics and provide some experiments to support their analysis.

Critical strengths

The manuscript provides a theoretical analysis of discrete Adam and derives conditions for preserving different properties when changing its memory parameters. The mathematical result is interesting.

Critical weaknesses

The manuscript assumes full-batch Adam. I am not sure how useful these results are for our real neural network training.

It also requires a sufficiently small learning rate but does not provide an explicit bound.

The CIFAR-10 experiment only uses linear heads on frozen representations. It reports a small change in a trajectory measure, rather than an improvement in loss or accuracy. I do not think this is enough to show that the method is useful in practice.

Questions for authors
  1. Can you give an explicit bound on the learning rate, or explain how to check whether a given setting satisfies the assumptions?

  2. Why is the effective clock in Eq. 5 chosen this way? Would the results change if the clock were shifted?

  3. Can the method be extended to mini-batch Adam or the constant ε setting?

  4. Does the method cover β₁ ≥ β₂ or β₁ = 0 (RMSProp-like)?

3 comments

Thank you for the questions about practical relevance and the reach of the analysis. We have added full-network results and concrete step-size, stochastic, and constant-epsilon analyses.

The revised Sections 5.4–5.5 bring finite-budget minibatch prediction and the CIFAR/SVHN network results into the main paper. Section 6 connects these uses with the initialization-checkable, stochastic and constant-epsilon results in Appendices D.2/D.4, E.3/E.4 and D.1. Appendix C provides the additional empirical protocols and complete comparisons.

1. Full-network training and practical value.

We now train all layers of ResNet-18 with minibatches and evaluate test loss and accuracy. CIFAR-10 uses the single-buffer (0,0.95)(0,0.95) configuration; the CIFAR-100 (0.8,0.95)(0.8,0.95) recipe is fixed from CIFAR-10.

Both the feature extractor and classifier are updated. Within each seed, methods share initialization and minibatch order, allowing paired comparisons of the configurations.

Across ten paired seeds per dataset:

DatasetRAW test CERATE test CEFixed-constant test CE
CIFAR-10.495048.471433.592715
CIFAR-1001.7810771.7241031.966834

RATE reduces CE relative to RAW by 4.77% and 3.20%, winning 9/10 and 10/10 seeds, and beats the fixed constants in all ten seeds on both datasets. Mean accuracy improves by 0.534 and 0.865 percentage points over RAW. Each method uses 2,000 updates, batch 256, no augmentation, and common parameter averaging/training-only BN. The constants are ϵ=10−8,η=.001\epsilon=10^{-8},\eta=.001 for CIFAR-10 and ϵ=.001,η=.003\epsilon=.001,\eta=.003 for CIFAR-100.

The CIFAR-10 implementation also reduces optimizer-state storage from 85.251 to 42.625 MiB by actually removing the first-moment buffer. These experiments directly demonstrate calibration benefits in stochastic full-network training.

The accuracy gains show that calibration also improves the final class predictions, alongside the reduction in cross-entropy.

We further transferred the same CIFAR single-buffer recipe to SVHN street-number images without retuning, retaining full-network ResNet-18, 2,000 updates, batch 256, parameter averaging and training-only BN. Five fresh paired seeds give mean validation CE 0.222395 for RATE versus 0.235317 for RAW, a 5.49% reduction with 5/5 wins; accuracy increases from 94.424% to 94.612%. RATE beats each of six constant-epsilon/learning-rate combinations in every seed, with mean CE 10.32% below their best mean of 0.247978. Among all 15 configurations, including learning-rate schedules and projection controls, RATE has the lowest mean CE, 1.59% below the best control mean, REF's 0.225985. Optimizer-state storage remains half of REF's. This transfer to another image domain connects calibration directly to stochastic full-network training and prediction quality.

2. Explicit step-size conditions and checks.

Appendix D.2 supplies data- and initialization-computable sufficient conditions for the full exponential-loss vector expansion. They verify bounded normalized states, moment-filter stability, descent, and contraction, and allow mixed signs, noncollinear rows and degenerate geometry.

We additionally check all 34 original deterministic H3/H4 configurations at their original common step η=.004735687324502032\eta=.004735687324502032, original weights and zero moments. Rigorous enclosures of 40,000 updates prove entry into a region invariant under the same step; Appendix D.4 connects this to the asymptotic conclusion. This makes the step-size requirement concrete for the experiments.

For a new exponential-loss configuration, the sufficient tests can be evaluated directly from the data and initialization. For the original H3/H4 runs, the completed prefix calculations establish entry into the region where subsequent updates are controlled at the same learning rate.

3. Why Eq. 5, and what happens under a clock shift?

The clock accumulates the response of Adam's memory to the schedule. For ϵ(S)=ϵ0e−kS(1+S/s0)λ,S=nη,\epsilon(S)=\epsilon_0e^{-kS}(1+S/s_0)^\lambda,\qquad S=n\eta, the local log-decay rate is k−λ/(S+s0)k-\lambda/(S+s_0). Expanding the memory response σ\sigma around kk and integrating gives s=σ(k)S−λσ′(k)log⁡(1+S/s0).s=\sigma(k)S-\lambda\sigma'(k)\log(1+S/s_0). The first term captures leading progress; the second retains the cumulative slower correction. The discrete analysis controls the remaining terms.

The first-order local response is σ(k)−λσ′(k)/(S+s0)\sigma(k)-\lambda\sigma'(k)/(S+s_0). Accumulating its constant part gives the linear term, and accumulating its 1/(S+s0)1/(S+s_0) part gives the logarithm. The two components of the clock therefore arise directly from the two time scales in the schedule.

Shifting s~x=sx+cx\widetilde s_x=s_x+c_x leaves the leading and logarithmic coefficients unchanged and transforms the risk amplitude as Γ~x=Γxerxcx.\widetilde\Gamma_x=\Gamma_x e^{r_xc_x}. For runs with matched leading and slower responses, a common shift preserves risk equivalence; separate shifts multiply the limiting risk ratio by er(ca−cb)e^{r(c_a-c_b)}. Comparisons at a fixed update count remain unchanged.

The amplitude transformation is visible directly in the exponential term: substituting s=s~−cs=\widetilde s-c gives e−rs=erce−rs~e^{-rs}=e^{rc}e^{-r\widetilde s}. The constant factor is absorbed into the risk amplitude, while the leading and logarithmic coefficients retain their values.

4. Minibatch and constant-epsilon extensions.

For actual minibatch Adam in linear classification, Appendix E.3 gives a conditional risk bound computable from the observed history and combines these bounds into a time-uniform probability guarantee. On six fresh H3/H4 reshuffling trajectories at batch 256 and the original reference step, the numerical risk envelope after 704 updates is .001077–.001433 times the initial risk, at a 95% level simultaneous over the six trajectories and all finite times.

The derivation retains the actual Adam update formed jointly by the sampled gradient and historical moments. Its risk envelope can be updated as the training history becomes available.

Appendix E.4 further proves long-run velocity for an explicit exponential-loss family with coordinate sign conflicts, fixed minibatches and nonzero memory. Two specified instances satisfy the conditions at batch 256 and the original reference step.

The risk bound applies along the observed random trajectory; the velocity result describes the long-run behavior of the specified family with sign conflicts.

The practical prediction result also transfers to batch 32: on two fresh feature heads, schedule-only forecasts at 1.28M updates have mean absolute log-ratio error .006273, versus .021294 for the best of seven alternatives, a 70.5% reduction. All eight targets beat their own best alternative.

For constant epsilon, Appendix D.1 gives a separate result for separable mixed-sign exponential loss: any fixed (β1,β2)∈[0,1)2(\beta_1,\beta_2)\in[0,1)^2 admits computable sufficiently small fixed steps with explicit two-sided risk bounds and Rn=Θ(1/n)R_n=\Theta(1/n) from the given initialization.

This gives an explicit constant-epsilon comparison for the annealed-schedule results, including bounds that start from the specified initialization.

5. Does the analysis include β1≥β2\beta_1\ge\beta_2 and β1=0\beta_1=0?

Yes. The original deterministic domain is [0,1)2[0,1)^2 with the corresponding step/schedule conditions. Figure 1 explicitly uses target (.5,.25)(.5,.25) and reference (0,0)(0,0). With β1=0\beta_1=0, the first moment becomes the current gradient, giving the RMSProp-like update used in our single-buffer experiment.

The general stochastic risk bound also permits either ordering. The specific stochastic velocity result in Appendix E.4 uses β12<β2\beta_1^2<\beta_2 together with its stated stability conditions.

Thank you for your response. Could you show your edits in the revised manuscript?

From above comments, I think the authors have addressed most of my original questions. It is good. But the main concern for me to raise the score is the presentation.

In fact, I read it with a really hard work, like a technical report. Then, the interesting thing is when I read the last part, AI use statement:"ChatGPT and OpenAI Codex were used during mathematical formulation and derivation, proof development and checking, experimental design and implementation, numerical analysis, and manuscript preparation. The authors reviewed the generated material, verified the reported claims and experiments, and take full responsibility for the content of the paper."

I do not think using AI is a problem, but my concern is the current wiriting still hard to follow, espeically because it contains so many technical things, regardless of how the text generated. So much things. Fig. 1 is just one example, I could not understand why .5856264389167615 occurs. I expected RATE and TWO to be different (maybe it is why so many numbers) but they are exactly the same. So what is the point?

For these reasons, I do not think the current version is ready for ICLR, so now I would like to keep my score at weak reject.

Reviewer_7fi7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to adjust Adam's stabilizer when its memory parameters change to keep the training trajectories close. The authors consider the full-batch Adam on separable linear data with a sufficiently small fixed learning rate. They identify two parameters, (r,Λ)(r, \Lambda), and provide adjusting rules for stabilizer to preserve these two parameters. They also show that difference between the original and modified trajectories converges to a finite vector if and only if this parameter pair is preserved.

Critical strengths

1.This paper uses two parameters, (r,Λ)(r,\Lambda), to characterize the Adam's long-time trajectories and provide the necessary and sufficient conditions for the trajectory equivalence at the equal effective time. It also provides the concrete rules for adjusting the stabilizer when the memory parameters change.

2.This paper gives explicit conditions for the relative-risk equivalence at the equal effective time and at the equal update count.

Critical weaknesses
  1. The motivation for preserving trajectory and achieving the relative-risk equivalence is not clear. A natural goal is to improve the optimization performance. The authors should explain why we need to preserve the trajectory and achieve the relative risk equivalence in practice.

  2. The theory results assume a sufficiently small learning rate, but do not give an explicit bound. Hence, it is not clear whether a chosen learning rate satisfies this assumption.

  3. This paper is somewhat hard to follow in some parts since some notations and formulas are introduced without definition and explanation. For example, the introduction section mentions "dynamical time" in the second paragraph and uses SS in the third paragraph before they are defined.

Questions for authors
  1. The paper compares the trajectories at the equal effective time. In practice, runs are typically compared after the same number of updates. Is the equal effective time mainly a tool for the theory analysis? Or does it have practical benefits?

  2. In Figure 3c, AMP achieves a lower training risk than the source risk while SYNC matches the source risk. This experiment aims to test whether the adjustment rules can match the source risk. So, what is the practical value of matching the source risk, even when AMP can achieve a lower training risk?

  3. How can we check whether the learning rates used in the experiments are small enough to satisfy the requirements for the theory results?

3 comments

Thank you for recognizing the trajectory and relative-risk characterizations. We address the purpose of preservation, its relation to update budgets, and the step-size and notation questions below.

The revised Introduction and Sections 3–4 explain the preservation goals and the three time coordinates through their roles; Figure 1 and Appendix C.1 provide a worked example. Sections 5.4–5.5 add prediction and full-network results at actual update budgets. Section 6 connects the computable step conditions and original-configuration verification in Appendices D.2/D.4.

1. Why preserve risk when AMP can achieve lower loss?

For the optimization objective shown in Figure 3c, AMP indeed achieves lower loss than SYNC. Preservation answers the complementary transfer question: which behavior of an established training recipe can be retained when Adam's memory changes, and by what explicit adjustment? It provides a controlled baseline for changing the optimizer while retaining its loss-versus-progress or loss-versus-update behavior. Improvements beyond the reference remain welcome.

Relative risk makes that transfer objective informative even when both losses vanish. One run's loss can remain twice the reference loss while their absolute difference tends to zero. The ratio detects this persistent discrepancy and gives preservation a precise observable target.

A practical motivation is removing the first-moment buffer. On CIFAR-10, the (0,0.95)(0,0.95) single-buffer implementation halves optimizer-state storage from 85.251 to 42.625 MiB. Across ten paired seeds of full-network ResNet-18 training, calibrated RATE reduces mean test CE from RAW's .495048 to .471433, with 9/10 seed-level improvements and +0.534 percentage points in accuracy. The runs use batch 256, 2,000 updates, no augmentation, and shared averaging/BN.

The same single-buffer recipe also transfers to SVHN without retuning, retaining the network and training settings above. Across five fresh paired seeds, mean validation CE decreases from RAW's 0.235317 to RATE's 0.222395, a 5.49% reduction with 5/5 wins; accuracy rises from 94.424% to 94.612%. RATE also beats each of six constant-epsilon/learning-rate combinations in 5/5 seeds, with mean CE 10.32% below their best. Its mean CE is the lowest among all 15 configurations, including learning-rate schedules and projection controls, and 1.59% below the best control mean, REF's 0.225985. Optimizer-state storage remains half of REF's. This illustrates how deriving compensation through preservation can lead naturally to improved loss in a concrete training task.

AMP and SYNC expose an important distinction. AMP preserves risk at aligned effective time; SYNC preserves risk at equal update counts. In the Figure 3 instance, reaching e−16e^{-16} takes 5,351/5,339/5,351 updates for REF/AMP/SYNC. Across all 13 fixed comparisons, SYNC matches REF's update count at this threshold, while AMP/REF ranges from .96063 to 1.04161. The choice of preserved quantity therefore has a measurable effect on update cost.

The reason is that the effective clock itself changes with memory. Two runs can have the same loss law at aligned progress yet require different numbers of updates to reach that progress. AMP and SYNC therefore correspond to two concrete questions: what is preserved at the same training progress, and what is preserved at the same computational update budget?

The broader theoretical consequence is that some memory changes cannot simultaneously preserve the trajectory and equal-update risk. The paper identifies these tradeoffs and supplies the calibration appropriate to each objective.

2. Is effective time useful beyond the analysis?

Effective time separates changes in training progress from changes in the path itself. It is computed from the configuration before observing a trajectory. For comparison at actual update counts, the theory gives

\qquad \vartheta=k\eta.$$ Matching $(\vartheta,\lambda,K)$ characterizes equal-update relative-risk preservation. It also quantifies threshold-reaching cost: as the target risk approaches zero, the ratio of the required update counts approaches the inverse ratio of the leading decay rates. We further evaluate prediction directly in update counts. On two fresh CIFAR-10 feature heads with minibatch Adam, batch 256, a predictor using only the first **320k target updates** forecasts risk at **1.28M updates**. Every forecast is fixed before continuation. Mean absolute log-risk error is **.019118**, versus **.232684** for the best of 17 equal-prefix controls: a **91.8%** reduction, with wins in all eight target/head combinations. The predictor uses the first quarter of the target budget to estimate loss at the full budget. This gives a forecast of the outcome of continued training while that outcome is still unobserved.

3. How can a chosen learning rate be checked?

For mixed-sign exponential-loss problems, Appendix D.2 computes sufficient conditions from the data, initialization and optimizer parameters. The procedure bounds the normalized gradient and moment states, checks filter stability and descent inequalities, and verifies contraction of the limiting subsystem. These tests establish the full vector expansion, including slower corrections and the finite residual. A noncollinear three-sample example passes all conditions at η=.01\eta=.01.

For the original H3/H4 experiments, we verify the actual common step η=.004735687324502032\eta=.004735687324502032 for all 34 original deterministic configurations, starting at their original weights and zero moments. Rigorous enclosures of the first 40,000 updates bring each run into a region invariant under that same step. Appendix D.4 provides this initialization-to-asymptotic verification. The chosen experimental step therefore has a concrete applicability argument.

The finite-prefix enclosure establishes entry from the original initialization; invariance controls subsequent updates. Both use the same experimental learning rate.

4. Definitions and readability.

The revised introduction defines the three coordinates before using them: nn counts completed updates, S=nηS=n\eta is accumulated step, and ss is configuration-computed effective time.

These quantities have three distinct jobs: nn describes the update budget, SS specifies where the run is in the declared epsilon schedule, and ss aligns progress between configurations.

The running example then identifies the reference and target, specifies the fixed and adjusted parameters, and explains the comparison clock before interpreting the plots. RATE, TWO, AMP and SYNC are introduced through the training property each preserves.

Thank the authors for the response. The memory-saving example explains why one might change Adam's memory parameters, but the motivation for matching the source risk is still my main concern. This is a key issue for my assessment since preservation is the central goal of this paper. With the same memory budget and the number of updates, when would matching the source risk be preferable to achieving a lower risk? I would appreciate a concrete example/application.

I am also still unclear about the practical benefit of comparing runs at equal effective time. What does this comparison help us decide beyond comparing runs at equal update count?

Presentation is another concern of mine (weakness 3). The issue was not only that some terms were used before they were defined. The paper introduced many technical terms and formulas without enough explanation, making the main ideas hard to follow. You mentioned a revised manuscript, but I have not seen it yet. Has it been uploaded?

Please keep the response concise and directly address each weakness and question.

Reviewer_ZXczOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper seems to study the dynamics of Adam for classification problems. The major technical innovation is to study the ϵ\epsilon in the denominator of Adam update. The manuscript designs an adaptive rule for determining ϵ\epsilon, and studies a series of theoretical properties.

Critical strengths

The problem studied seems to be important.

Critical weaknesses
  1. The contribution is unclear. The setting is rather artificial as most practitioners set ϵ\epsilon to be constant. The motivation of having a dynamic update rule for ϵ\epsilon is not well established. The numerical settings are not clearly introduced as well. Therefore, it is difficult to evaluate how important the proposed techniques are.

  2. The writing is difficult to understand. There are excessive jargons, including "outside-root stabilizer," "pole legality," "eligible run," "weak tight rows," "KL-selected primary dual multiplier," "Burg-barrier selector," "native-softmax," "nonradial instance." Many paragraphs have discontinuous semantic meanings.

Questions for authors

Can authors summarize the contributions in simple language? What new knowledge does the paper contribute?

Why should practitioners care about the findings in the paper?

2 comments

Thank you for asking us to clarify the motivation and explain the contribution in accessible terms.

We have rewritten the Introduction around the role of the stabilizer schedule and the new knowledge supplied by the preservation criteria. Section 3 explains the clocks and terminology, while Sections 5.4–5.5 report finite-budget prediction and full-network transfer. Appendix C includes the worked example and complete additional comparisons, including the five-seed SVHN study.

1. Why study dynamic epsilon when constant epsilon is common?

A fixed epsilon has a changing role during training. In Adam's denominator vt+ϵ\sqrt{v_t}+\epsilon, the balance between the historical gradient scale and epsilon determines how strongly coordinates are normalized. As gradients decrease, even a small constant epsilon can reshape the update. Scheduling epsilon lets us study and control this interaction with Adam's memory.

This effect also differs from changing a scalar learning rate. A learning-rate change scales every coordinate's update by the same factor. Changing epsilon has different relative effects on coordinates with different second-moment estimates, and therefore changes their relative update sizes. Its interaction with the moment averages deserves analysis even when a constant value is the usual practical choice.

Our schedule is prescribed before training. Its exponential and polynomial factors expose the leading response and a slower correction. The concrete question is: when memory changes, how should the schedule change to preserve a specified training behavior?

Because the schedule is prescribed, the compensation is computed from the configurations before training and can then be tested against the resulting trajectories and losses.

2. What new knowledge does the paper contribute?

The closest annealing analysis establishes direction selection for a memoryless smoothed-sign update Wang et al., 2026. We retain both Adam moment averages and fixed steps, and establish three further conclusions for separable linear classification:

Memory changes the effective schedule. We quantify how the two averages respond to annealing and derive explicit compensation rules.

Matching the leading trend is insufficient. Runs can share a limiting classifier direction while their parameters keep separating. We identify the second condition that removes this growing discrepancy, including drift transverse to the main direction.

Intuitively, agreeing on where the classifier points does not determine the path taken to get there. Even after aligning leading progress, a slower sideways separation can accumulate. The second calibration removes this component, and the two-dimensional example makes the distinction visible.

Trajectory and risk preservation are different objectives. Even a finite parameter offset can leave a persistent loss ratio. We characterize risk preservation at aligned progress and at equal update counts, and show when it is incompatible with trajectory preservation.

3. Why should practitioners care?

One use is transferring a working recipe to a lower-storage optimizer. Our CIFAR-10 single-buffer implementation uses (β1,β2)=(0,0.95)(\beta_1,\beta_2)=(0,0.95) and halves optimizer-state storage, from 85.251 to 42.625 MiB. Full-network ResNet-18 results across ten paired seeds are:

DatasetRAW test CERATE test CEFixed-constant test CE
CIFAR-10.495048.471433.592715
CIFAR-1001.7810771.7241031.966834

Both use 2,000 updates, batch 256, no augmentation, and common averaging/training-only BN. The CIFAR-100 (0.8,0.95)(0.8,0.95) recipe is fixed from CIFAR-10. RATE improves over RAW by 4.77%/3.20%, winning 9/10 and 10/10 seeds; it beats the fixed constants in all ten seeds on each dataset. Accuracy increases by 0.534/0.865 percentage points over RAW. The constants are ϵ=10−8,η=.001\epsilon=10^{-8},\eta=.001 for CIFAR-10 and ϵ=.001,η=.003\epsilon=.001,\eta=.003 for CIFAR-100.

We also compare against constant epsilon equal to the dynamic schedule's geometric mean, at the same learning rate. In five-seed comparisons, RATE/GMEAN test CE is .464683/.565993 on CIFAR-10 and 1.731017/1.937397 on CIFAR-100, with RATE winning every seed.

This comparison holds the learning rate fixed and matches a typical numerical scale of epsilon, making the temporal schedule itself the object of comparison. Together with RAW versus RATE on the same target-memory implementation, it explains how the calibration contributes to the resource-motivated transfer.

The completed SVHN study transfers the same CIFAR single-buffer recipe to street-number images without retuning, retaining the network, budget and common processing above. Across five fresh paired seeds, mean validation CE is 0.222395 for RATE versus 0.235317 for RAW, a 5.49% reduction. The best of six constant-epsilon/learning-rate combinations gives 0.247978, and GMEAN gives 0.266365: RATE is 10.32% and 16.51% lower, respectively. RATE beats RAW and each of these seven fixed-epsilon controls in 5/5 seeds. Its mean CE is the lowest among all 15 configurations, including learning-rate schedules and projection controls, and is 1.59% below the best control mean, REF's 0.225985. Accuracy rises from RAW's 94.424% to 94.612%, while optimizer-state storage remains half of REF's.

A second use is predicting the effect of configuration changes. A schedule-based rule, fixed before training, predicts the target/reference loss ratio at 1.28M updates. Mean absolute log-ratio error is .005969 versus .021345 for the best of seven alternatives at batch 256, and .006273 versus .021294 at batch 32: reductions of 72.0% and 70.5%. Each study uses two fresh CIFAR-10 feature heads, and all eight target/head combinations beat their own best alternative.

The practical question is: with this memory and schedule, what loss should we expect relative to the reference after the planned number of updates? The prediction connects optimizer settings to a concrete training outcome that can be evaluated in advance.

4. Experimental setup and clearer exposition.

Figure 1 provides a running example. All runs use L(w)=e−z⊤wL(w)=e^{-z^\top w}, z=(1/3,1)z=(1/3,1), zero weights/moments, both bias corrections, η=.01\eta=.01, ϵ0=1\epsilon_0=1, s0=10s_0=10, and 10610^6 updates. For S=nηS=n\eta and ϵ(S)=e−kS(1+S/10)λ\epsilon(S)=e^{-kS}(1+S/10)^\lambda:

Run(β1,β2)(\beta_1,\beta_2)kkλ\lambda
REF(0,0)(0,0)7/127/124
RATE(.5,.25)(.5,.25).585626444
TWO(.5,.25)(.5,.25).585626444.01583278

RATE matches the leading response; TWO also matches the slower correction. Configuration-computed clocks align progress. The left panel shows projected parameter drift; the right shows its growth per unit log-time, with theoretical levels .01577 for RATE and zero for TWO.

We introduce terminology through its role: "epsilon added after the square root," "stable moment averaging," and "a configuration satisfying the stated conditions." Weak tight rows are minimum-margin samples with zero selected dual weight; the KL/Burg optimization problems select the coefficients governing the dynamics. We describe native softmax as direct multiclass softmax training, and nonradial drift as drift not parallel to the main motion.

These results distinguish what is needed to preserve the leading motion, the full divergent trajectory, and the loss ratio. They also identify changes in memory for which the desired properties cannot all be retained.

Reviewer_XCjnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether changes to Adam’s two moment-decay parameters β1\beta_1 and β2\beta_2, can be compensated by giving a schedule to the stabilizer ϵ\epsilon with polynomial modulation. For full-batch Adam on linearly separable data, with sufficiently small fixed step sizes, it identifies two effective coordinates, rr and Λ\Lambda, that characterize all divergent terms of the parameter trajectory under a prescribed, configuration-dependent time reparameterization. Matching these coordinates makes the parameter difference converge to a finite, potentially nonzero vector. The paper further derives conditions for matching relative training risk at equal effective time or equal update count, and shows that these preservation objectives can be incompatible.

The validations include synthetic experiments comparing actual discrete Adam iterates with theoretical predictions for velocity deformation, logarithmic trajectory drift, and relative-risk matching. An additional experiment on frozen CIFAR-10 representations evaluates the reduction in trajectory mismatch obtained by matching both effective coordinates rather than the effective rate rr alone.

Critical strengths
  1. The theoretical analysis addresses a nontrivial level of asymptotic detail. It concerns the actual fixed-step Adam recurrence with both moment filters, rather than only a continuous-time approximation. The treatment of degenerate geometries, including iterated-logarithmic corrections and convergence of the terminal residual, appears technically substantial. The analysis also does not require a full-rank margin matrix or exclude weak tight constraints.

  2. The paper distinguishes several notions of preservation that should not be conflated. In particular, it separates bounded parameter differences from unit training-risk ratios, and equal effective time from equal update count. These distinctions are supported by explicit calibration rules and separating examples. The synthetic experiments are closely aligned with the theoretical claims, comparing measured quantities against configuration-based predictions rather than relying only on qualitative agreement.

Critical weaknesses
  1. The motivation and significance of the preservation objective are not sufficiently established. It is unclear what practical need motivates changing β1\beta_1 and β2\beta_2 on the same learning problem while seeking to reproduce the reference run’s asymptotic training behavior. Matching relative training risk can be a meaningful objective in its own right, but the manuscript does not sufficiently explain why it is an important target in this setting, or what broader optimization insight follows from achieving it. The introduction establishes a mathematically well-defined compensation question, but I remain unconvinced that it establishes the importance of that question.

  2. The scope and interpretation of the guarantees substantially limit their operational relevance. The analysis requires deterministic full-batch updates, fixed separable linear data, a particular ϵ\epsilon-annealing family, and sufficiently small fixed step sizes. Moreover, trajectory equivalence is established after configuration-dependent time alignment and permits a nonzero limiting parameter offset. It therefore does not imply that the runs produce close parameters at equal update counts, or that their finite-budget training behavior is similar. These limitations are acknowledged, but the paper does not provide a sufficiently clear connection between the asymptotic guarantees and a useful finite-budget consequence. The absence of a computable common step-size threshold further complicates applying the result to a chosen configuration.

  3. The empirical evidence supports the proposed asymptotic mechanisms more clearly than it establishes their significance. The synthetic experiments provide useful checks of the formulas, but do not resolve why preserving these quantities is desirable. The frozen-representation experiment reports a mean relative reduction of approximately 0.306% in its trajectory-mismatch diagnostic compared with rate-only matching. Although the numerical precision checks support the consistency of this effect, its implications for predictions or other finite-budget training outcomes remain unclear. I do not regard a performance improvement as necessary for a theoretical contribution, but this experiment provides only limited support for the practical relevance of the calibration rules.

  4. The presentation in the technical part makes the central ideas unnecessarily difficult to access. The reader must track several time variables, effective coordinates, preservation criteria, and calibration names before their relationships become intuitive. Although the relevant mathematical definitions are generally present, they do not always explain why a particular comparison is being made or how to interpret the resulting plots. A running example that explicitly lists the reference and target configurations, identifies which quantities are held fixed, and explains each observation clock would substantially improve readability.

Questions for authors

Questions to the Authors

  1. What is the main motivation for preserving the reference run' s asymptotic training risk after changing β1\beta_1 and β2\beta_2, and without changing the task itself? Could you provide either a concrete use case or a broader theoretical consequence that makes this objective important, independently of exact parameter matching?
  2. In the experiment where the CIFAR-10 feature representations are kept fixed and only the linear classification heads are trained, TWO reduces the trajectory-mismatch measure by approximately 0.306% on average compared with RATE. Could you report the actual values of this measure for RATE and TWO, rather than only the percentage reduction, and explain how to interpret the size of the difference? In particular, does this reduction correspond to any observable difference in the models’ predictions or training loss, or is its main significance that the parameter trajectories are more closely matched under the chosen measure?
  3. For the two-dimensional synthetic example in Figure 1, could you walk through how the REF, RATE, and TWO runs are configured and compared? A single table listing the loss function, initialization, bias-correction setting, number of updates, and hyperparameters (β1,β2,η,ϵ0,k,λ,s0)(\beta_1,\beta_2,\eta,\epsilon_0,k,\lambda,s_0) for each run would be helpful. Please indicate which values are kept fixed and how the adjusted kk and λ\lambda are obtained. It would also help to explain how the runs are aligned at the same effective time and how the quantities plotted in the left and right panels are calculated from the resulting parameter trajectories.
3 comments

Thank you for recognizing the technical contribution and for identifying where its significance needs to be clearer.

The revised Introduction explicitly explains the motivation for changing memory and preserving relative risk. Sections 5.4–5.5 and Tables 1–2 present the finite-budget forecasts and full-network results, including SVHN. Figure 1 and Appendix C.1 now provide the complete worked example; Appendix C gives the additional protocols and comparison tables.

1. Motivation and significance: why change memory and preserve relative risk?

Our question is whether an established training recipe can be transferred to a different Adam memory configuration through explicit compensation. A concrete reason is resource use: setting β1=0\beta_1=0 and removing the first-moment buffer halves optimizer-state storage. The practical objective is to retain or improve training quality under the new resource constraint.

Risk preservation makes the transfer problem precise: starting from a working recipe, which changes in memory can be compensated, and what schedule adjustments recover its loss behavior?

Directly searching for lower loss remains useful, but the search itself does not explain how a successful configuration should change when its memory settings change. Our analysis supplies that relationship. Its compensation equations give a starting configuration under the new resource constraint, with the reference loss as a known benchmark. Further tuning can then pursue lower loss. Preserving relative risk is therefore a way to transfer an effective recipe systematically, rather than treating each memory configuration as a separate tuning problem.

Relative risk is informative even when both losses vanish: if the target loss remains twice the reference loss, their absolute difference still tends to zero, whereas their ratio detects the persistent discrepancy.

This criterion concerns the loss itself and remains meaningful for diverging parameter norms in separable classification. A finite offset in the parameters can change margins enough to leave a constant loss factor. Consequently, agreement in classifier direction, convergence of the parameter difference, and a risk ratio approaching one are distinct conclusions. Our characterization explains how they are connected.

The resulting insight is that different preservation goals require different calibrations. Matching the divergent trajectory terms can leave a non-unit risk ratio, which requires amplitude calibration. Moreover, we characterize when trajectory preservation and equal-update risk preservation are incompatible. Thus the analysis identifies both the available compensation and the tradeoffs between training objectives.

We now support this motivation with full-network ResNet-18 experiments. On CIFAR-10, the single-buffer (0,0.95)(0,0.95) implementation reduces optimizer-state storage from 85.251 to 42.625 MiB. Across ten paired seeds, using 2,000 updates, batch 256, no augmentation, and common parameter averaging/training-only BN, we obtain:

Dataset / target recipeUncalibrated RAW test CECalibrated RATE test CEReduction
CIFAR-10 / single buffer0.4950480.4714334.77%
CIFAR-100 / transferred (0.8,0.95)(0.8,0.95) recipe1.7810771.7241033.20%

RATE improves CE in 9/10 and 10/10 seeds, respectively, with mean accuracy gains of 0.534 and 0.865 percentage points. The CIFAR-100 recipe is fixed from CIFAR-10.

RAW and RATE use the same target memory configuration and, on CIFAR-10, the same single-buffer implementation. Their comparison therefore measures the quality benefit of calibration within the resource-saving configuration. Methods share initialization and minibatch order within each seed. The ten-seed results assess repeatability, while the fixed CIFAR-100 transfer assesses reuse of the recipe.

We further transferred the CIFAR single-buffer recipe to SVHN street-number images without retuning. With the same full-network ResNet-18, 2,000 updates, batch 256, parameter averaging and training-only BN, five fresh paired seeds give mean validation CE 0.222395 for RATE versus 0.235317 for RAW, a 5.49% reduction with 5/5 wins. Accuracy increases from 94.424% to 94.612%. RATE also beats each of six constant-epsilon/learning-rate combinations in all five seeds; its mean CE is 10.32% below their best mean of 0.247978. Across all 15 configurations, including the reference, constant controls, learning-rate schedules and projection controls, RATE has the lowest mean CE, 1.59% below the best control mean, REF's 0.225985, while retaining half of REF's optimizer-state storage.

The same calibrated single-buffer recipe thus improves loss and accuracy on a second image domain without a new parameter search.

2. What useful finite-budget consequence follows?

The trajectory analysis identifies terms that carry information about subsequent loss. We tested the original higher-order predictor under minibatch Adam on two fresh CIFAR-10 feature heads, using the reference curve and the first 320k target updates. At the prespecified 1.28M-update endpoint, mean absolute log-risk-ratio error is 0.031288, versus 0.044808 for the best of 29 generic controls. Removing the shape or memory term raises error to 0.371662 or 0.111355. These ablations connect the theoretical structure directly to predictive information.

The ablations test whether the memory and shape terms help predict an unobserved later loss. Their removal increases error substantially, showing that the slower structure contains predictive information beyond the leading trend.

The risk analysis also motivates a simpler schedule-based predictor. Define Ix(n)=∑j=1nηxϵx((j−1)ηx).I_x(n)=\sum_{j=1}^{n}\frac{\eta_x}{\epsilon_x((j-1)\eta_x)}. We predict Rtarget(n)/RREF(n)R_{\rm target}(n)/R_{\rm REF}(n) by IREF(n)/Itarget(n)I_{\rm REF}(n)/I_{\rm target}(n), with every prediction fixed before training. On two fresh heads per batch-size study, the 1.28M-update results are:

Batch sizePrediction errorBest of seven alternativesReduction
2560.0059690.02134572.0%
320.0062730.02129470.5%

In each study, all eight target/head combinations beat their own best alternative. This provides quantitative predictions of configuration changes at the same actual update budget.

The quantity Ix(n)I_x(n) accumulates the schedule-determined step-to-stabilizer ratio, while evaluation is performed at the actual update count nn. Log-ratio error measures accuracy on a relative loss scale. The two batch-size studies show that the same rule continues to predict the target's loss relative to the reference when the gradient-noise level changes.

3. How is the chosen step size checked?

Appendix D.2 gives initialization-computable sufficient tests for the full vector expansion on mixed-sign exponential-loss problems. The checks use the data, initial state, and declared optimizer parameters to verify moment-filter stability and descent conditions.

For all 34 original deterministic H3/H4 configurations, we additionally checked the original initialization, zero moments, and original common step η=0.004735687324502032\eta=0.004735687324502032. Rigorous enclosures of the first 40,000 updates place every run in a region proved invariant under that same step. Appendix D.4 supplies the resulting initialization-to-asymptotic applicability argument.

The finite-prefix calculation establishes entry from the stated initialization; invariance then controls all subsequent updates. Both parts use the actual experimental step.

The following sections give the requested absolute measurements and the complete Figure 1 construction.

4. What does the 0.306% change mean?

The diagnostic measures residual drift of the class-centered reference-minus-target parameter difference Dx(s)D_x(s):

\frac{\|D_x(1.25s)-D_x(s)\|_F}{\log 1.25}.$$ The absolute values are: | $(\beta_1,\beta_2)$ | H3 RATE | H3 TWO | H4 RATE | H4 TWO | |---|---:|---:|---:|---:| | (0.55, 0.965) | .083397 | .083182 | .090267 | .090065 | | (0.55, 0.985) | .085470 | .085262 | .091896 | .091698 | | (0.55, 0.9925) | .087756 | .087554 | .093835 | .093643 | | (0.55, 0.9975) | .093727 | .093578 | .099294 | .099152 | | (0.925, 0.965) | .071735 | .071493 | .080771 | .080552 | | (0.925, 0.985) | .071202 | .070962 | .080051 | .079834 | | (0.925, 0.9925) | .065910 | .065673 | .074630 | .074419 | | (0.925, 0.9975) | .021518 | .021344 | .024833 | .024696 | All 16 pairs improve, with absolute decreases of $1.37\times10^{-4}$ to $2.42\times10^{-4}$ and a mean relative decrease of 0.306%. At aligned time $s=400$, mean training CE changes from $1.218487007\times10^{-6}$ to $1.219517333\times10^{-6}$, and validation CE from **0.5358221313 to 0.5357983801** (RATE to TWO). Predicted classes coincide at $s=100,200,400$ on the original interpolation grid. Refining the grid changes one of 5,000 validation predictions for one configuration at one time, with TWO 0.02 percentage points lower in accuracy. Thus this experiment detects the predicted reduction in residual drift; its contribution is a mechanism check. The higher-order prediction ablations and full-network quality results in our responses above establish the additional finite-budget value of the analysis and calibration.

5. Figure 1: configuration, alignment, and panels.

All three runs use a single sample z=(1/3,1)z=(1/3,1) with weight one:

SettingREFRATETWO
Losse−z⊤we^{-z^\top w}samesame
Initial statew=m=v=0w=m=v=0samesame
Bias correctionboth momentsboth momentsboth moments
(β1,β2)(\beta_1,\beta_2)(0,0)(0,0)(0.5,0.25)(0.5,0.25)(0.5,0.25)(0.5,0.25)
η\eta.01.01.01
ϵ0\epsilon_0111
s0s_0101010
kk7/127/12.5856264389167615.5856264389167615
λ\lambda444.015832783340009
Updates10610^610610^610610^6

The denominator is v^+ϵ(S)\sqrt{\hat v}+\epsilon(S), where S=nηS=n\eta and ϵ(S)=ϵ0e−kS(1+S/s0)λ.\epsilon(S)=\epsilon_0e^{-kS}(1+S/s_0)^\lambda. For each configuration, define

\quad \sigma(k)=\frac{A_1(k)}{\sqrt{A_2(k)}}, \quad \chi(k)=1-k\partial_k\log\sigma(k).$$ The effective rate and shape are $r=k/\sigma$ and $\Lambda=\lambda\chi$. REF has $\sigma=\chi=1$. RATE solves $k/\sigma(k)=7/12$ while retaining $\lambda=4$, giving $\Lambda=3.9842296388$. TWO additionally sets $\lambda=4/\chi$, giving $\Lambda=4$. Only the displayed schedule parameters change. Here $\sigma$ summarizes the different responses of the first- and second-moment averages to the decaying gradient. The factor $\chi$ describes how the slower schedule modulation is transformed by that response. RATE first aligns the leading behavior; TWO then compensates the discrepancy that would otherwise continue to accumulate. We compute each run's clock directly from its configuration: $$s_x(n)=\sigma_x n\eta-\lambda_x\sigma'_x\log(1+n\eta/s_0).$$ Here $\sigma'_x$ differentiates with respect to $k_x$, holding memory and step fixed. Linear interpolation between consecutive clock nodes gives $W_x(s)$. We compare 64 logarithmically spaced points in the common interval, approximately $[0.0100123,10^4]$. A common effective-time value can therefore correspond to different update counts in different runs. Alignment consists of computing these clocks and interpolating on them. This makes the comparison reproducible directly from the declared configurations and recorded parameter iterates. The left panel plots $$\Delta_x(s)=\langle(-0.6,-0.8),W_{\rm REF}(s)-W_x(s)\rangle,$$ the parameter difference along the predicted RATE drift direction. RATE retains logarithmic growth; TWO cancels that growth and approaches a finite offset. The right panel plots the 63 adjacent log-time slopes $$q_{x,j}=\frac{\Delta_x(s_{j+1})-\Delta_x(s_j)} {\log s_{j+1}-\log s_j}$$ at $\sqrt{s_js_{j+1}}$. The theoretical horizontal levels are **0.0157703612 for RATE and 0 for TWO**. A zero slope means that persistent logarithmic drift has been removed. Together, the panels answer two questions: how much discrepancy has accumulated, and whether it is still growing on the logarithmic scale. TWO can have a nonzero constant level in the left panel while approaching zero slope in the right. That is the finite-offset trajectory relation characterized by the theory. We have incorporated this explanation into Figure 1 and Appendix C.1, presenting the configurations and alignment procedure before interpreting the two panels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.