acceptodds
Under review as a conference paper at ICLR 2027

Stored Sampling Noise Changes Recall Learning in AdamW

Abstract

Adam and AdamW keep running averages of past gradients and of their squares, the first and second moments, so the sampling noise of one minibatch is not applied once but again and again over later updates. Common accounts describe minibatch noise by its size or covariance at each step. We ask whether, with both fixed, it matters how the stored noise lines up with the noise that each step applies. Associative recall lets us answer this. Because a context's queries can be enumerated, the gradient of one sampled query splits exactly into their mean and a residual, and we can choose what the first moment stores while every update still uses the sampled gradient. In a small transformer that plain AdamW leaves on a plateau, removing the residual from the first moment alone let recall form at default momentum. We then built pairs of stored copies with the same per-step covariance. A copy that mostly opposes the residual let 18 of 24 seeds learn recall, and its sign-flipped counterpart let 5 of the same 24, a gap that held in a model twice as deep and when the second moment ignored the residual's sign. A copy that cancels the residual over later updates adds noise at every step, yet with a short first-moment memory and matched early step sizes it led to recall sooner than storing only the mean. In a language model trained with loss on an artificially small subset of positions, flipping the stored residual's sign still lowered held-out loss, but storing only the mean did better at matched update size. What AdamW stores, and not only how much noise each step carries, can decide whether a small transformer learns in-context retrieval within the training budget we give it.

Primary area:
interpretability and explainable AI
Keywords:
AdamW, Sampling Noise, Recall

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_VHGuOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks whether the minibatch noise that AdamW stores in its first moment affects learning beyond its per-step size or covariance. It studied this problem in associative recall, where the gradient can be split exactly into the mean over all queries and a residual term. In a 1.2M-parameter transformer, stored copies with the same per-step covariance lead to very different rates of learning recall, which the authors explain with a linear model of how a stored residual accumulates over updates. The results are also partially supported by a 29M-parameter language model trained with loss on a subset of positions.

Critical strengths
  1. The question the paper studies is interesting. The case study on the associative-recall experiments is also interesting and novel.
Critical weaknesses
  1. The paper is hard to follow. The writing feels very strange in many parts and seems to have been generated by LLMs without further polishing. The organization also needs improvement. For instance, the introduction contains many terms that are hard to understand without reading later sections, e.g., the masks’ “gains of −0.35 and 0.55” and “the residual’s net weight” (Sec. 1), while κ and the masks are defined only in Sec. 2.3–2.4. Also, the results in the paper come from multiple separate studies with different recipes, so readers need to refer to the Appendix to know which study produced a given number.

  2. The experimental evidence comes only from toy settings: associative recall with a 1.2M-parameter model, and a 29M-parameter language model trained with loss on an artificial subset of 64 of its 511 positions. The paper feels more like a case study to me, without proposing a general framework or method. For instance, the method in the paper needs the exact mean of the sampled gradient, which is not generally available in practice.

Questions for authors
  1. The analysis needs the exact mean of the sampled gradient, which is available only in associative recall and in a language model with an artificially subsampled loss. How could the approach and the findings generalize to other cases?

  2. Is there evidence that the finding holds in more realistic settings, for example, with larger models or standard language-model training? Without it, what should a reader take away from this case study for training or optimizer design in practice?

3 comments

Thank you for your careful review. We have tried to address each of your concerns, and we would be grateful if you could let us know whether any remain.

Weakness 1: presentation

We agree. Both terms you cite have short definitions, and the revision will give each where it first appears.

The long-run gain κ\kappa is what the introduction calls "the residual's net weight". In a linear model with a frozen preconditioner, it is the total weight with which a residual reaches the parameters, now and through its stored copy, in the direction of the residual.

RuleStored in mmκ\kappa at β1=0.9\beta_1 = 0.9
plain AdamWgˉ+r\bar{g} + r1
mean-onlygˉ\bar{g}0.1
reflectiongˉ−r\bar{g} - r−0.8-0.8
sign maskgˉ+D6/8r\bar{g} + D_{6/8} r−0.35-0.35
sign maskgˉ+D2/8r\bar{g} + D_{2/8} r0.55

The revision will describe κ\kappa in words at first mention, derive it before any result uses it, and map every result to its study, seeds and recipe. We will also rewrite the text throughout in plain, connected prose. These changes cover each problem you name.

Weakness 2: every seed set

The result replicated on every independent seed set at the reference shape, so it does not rest on one draw of seeds. Each pp is an exact one-sided paired sign test on the area under the accuracy curve.

Seed set, recall taskPairs with D6/8D_{6/8} ahead
mask study, 24 seeds21 of 24, p=1.4×10−4p = 1.4 \times 10^{-4}
matched vv, 16 new seeds16 of 16, p=1.5×10−5p = 1.5 \times 10^{-5}
β1=0.8,0.9,0.95\beta_1 = 0.8, 0.9, 0.95, 16 new seeds16, 16 and 14 of 16
β2=0.999\beta_2 = 0.999, 16 further seeds16 of 16, in speed only
eight blocks, 8 seeds, then 16 new seeds5 of 8, then 13 of 16, p=0.011p = 0.011

Every language-model seed set below also favored D6/8D_{6/8}. This replication, including 16 new seeds at eight blocks, is why we believe the result is sound.

Weakness 2 and Q1: generalizing without an exact mean

The approach is not limited to associative recall: the split applies to any minibatch training with a computed pool, and the memory it probes exists in every AdamW run.

Plain AdamW stores and reapplies each residual with long-run gain κ=1\kappa = 1, whether or not its mean is known. The exact mean is needed only to edit that memory. The revision will state that an exact mean exists whenever the applied batch is a uniformly drawn part of a computed pool, as in the language-model runs below. That pool costs four times the gradient work, so the split is a measuring framework, not a training method.

Without a pool, an estimate of the mean from kk extra draws stores its own noise, like 1/k1/\sqrt{k} times an independent query's residual in the linear model. In recall, a copy with the noise of four extra draws let only 4/8 runs learn, which is why we use the exact pool.

So any minibatch training with a computed pool can use the split.

Weakness 2 and Q2: evidence beyond the toy setting, and the takeaway

The finding also holds in standard language-model training, and we list three takeaways below. With loss at every position, the mask that helped recall gave lower held-out loss than the opposite mask in every seed at all three sizes we ran.

The sign mask D6/8D_{6/8} flips the stored residual's sign on six of eight random coordinate bins, and D2/8D_{2/8} on the other two. By construction, the two copies have identical per-step covariance. Each update applies one of four micro-batches drawn uniformly from a pool of 64 WikiText-103 sequences, whose mean is the exact mean of the applied gradient.

Language modelSeeds with D6/8D_{6/8} lowerMedian loss gap (nats)
29M, the paper's8/80.050
29M, update norms matched8/80.0095
162.7M, GPT-2 small shape8/80.072
162.7M, update norms matched8/80.069
405.9M, GPT-2 medium block shape, 4,000 updates6/60.086

Larger steps do not explain the gap at the GPT-2 small shape: 96.5% of it remained with update norms matched, and D6/8D_{6/8} reached D2/8D_{2/8}'s final loss with about 16% fewer updates. At 29M, larger steps explain most of the gap.

(1) Per-step summaries of minibatch noise are incomplete. Two stored copies with identical per-step covariance led to different recall in the toy task and different held-out loss in language models. At the GPT-2 small shape, D6/8D_{6/8} closed about a third of the gap between D2/8D_{2/8} and storing only the mean.

(2) Momentum does not cancel stored noise. A larger β1\beta_1 lowers per-step noise but keeps the residual's long-run gain at 1. This fits the runs at β1=0.95\beta_1 = 0.95, where plain AdamW stalled but every run storing only the mean learned.

(3) Where an exact mean exists, averaging one source at a time shows where noise acts. In recall, averaging only mm let every run learn, and averaging only the update let none.

So the finding holds beyond the toy setting, and it names a variable that analyses of training noise leave out.

Thanks for the response and the new language-model results. However, the presentation issues have not yet been addressed in the paper, and the experiments still rely on an artificial intervention using an exact-mean pool. As the authors note, the approach is a measuring framework rather than a method, so its practical value remains unclear to me. I will keep my score.

Thank you for reading our response and considering the additional language-model results. We have uploaded a revised manuscript that addresses the three points you raised.

Presentation

The introduction now defines the sampling residual and explains which signs each mask reverses. It then follows one residual through the current and later updates to explain its long-run gain before introducing the notation in Section 3. Table 1 shows what each rule stores in the first moment and gives its corresponding gain. Related work now follows the introduction, and Table 9 reports the number of seeds and the training settings that varied across studies.

The exact-mean pool

In the language-model experiments, we compute gradients for four micro-batches, with supervision at every token position. We select one micro-batch uniformly for the current step. The average of the four gradients is therefore the exact conditional mean of the selected gradient, and subtracting this mean gives its sampling residual.

At the same model and optimizer state, the two mask conditions use the same current gradient for a given draw and the same second-moment rule. They differ only in which signs of the residual enter the first moment. Plain AdamW already carries that residual into later updates. Computing the pool mean lets us isolate and change the stored copy.

Practical value

Section 5 presents the intervention as a diagnostic. The four-part pool costs four gradient evaluations per update. That cost makes it possible to distinguish the effect of noise in the current gradient from the effect of carrying that noise forward in memory.

ComparisonResult in the revised manuscript
Averaging over queries in the current step or in the first momentAll eight runs learned recall when only the first moment received the mean gradient. None learned when only the current step used the mean and both moments retained the sampled gradient (Table 2).
Changing the stored signs at equal conditional covarianceThe opposing mask yielded recall in 18 of 24 runs, compared with 5 of 24 for the agreeing mask (Table 3). At the same learning rate, the opposing mask also gave lower final held-out loss in all eight pairs at each model size, 29M and 162.7M (Table 5).
Reaching the same held-out lossAt 162.7M and the same learning rate, the opposing mask reached the agreeing mask’s final loss in about 16% fewer updates (Section 4.5).

At 162.7M, nearly all of the final loss gap also persisted when we matched median update norms (Table 5). The two masks stored noise with the same covariance at any given state, yet changing how the stored copy aligned with the original draw affected learning even at comparable step sizes.

Reviewer_nPAqOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how sampling noise stored in AdamW's first and second moments affects the emergence of associative recall in Transformers. By decomposing sampled gradients into their exact conditional mean and a sampling residual, the authors independently manipulate the gradients used for current updates and stored optimizer states. They introduce a long-run gain analysis showing that residuals with identical per-step covariance can have different accumulated effects due to temporal correlations. Experiments on synthetic associative recall and language modeling demonstrate that changing the stored residual can substantially affect learning dynamics, even when its covariance is preserved.

Critical strengths
  1. The separation of current and stored gradient sources provides a useful experimental framework for studying optimizer memory. The complementary sign-mask experiments are particularly compelling, demonstrating substantial differences in recall acquisition despite matched conditional covariance and coordinate-wise noise magnitudes.

  2. The theoretical analysis provides a plausible explanation based on temporal noise accumulation. The comparison between reflected and mean-only residuals is especially interesting, since reflection can accelerate recall despite introducing greater per-step noise. The authors also provide extensive ablations and acknowledge important limitations of their analysis.

Critical weaknesses
  1. The proposed long-run gain does not fully explain the experimental outcomes. The analysis assumes a fixed preconditioner and ignores feedback through the evolving model parameters. In actual AdamW, complementary stored residuals can also change the interaction between first and second moments. The matched second-moment experiments are useful, but higher-order dependencies remain.

  2. The strongest results depend on a small synthetic recall task and a particular optimization regime. The effect becomes weaker or disappears under several changes in model width, normalization, and optimizer configuration. The language-modeling experiments use artificially subsampled token supervision, and reflection no longer outperforms mean-only storage when update norms are matched. Consequently, it remains unclear how important the proposed phenomenon is for standard large-scale language-model training.

Questions for authors
  1. Could the authors directly measure the temporal covariance of the effective preconditioned AdamW updates and compare it with the predictions of the long-run gain analysis? This could help distinguish temporal cancellation from effects arising through the interaction of the first and second moments.

  2. How much of the effect depends on the specific optimization regime in which ordinary AdamW fails to acquire recall? Could the authors clarify whether stored-noise alignment remains consequential when the baseline learns reliably?

  3. The experiments rely on computing the exact conditional mean gradient, which is feasible in the synthetic task but expensive in realistic training. Is there a computationally practical approximation that preserves the effect? Could the authors further investigate the phenomenon under conventional language-model training without artificially subsampled token losses?

4 comments

Thank you for your close reading and precise comments.

W1 and Q1: the long-run gain and the temporal covariance of actual updates

Actual preconditioned AdamW updates show the cancellation over time that the long-run gain describes, and the mask contrast does not rest on the coupling of mm and vv.

We froze θ\theta at four early checkpoints of β1=0.5\beta_1 = 0.5 runs, so there is no feedback. For each stored-mm rule, 32 replicas with different query draws ran 16 AdamW updates, with mm and vv updating. The vv input was the sampled square or, for the matched vv, gˉ2+r2\bar{g}^2 + r^2, which drops the 2gˉr2\bar{g}r term. Noise is the variance across replicas, and κeff2\kappa^2_{\rm eff} is the growth of summed noise over the last 8 updates.

Stored mmNoise per updateNoise over 16 updatesκeff2\kappa^2_{\rm eff}κ2\kappa^2, linear model
sampled, gˉ+r\bar{g} + r1111
mean-only, gˉ\bar{g}0.74 to 0.770.26 to 0.280.23 to 0.260.25
reflected, gˉ−r\bar{g} - r0.99 to 1.050.029 to 0.0400.008 to 0.0270

Noise over 16 updates ranks the rules in the order their runs formed recall, reflected first and sampled last, while noise per update does not. The 2gˉr2\bar{g}r term changed κeff2\kappa^2_{\rm eff} by at most 0.025, far below the gaps between rules. Over 128 updates the ranking held, though reflection cancelled less, with ∣κeff∣|\kappa_{\rm eff}| up to 0.36.

The mask contrast needs no linear model and survives the couplings you name. The sign mask D6/8D_{6/8} flips the stored rr on six of eight random coordinate bins, and D2/8D_{2/8} on the other two. With the matched vv, D6/8D_{6/8} was ahead in 16/16 recall pairs. A coupling through E[r3]\mathbb{E}[r^3] remains in recall. It vanishes when a language-model update applies one of two micro-batches from a computed pool, since the matched vv is then the same for both. There, with equal update norms, D6/8D_{6/8} still gave the paper's model lower held-out loss in 7 of 8 new seeds, by 0.005 nats.

So the cancellation appears in actual AdamW updates, and the mask contrast does not rest on the coupling of mm and vv.

W2 and Q3: conventional training without subsampled losses, and the exact mean

The phenomenon holds in conventional language-model training, where the exact mean costs four times the gradient work: with loss at every position, D6/8D_{6/8} ended with lower held-out loss than D2/8D_{2/8} in every seed at all three sizes.

Each update applied one of four micro-batches drawn uniformly from a pool of 64 WikiText-103 sequences. The revision will state that such a pool always gives the exact conditional mean.

Language modelSeeds with D6/8D_{6/8} lowerMedian loss gap (nats)
29M, the paper's8/80.050
29M, update norms matched8/80.0095
162.7M, GPT-2 small shape8/80.072
162.7M, update norms matched8/80.069
405.9M, GPT-2 medium block shape, 4,000 updates6/60.086

Larger steps do not explain the GPT-2 small gap: 96.5% of it remained with update norms matched. They explain most of the 29M gap.

No cheaper approximation of the exact mean has been validated. An estimate of the mean from kk extra draws stores its own noise, like 1/k1/\sqrt{k} times an independent query's residual in the linear model. In recall at a fixed long-run gain, the noise of four extra draws let only 4/8 runs learn, and more noise let fewer.

The effect needs no subsampled loss, and conventional training supplies the exact mean at that fourfold cost.

W2 and Q2: when plain AdamW learns reliably

Alignment still matters where training does not stall: in the language models no run stalled, yet the mask contrast held in every seed at every size. It also held at three β1\beta_1 values on 16 new seeds.

In recall, plain AdamW learned reliably with Q/K normalization or at width 640, and there every stored copy formed recall in every run. So a failing regime is needed only to see the effect in recall.

W2: reflection at matched update norms

The reflection result limits the long-run account, not the finding that the stored residual matters: reflection still beat sampled mm with smaller update norms. We agree that mean-only mm beats it at matched update norms in language models, as our 29M runs showed again:

29M, β1=0.5\beta_1 = 0.5Lower arm, seedsMedian gap (nats)
reflected vs sampled mm, same learning ratereflected, 8/80.118
reflected at 0.845×0.845 \times learning rate vs sampled mmreflected, 7/80.015
the same vs mean-only mmmean-only, 8/80.061

At that rate, reflection's update norms matched mean-only mm's and were 0.93 times sampled mm's. The linear model then gives reflection about 5% less noise per step than mean-only mm, and the revision will state this. So neither account predicts mean-only mm's lead. This limits the long-run account, while reflection still led sampled mm with smaller update norms.

Thank you again for your close reading and for asking whether the cancellation predicted by the linear model can be observed in actual AdamW updates. We have uploaded a revised manuscript with this measurement in Section 4.4. We held the parameters fixed and continued updating both AdamW moments. Reflection left the variance of a single update close to that under sampled first-moment storage, but the variance of the sum of 16 updates was only 3–4% as large (Table 4 and Appendix I).

The revision also clarifies the exact-mean construction and the second-moment controls. Section 3.1 explains why the pool average is the exact conditional mean when one of its equal-sized micro-batches is sampled uniformly. In recall, the opposing mask gave higher area under the accuracy curve in all 16 pairs with a matched second moment (Section 4.3). In the two-micro-batch language-model control, the second-moment input was independent of which micro-batch was sampled, and the opposing mask gave lower loss in seven of eight pairs (Section 4.5 and Appendix K.4).

Finally, Section 4.5 distinguishes the prediction at matched update norms from the observed loss. The linear model predicts about 5% lower per-step update variance under reflection than under mean-only storage, yet mean-only storage gave lower held-out loss in all eight pairs (Table 24 and Appendix K.5). We would welcome your assessment of whether these measurements resolve the questions you raised.

I thank authors for the responses. The responses answer my questions and I raise my score.

Thank you for your careful reading and for suggesting that we test the predicted cancellation directly in AdamW updates. We appreciate your taking the time to consider our response.

Reviewer_t5SeOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores how the accumulation of minibatch sampling noise affects recall learning in Adam / AdamW. While previous work examines the effect of noise on learning at any given step in isolation, this paper draws attention to the accumulation of noise across training when using optimizers like AdamW that average the noise cumulatively.

For this, the authors use an associative recall task to compute the gradient at each step which is expressed as a combination of that minibatch's mean gradient gˉ\bar{g} and a residual rr. Standard AdamW stores the sum gˉ+r\bar{g} + r in its first moment. The authors argue that storing the difference gˉ−r\bar{g} - r instead, for a majority of the queried samples (but not all) across the minibatch leads to more effective learning of the associate recall task (18/24 seeds) compared to its sign-flipped version (5/24 seeds). In other words, when the current step noise aligns with the stored noise, it can lead to accumulation of noise, pushing the model into a plateau, whereas removing the noise can accelerate learning of the signal.

In conclusion, the sample-efficiency of small transformers on learning in-context retrieval can depend not only on how much noise any single step adds, but also on how the optimizer stores that noise across training steps.

Critical strengths

S1. Insightful findings: The paper provides an in-depth treatment of how optimizers like Adam can potentially benefit from a slight reformulation in objective when standard Adam does not work on certain tasks.

S2. Experimental design: The matched experimental design and clever formulation of the associative recall task provide a convincing story for understanding the effects of stored noise in learning similar tasks.

Critical weaknesses

W1. Limited hyperparameter sweep: The β1,β2\beta_1, \beta_2 hyperparameters are limited and do not show a sweep across parameters.

W2. Assumptions regarding the base task setup: The experiments are limited to tasks that do not perform well with standard AdamW and this would not be known prior to training. In addition, QK norm, Lion, and Muon have eliminated the need for it in the first place.

Questions for authors

Not so much of a question, but a suggestion for future iterations. It would be interesting to see if the noise removal trick can help or hurt training techniques like curriculum learning which implicitly rely on the degree of sample noise to determine the order in which samples are learned.

3 comments

Thank you for your thoughtful review. We would be glad if the β1\beta_1 sweep and the standard training runs below address your concerns and questions.

Weakness 1: hyperparameter sweep

The results do not hinge on one β1\beta_1 or β2\beta_2: on 16 new seeds, the mask contrast held at each of three β1\beta_1 values.

A sampled recall gradient splits into its exact mean gˉ\bar{g} and a residual rr. The sign mask D6/8D_{6/8} flips the sign of the rr stored in AdamW's first moment mm on six of eight random coordinate bins, so its copy mostly opposes rr. The mask D2/8D_{2/8} flips the other two, so its copy mostly agrees with rr. Both copies have identical per-step covariance by construction, so a per-step account cannot explain a difference between them.

β1\beta_1, with β2=0.95\beta_2 = 0.95Pairs with D6/8D_{6/8} aheadRuns forming recall, D6/8D_{6/8} vs D2/8D_{2/8}pp
0.816/1616/16 vs 0/161.5×10−51.5 \times 10^{-5}
0.9, as in the paper16/1615/16 vs 4/161.5×10−51.5 \times 10^{-5}
0.9514/1613/16 vs 7/160.0021

Each pp is an exact one-sided paired sign test on the area under the accuracy curve. At β1=0.8\beta_1 = 0.8, D6/8D_{6/8} formed recall in every run, while D2/8D_{2/8} and plain AdamW formed it in none. Early steps under D6/8D_{6/8} were a median 1.08 times those under D2/8D_{2/8}. That seems unlikely to explain the contrast, since plain AdamW stalled even at double the learning rate. In the paper, at β2=0.999\beta_2 = 0.999, both masks formed recall in every run, and the contrast remained as a small speed difference in D6/8D_{6/8}'s favor.

The stall itself is not an artifact of tuning either. Earlier runs changed one setting at a time from the reference recipe:

SettingPlain AdamWStoring only gˉ\bar{g} in mm
reference, β1=0.9\beta_1 = 0.9 and β2=0.95\beta_2 = 0.950/88/8
β1=0.95\beta_1 = 0.950/88/8
β2=0.999\beta_2 = 0.9990/88/8
cosine schedule0/88/8

Plain AdamW also stalled at half the learning rate and at β1=0.97\beta_1 = 0.97. With β2=0.999\beta_2 = 0.999, it stalled at β1=0.99\beta_1 = 0.99 as well. The revision will add both tables. So neither the stall nor the mask contrast hinges on one setting of β1\beta_1 or β2\beta_2.

Weakness 2-1: tasks where plain AdamW fails

The effect is not confined to tasks where AdamW fails, and the split can be used without knowing the regime in advance. In standard language-model training, where no run stalled, D6/8D_{6/8} ended with lower held-out loss than D2/8D_{2/8} in every seed at all three sizes.

Each update applied one of four micro-batches drawn uniformly from a pool of 64 WikiText-103 sequences, so the pool mean is the exact mean of the applied gradient. Held-out loss on the main 1,024-sequence set fell at every evaluation in every run.

Language modelSeeds with D6/8D_{6/8} lowerMedian loss gap (nats)
29M, the paper's8/80.050
29M, update norms matched8/80.0095
162.7M, GPT-2 small shape8/80.072
162.7M, update norms matched8/80.069
405.9M, GPT-2 medium block shape, 4,000 updates6/60.086

To match update norms, we lowered D6/8D_{6/8}'s learning rate.

Larger steps do not explain the gap at the GPT-2 small shape: 96.5% of it remained with update norms matched, and D6/8D_{6/8} reached D2/8D_{2/8}'s final loss with about 16% fewer updates. At 29M, larger steps explain most of the gap.

We use the edited copies only for matched comparisons and do not propose them as a change to Adam, so the regime does not need to be known before training.

So the effect does not require a failing task, and the split applies whether or not AdamW stalls.

Weakness 2-2: Q/K norm, Lion and Muon

Q/K normalization, Lion and Muon remove the stall, not the stored noise, so they leave the paper's question open. As the paper reports, all three let recall form on the reference task. Yet momentum methods still store each residual and apply it again later. In our linear model, SGD with momentum, and Lion before its sign is taken, give each residual the same long-run weight as the mean. Lion's sign and Muon's orthogonalization lie outside that model, and we have not yet tested the masks under them.

These methods fix the stall on our task, but momentum optimizers still carry the stored noise that our edits probe.

Question: curriculum learning

The split gives a way to test this, because it separates a curriculum's order of examples from the noise AdamW stores. Our edits change only how the sampled query's noise is stored within each context, so the order of examples is unchanged. Training a curriculum with and without the stored residual would show whether the curriculum acts through that noise. Data order can itself act through AdamW's memory (Sweeney, 2026). We did not study curricula, and the revision will name them as an open question that the split can test directly.

The concerns have been addressed and I have revised my score in light of this.

Thank you for revisiting the paper. Your questions helped us clarify how the recall results depend on the training setting.

Reviewer_UEq3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This study finds that the way sampling noise is stored in AdamW's momentum impacts a model's ability to learn. The authors find that in a synthetic associative recall task, reducing the influence of stored sampling noise improves recall learning, while experiments on language modeling show that the effects depend on how the noise is modified.

Critical strengths

This is a potentially interesting problem.

Critical weaknesses

This setup is extremely toy. The task is very constrained, and in real language model settings (which are still with very small models), the results are also unclear and don't support the findings from the synthetic setup. Therefore, it doesn't seem probable that the findings from this paper are generalizable.

Questions for authors
  1. How does this scale with model size?
2 comments

Thank you for your constructive review. We hope that the language-model results below, at three model sizes, may help address your concern about whether the findings extend beyond the toy setup.

Weakness 1: what is new and presentation

The paper isolates a variable that per-step accounts of minibatch noise miss: how the noise AdamW stores lines up with the residual it copies. To our knowledge, no prior work edits only AdamW's stored noise while the applied gradient, its mean and its per-step covariance stay as drawn. With this edit, the variable changed the outcome in recall, on new seeds at three β1\beta_1 values, and in standard training with update norms matched. What the paper adds is a controlled test showing that the direction of stored noise, at fixed size and covariance, changes learning.

We agree that the presentation needs work. The revision will define each term before its first use and map every result to its study, seeds and recipe, so that each result can be traced from the main text. These are changes to the writing only, and no result changes.

Weakness 1-1 and Q1: evidence beyond the toy task, and model size

The finding is not limited to the toy task or to one model size. In standard language-model training, the mask that helped recall also gave lower held-out loss than the opposite mask in every seed at all three sizes we ran.

In associative recall, a sampled gradient splits exactly into its mean gˉ\bar{g} and a residual rr. The sign mask D6/8D_{6/8} flips the sign of the rr stored in AdamW's first moment mm on six of eight random coordinate bins, so the stored copy mostly opposes rr. The mask D2/8D_{2/8} flips the other two, so its copy mostly agrees with rr. Both copies have identical per-step covariance by construction, so a per-step account cannot explain a difference between them.

The split stays exact in standard language-model training. Each update applies one of four micro-batches drawn uniformly from a pool of 64 WikiText-103 sequences, with loss at every position, so the pool mean is the exact mean of the applied gradient.

ModelParametersD6/8D_{6/8} vs D2/8D_{2/8}
recall, reference shape1.24Mrecall in 18/24 vs 5/24 runs
recall, eight blocks, new seeds2.5Mrecall in 11/16 vs 1/16 runs
recall, width 3204.9Mrecall in 8/8 vs 8/8 runs
recall, width 64019.7Mrecall in 8/8 vs 8/8 runs
LM, paper's model29Mlower loss in 8/8 seeds, by 0.050 nats
LM, paper's model, update norms matched29Mlower loss in 8/8 seeds, by 0.0095 nats
LM, GPT-2 small shape162.7Mlower loss in 8/8 seeds, by 0.072 nats
LM, GPT-2 small shape, update norms matched162.7Mlower loss in 8/8 seeds, by 0.069 nats
LM, GPT-2 medium block shape, 4,000 updates405.9Mlower loss in 6/6 seeds, by 0.086 nats

To match update norms, we lowered D6/8D_{6/8}'s learning rate.

Larger steps do not explain the gap at the GPT-2 small shape, since 96.5% of it remained with update norms matched. There, D6/8D_{6/8} reached D2/8D_{2/8}'s final loss with about 16% fewer updates. At 29M, larger steps explain most of the gap.

In recall, the masks change how long recall takes to form, so they matter where it forms slowly. At widths 320 and 640, every mask run formed recall early, leaving little delay to change.

We read the language-model sizes as separate tests, not as a trend. At every one of them, the stored copy's alignment changed held-out loss in every seed, so the finding is not limited to the toy task or to one model size.

Weakness 1-2: do the language-model results support the synthetic finding?

We think they do. Two stored copies with identical per-step covariance led to different outcomes. At the GPT-2 small shape, three arms that differ only in what mm stores ended in the same order in every seed:

Held-out loss at update 10,000, median (nats)mm stores only gˉ\bar{g}D6/8D_{6/8}D2/8D_{2/8}
GPT-2 small3.583.723.79

The mask D6/8D_{6/8} closed about a third of the gap between D2/8D_{2/8} and storing only the mean. At every checkpoint from update 2,000 on, it was ahead in every 64-token position bin of the context, in every seed. The gain from a token's earlier occurrence in the context was also larger under D6/8D_{6/8} in every seed at both larger sizes, on WikiText-103 and on FineWeb-Edu.

Stored noise also mattered more than applied noise in the paper's own language model. In both replications, sampling the stored moments instead of averaging them delayed the onset of induction-style heads about twofold in every pair. Changing the source of the update itself did not shift the onset. In recall, likewise, averaging only mm let every run learn, and averaging only the update let none.

In recall and in the language models alike, what AdamW stores, beyond the noise each step applies, changed how the models learned.

Thank you again for asking how the recall findings carry over to language modeling and how they vary with model size. We have uploaded a revised manuscript that addresses both questions in Section 4.5.

At the same learning rate, the opposing mask gave lower held-out loss than the agreeing mask in every paired run at 29M and 162.7M parameters (Table 5), as well as in the shorter 405.9M study (Table 23). Matching median update norms closed most of the gap at 29M but left nearly all of it at 162.7M. Section 4.5 also reports the three-way comparison at 162.7M, where storing only the mean in the first moment gave lower loss than either mask. Appendix K.2 examines predictions for tokens that appeared earlier in the same context. At both larger model sizes, the opposing mask showed a greater benefit from token reuse on both WikiText-103 and FineWeb-Edu.

To make these comparisons easier to follow, the introduction now explains the sampling residual, the two masks, and the long-run gain before Section 3 introduces the notation. Table 9 reports the seed counts and training settings for each study. We would welcome your assessment of whether the revision addresses your concerns.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for overseeing the review of our paper. We appreciate the reviewers’ careful reading and have summarized the main questions, our findings, and the relevant passages in the revised manuscript below.

QuestionReviewer(s)Evidence and clarificationRevised Location
How was the presentation improved?VHGu, UEq3The introduction now explains in words how noise from one sampled gradient affects the current update and, through AdamW’s first moment, later updates. Table 1 shows what each condition stores and gives its predicted long-run gain.Section 1, Table 1
What does the exact-mean pool reveal about ordinary AdamW?VHGu, nPAqOrdinary AdamW already carries sampling noise into later updates. Averaging the gradients over the pool gives us the exact conditional mean, so we can change the stored noise while applying the same sampled gradient at a given state. In the main language-model experiments, this measurement costs four gradient evaluations per update.Sections 3.1 and 5
How sensitive is the recall result to the training setting?t5Se, nPAqThe opposing mask reverses the stored noise on six of eight coordinate groups. The agreeing mask reverses it on the other two, giving the same per-step covariance. By update 8,000, 18 of 24 runs reached 95% recall accuracy with the opposing mask, compared with 5 of 24 with the agreeing mask. The opposing mask also gave higher recall accuracy integrated over training in most pairs at each of three first-moment decay rates. At larger widths, both masks learned recall at nearly the same time.Sections 4.3 and 5, Tables 3 and 13
Can cancellation be measured directly, and could the second moment explain the mask comparison?nPAqWith parameters fixed and both moments updating, reversing all signs of the stored noise left single-update variance nearly unchanged. The variance of the sum of 16 updates fell to 3–4% of that under sampled storage. In a separate language-model control, either micro-batch gave the same second-moment input, yet the opposing mask gave lower loss in 7 of 8 pairs.Section 4.4, Table 4, Appendix K.4
Does the effect extend beyond recall when training proceeds without a stall?UEq3, VHGu, nPAq, t5SeOn WikiText-103 with loss at every token position, both masks learned. At the same learning rate, the opposing mask gave lower final held-out loss in every pair at 29M and 162.7M parameters and in the shorter 405.9M study. Matching median update norms closed most of the gap at 29M and left nearly all of it at 162.7M.Section 4.5, Tables 5 and 23

At 162.7M, the opposing mask reached the agreeing mask’s final loss in about 16% fewer updates at the same learning rate, taking the median across paired runs. Storing only the mean gradient in the first moment gave the lowest final loss of the three conditions (Section 4.5 and Appendix K.2).

We would be grateful if you would consider these findings in your discussion with the reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.