- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper asks whether the minibatch noise that AdamW stores in its first moment affects learning beyond its per-step size or covariance. It studied this problem in associative recall, where the gradient can be split exactly into the mean over all queries and a residual term. In a 1.2M-parameter transformer, stored copies with the same per-step covariance lead to very different rates of learning recall, which the authors explain with a linear model of how a stored residual accumulates over updates. The results are also partially supported by a 29M-parameter language model trained with loss on a subset of positions.
Critical strengths
- The question the paper studies is interesting. The case study on the associative-recall experiments is also interesting and novel.
Critical weaknesses
-
The paper is hard to follow. The writing feels very strange in many parts and seems to have been generated by LLMs without further polishing. The organization also needs improvement. For instance, the introduction contains many terms that are hard to understand without reading later sections, e.g., the masks’ “gains of −0.35 and 0.55” and “the residual’s net weight” (Sec. 1), while κ and the masks are defined only in Sec. 2.3–2.4. Also, the results in the paper come from multiple separate studies with different recipes, so readers need to refer to the Appendix to know which study produced a given number.
-
The experimental evidence comes only from toy settings: associative recall with a 1.2M-parameter model, and a 29M-parameter language model trained with loss on an artificial subset of 64 of its 511 positions. The paper feels more like a case study to me, without proposing a general framework or method. For instance, the method in the paper needs the exact mean of the sampled gradient, which is not generally available in practice.
Questions for authors
-
The analysis needs the exact mean of the sampled gradient, which is available only in associative recall and in a language model with an artificially subsampled loss. How could the approach and the findings generalize to other cases?
-
Is there evidence that the finding holds in more realistic settings, for example, with larger models or standard language-model training? Without it, what should a reader take away from this case study for training or optimizer design in practice?
3 comments
Thanks for the response and the new language-model results. However, the presentation issues have not yet been addressed in the paper, and the experiments still rely on an artificial intervention using an exact-mean pool. As the authors note, the approach is a measuring framework rather than a method, so its practical value remains unclear to me. I will keep my score.
Thank you for reading our response and considering the additional language-model results. We have uploaded a revised manuscript that addresses the three points you raised.
Presentation
The introduction now defines the sampling residual and explains which signs each mask reverses. It then follows one residual through the current and later updates to explain its long-run gain before introducing the notation in Section 3. Table 1 shows what each rule stores in the first moment and gives its corresponding gain. Related work now follows the introduction, and Table 9 reports the number of seeds and the training settings that varied across studies.
The exact-mean pool
In the language-model experiments, we compute gradients for four micro-batches, with supervision at every token position. We select one micro-batch uniformly for the current step. The average of the four gradients is therefore the exact conditional mean of the selected gradient, and subtracting this mean gives its sampling residual.
At the same model and optimizer state, the two mask conditions use the same current gradient for a given draw and the same second-moment rule. They differ only in which signs of the residual enter the first moment. Plain AdamW already carries that residual into later updates. Computing the pool mean lets us isolate and change the stored copy.
Practical value
Section 5 presents the intervention as a diagnostic. The four-part pool costs four gradient evaluations per update. That cost makes it possible to distinguish the effect of noise in the current gradient from the effect of carrying that noise forward in memory.
| Comparison | Result in the revised manuscript |
|---|---|
| Averaging over queries in the current step or in the first moment | All eight runs learned recall when only the first moment received the mean gradient. None learned when only the current step used the mean and both moments retained the sampled gradient (Table 2). |
| Changing the stored signs at equal conditional covariance | The opposing mask yielded recall in 18 of 24 runs, compared with 5 of 24 for the agreeing mask (Table 3). At the same learning rate, the opposing mask also gave lower final held-out loss in all eight pairs at each model size, 29M and 162.7M (Table 5). |
| Reaching the same held-out loss | At 162.7M and the same learning rate, the opposing mask reached the agreeing mask’s final loss in about 16% fewer updates (Section 4.5). |
At 162.7M, nearly all of the final loss gap also persisted when we matched median update norms (Table 5). The two masks stored noise with the same covariance at any given state, yet changing how the stored copy aligned with the original draw affected learning even at comparable step sizes.
Thank you for your careful review. We have tried to address each of your concerns, and we would be grateful if you could let us know whether any remain.
Weakness 1: presentation
We agree. Both terms you cite have short definitions, and the revision will give each where it first appears.
The long-run gain κ is what the introduction calls "the residual's net weight". In a linear model with a frozen preconditioner, it is the total weight with which a residual reaches the parameters, now and through its stored copy, in the direction of the residual.
The revision will describe κ in words at first mention, derive it before any result uses it, and map every result to its study, seeds and recipe. We will also rewrite the text throughout in plain, connected prose. These changes cover each problem you name.
Weakness 2: every seed set
The result replicated on every independent seed set at the reference shape, so it does not rest on one draw of seeds. Each p is an exact one-sided paired sign test on the area under the accuracy curve.
Every language-model seed set below also favored D6/8. This replication, including 16 new seeds at eight blocks, is why we believe the result is sound.
Weakness 2 and Q1: generalizing without an exact mean
The approach is not limited to associative recall: the split applies to any minibatch training with a computed pool, and the memory it probes exists in every AdamW run.
Plain AdamW stores and reapplies each residual with long-run gain κ=1, whether or not its mean is known. The exact mean is needed only to edit that memory. The revision will state that an exact mean exists whenever the applied batch is a uniformly drawn part of a computed pool, as in the language-model runs below. That pool costs four times the gradient work, so the split is a measuring framework, not a training method.
Without a pool, an estimate of the mean from k extra draws stores its own noise, like 1/k times an independent query's residual in the linear model. In recall, a copy with the noise of four extra draws let only 4/8 runs learn, which is why we use the exact pool.
So any minibatch training with a computed pool can use the split.
Weakness 2 and Q2: evidence beyond the toy setting, and the takeaway
The finding also holds in standard language-model training, and we list three takeaways below. With loss at every position, the mask that helped recall gave lower held-out loss than the opposite mask in every seed at all three sizes we ran.
The sign mask D6/8 flips the stored residual's sign on six of eight random coordinate bins, and D2/8 on the other two. By construction, the two copies have identical per-step covariance. Each update applies one of four micro-batches drawn uniformly from a pool of 64 WikiText-103 sequences, whose mean is the exact mean of the applied gradient.
Larger steps do not explain the gap at the GPT-2 small shape: 96.5% of it remained with update norms matched, and D6/8 reached D2/8's final loss with about 16% fewer updates. At 29M, larger steps explain most of the gap.
(1) Per-step summaries of minibatch noise are incomplete. Two stored copies with identical per-step covariance led to different recall in the toy task and different held-out loss in language models. At the GPT-2 small shape, D6/8 closed about a third of the gap between D2/8 and storing only the mean.
(2) Momentum does not cancel stored noise. A larger β1 lowers per-step noise but keeps the residual's long-run gain at 1. This fits the runs at β1=0.95, where plain AdamW stalled but every run storing only the mean learned.
(3) Where an exact mean exists, averaging one source at a time shows where noise acts. In recall, averaging only m let every run learn, and averaging only the update let none.
So the finding holds beyond the toy setting, and it names a variable that analyses of training noise leave out.