Stored Sampling Noise Changes Recall Learning in AdamW

Adam and AdamW keep running averages of past gradients and of their squares, the first and second moments, so the sampling noise of one minibatch is not applied once but again and again over later updates. Common accounts describe minibatch noise by its size or covariance at each step. We ask whether, with both fixed, it matters how the stored noise lines up with the noise that each step applies. Associative recall lets us answer this. Because a context's queries can be enumerated, the gradient of one sampled query splits exactly into their mean and a residual, and we can choose what the first moment stores while every update still uses the sampled gradient. In a small transformer that plain AdamW leaves on a plateau, removing the residual from the first moment alone let recall form at default momentum. We then built pairs of stored copies with the same per-step covariance. A copy that mostly opposes the residual let 18 of 24 seeds learn recall, and its sign-flipped counterpart let 5 of the same 24, a gap that held in a model twice as deep and when the second moment ignored the residual's sign. A copy that cancels the residual over later updates adds noise at every step, yet with a short first-moment memory and matched early step sizes it led to recall sooner than storing only the mean. In a language model trained with loss on an artificially small subset of positions, flipping the stored residual's sign still lowered held-out loss, but storing only the mean did better at matched update size. What AdamW stores, and not only how much noise each step carries, can decide whether a small transformer learns in-context retrieval within the training budget we give it.