When Does Momentum Help Online Associative Memory? A Random-Query Risk Analysis

We study online associative memories through their retrieval error on queries that arrive at random times while key–value pairs stream in. The delta rule corrects along the arriving key, while heavy-ball momentum also carries decayed corrections from earlier keys. We ask when this accumulated correction lowers the expected retrieval error, and find that the answer depends first on when the memory updates. When momentum evolves continuously between key changes, we derive leading-order risks in closed form for two nearly collinear keys under Markov access and an exponential query time. From the weak direction, a fixed momentum rule beats every continuous-time delta rule with gain below one. An over-relaxed reflect–project schedule beats that momentum rule for every nonzero initial error, although momentum with large $\beta$ overtakes it from the weak direction. With each family tuned at a fixed key separation, continuous-time momentum beats the best constant gain found only for nearly identical keys (cosine above $0.98$ at every tested $\rho$) under reconstruction. For $K>2$ keys, a compatibility matrix that pairs the access modes with the keys' weak directions decides the comparison at small separation. Exact risks on frozen CIFAR-100 and Omniglot features reproduce these effects, and there the tuned delta rule selects a gain above one under reconstruction and below one under the associative query. With one update per token, the comparison reverses: tuned momentum attains lower mean risk than the tuned constant-gain delta rule in every tested synthetic and frozen-feature condition, by up to $0.034$ in normalized reconstruction risk.