Delta-Rule Memories Store More Than They Recall, and a Load Curriculum Closes the Gap

Delta-rule layers are now common in open frontier models, where instead of a growing KV cache (as in attention), they keep a fixed $d_k\times d_k$ matrix per head and use it as an associative memory. This rank $d_k$ matrix can return at most $d_k$ exact associations, but a language model only needs the correct token to win the argmax, and recent theory shows that an offline memory then returns the right token for $n$ associations whenever $d_k^2>2n\ln n$. We ask how many associations these models recall in practice, in two-layer, single-head DeltaNet models trained on multi-query associative recall. Using state reads and edits built from the model's own primitives, we identify a circuit in which the first layer carries each key token to its paired value and the second layer stores the associations. We then distinguish failures of writing from failures of retrieval. In width-128 models trained on 64 pairs, recall falls to $16\%$ at a load of $4d_k$. Reading the same state with each association's write key recovers $50\%$, and a matrix built by ridge regression from the same keys and values raises this to $83\%$, so both online writing and retrieval limit recall, and many associations remain accessible after the model fails to recall them. Finally, a training curriculum that progressively increases the number of pairs (the load) raises the load at which the model reaches $90\%$ recall from $1.7d_k$ to $3.6d_k$ at $d_k=128$, and reaches it faster than training directly at the target load. Effective recall capacity therefore depends on the learned writing and retrieval procedures and can be increased through training without enlarging the memory state.