acceptodds
Under review as a conference paper at ICLR 2027

Delta-Rule Memories Store More Than They Recall, and a Load Curriculum Closes the Gap

Abstract

Delta-rule layers are now common in open frontier models, where instead of a growing KV cache (as in attention), they keep a fixed matrix per head and use it as an associative memory. This rank matrix can return at most exact associations, but a language model only needs the correct token to win the argmax, and recent theory shows that an offline memory then returns the right token for associations whenever . We ask how many associations these models recall in practice, in two-layer, single-head DeltaNet models trained on multi-query associative recall. Using state reads and edits built from the model's own primitives, we identify a circuit in which the first layer carries each key token to its paired value and the second layer stores the associations. We then distinguish failures of writing from failures of retrieval. In width-128 models trained on 64 pairs, recall falls to at a load of . Reading the same state with each association's write key recovers , and a matrix built by ridge regression from the same keys and values raises this to , so both online writing and retrieval limit recall, and many associations remain accessible after the model fails to recall them. Finally, a training curriculum that progressively increases the number of pairs (the load) raises the load at which the model reaches recall from to at , and reaches it faster than training directly at the target load. Effective recall capacity therefore depends on the learned writing and retrieval procedures and can be increased through training without enlarging the memory state.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, linear attention, delta rule, causal intervention, recurrent state, associative recall, Gated DeltaNet, activation patching, in-context memory, linear RNNs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_paXWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores, in practice, how many key-value associations are able to be recalled in linear-state models. The authors perform experiments on single-head, two-layer deltanet models trained for MQAR. They find that slowly increasing the number of pairs stored in the state during training can help in this setting.

Critical strengths

None

Critical weaknesses
  1. I found the language and writing style in this paper to be quite poor, sounding very AI-generated. This made it extremely difficult to follow and lacked actual technical depth.

  2. The experiment results are not very rigorous as this is demonstrated on a single-head, two-layer model. The suggested training curriculum is not validated at any sort of real-model scale on real tasks.

Questions for authors

None

Reviewer_9YSYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies how many key-value association paris a delta-rule memory recalls in practice. The main text use a 2-layer, single-head deltanet (without FFN), trained on multi-query associative recall synthetic task.

Using this setup, the authors show that layer 1 acts as a one-step previous-token register and layer 2 stores the assiciation.

Using three measurements: a) retrieval using model's own query; b) retrieval using each pair's original write key; c) use a ridge-regression memory (batch, all pairs at once), the authors discover two failures: 1) retrieval failure (measurement b - a), and storage failure (c - b).

Critical strengths
  • The three measurements: a) retrieval using model's own query; b) retrieval using each pair's original write key; c) use a ridge-regression memory (batch, all pairs at once) are clean ways to measure storage failure and retrieval failure.

  • The two failures (storage and retrieval) are helpful for the understanding of linear attention / RNN.

  • This work is candid about limitations.

  • Some parctical relevance are also provided in appendix.

Critical weaknesses
  • The two-stage circuit (layer 1 acts as a one-step previous-token register and layer 2 stores the assiciation) was already known in the study of "induction-head" [1,2]. This limits the novelty of this work.

  • The architecture in main text is an over-simple toy architecture without FFN, multi-heads, short convolution. The circuit and the storage/retrieval split may therefore not carry over to standard architectures.

[1] Olsson, Catherine, et al. "In-context learning and induction heads." arXiv preprint arXiv:2209.11895 (2022).

[2] Bietti, Alberto, et al. "Birth of a transformer: A memory viewpoint." Advances in Neural Information Processing Systems 36 (2023): 1560-1588.

Questions for authors
  • Do the storage/retrieval failures hold in a standard DeltaNet with short convolution and FFN layers?
Reviewer_1r6NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the recall capacity of delta-rule layers. A model of two delta-rule (DeltaNet) layers with a single head per layer is trained on a multi-query associative recall (MQAR) task. Models of different widths are trained on synthetic data to understand how many key-value pairs the model can retain. The work identifies that recall failures can arise from failures of writing or failures of retrieval, and presents measures to distinguish the two and see where they arise. Furthermore, the paper designs a load curriculum to improve the final accuracy, and measures the speed of learning against training at a fixed load.

Critical strengths
  • Recall is one of the important capabilities of a language model; understanding its limits and stress-testing them is an interesting idea. The mechanistic interpretability approach of reading and editing the state is a nice tool for this.

  • A couple of interesting ideas, such as the own-key read and the ridge-regression read, which separate retrieval failures from writing (storage) failures

Critical weaknesses
  • The empirical evaluation is a bit weak. Only a few random seeds are reported for the main claims of Section 6 (one or two seeds per width for the curriculum, and a single seed for the fixed-load controls), and the training dynamics of the models trained on 64 pairs are not reported.

  • The analysis is purely empirical.

  • The experimental evidence is only for two-layer models with one head per layer and no MLPs, at different widths. Beyond one appendix experiment on a 1.3B model, it is not clear how the intuitions extend to multi-head layers, models with MLPs or short convolutions, and hybrid models.

  • The writing can be made a lot clearer. The terms binding, storage and retrieval are defined only informally, and the technical details are not properly formulated: the own-key read and key survival are given without formulas, the shared direction is ambiguous for the second layer, and the training loss is not stated. It has not been an easy read.

Questions for authors
  1. The own-key read uses the key computed at the pair's value token, so it also contains the value token, which the model's query cannot know. When the state is read with the token's mean key instead, recall is 0.26 rather than 0.59 (seed 0, 4dk). How much of the retrieval gap can a query computed from the key token alone actually close?

  2. Is it the gradual increase of the load that helps, or simply training on a mixture of loads? A control trained on the same loads in random order, with the same optimizer settings as the curriculum, would answer this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.