acceptodds
Under review as a conference paper at ICLR 2027

DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time

Abstract

Test-time memorization of sequential information is a foundational capability for any intelligent system that needs to rapidly store and retrieve data, as demonstrated by the attention mechanism in large language models. However, Transformers and linear attention variants are trained via backpropagation through time (BPTT), which has drawbacks such as its enormous memory requirements. Predictive coding (PC)—an algorithm that optimizes a prediction-error energy function with local updates—offers a compelling solution but still lacks test-time memory for sequential data. We develop a solution to this problem by treating the per-step memory update as a latent variable, as opposed to a slow weight to be learned. By taking the log probability of a standard PC-based probabilistic model, we derive a PC-native solution for verbatim sequence memory, and relate our update rule to common choices in the linear attention literature (e.g., DeltaNet). We present an extensive set of empirical tests to probe our system's properties, including its scaling capacity, memory footprint, and ability to infer a context from evidence (e.g., bank (river) vs. bank (money)). We show how DeltaNet-PC can be used for storing single sequences of data, multiple sequences of data, and key-value associative memory pairs with perfect recall accuracy.

Primary area:
applications to neuroscience & cognitive science
Keywords:
predictive coding, associative memory, fast weights, linear attention, local learning rules, test-time learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eqK7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors claim that the memory requirements of BPTT in training transformers and linear attention variants is limiting. In order to address this problem, the authors present DeltaNet-PC, a method for sequential modeling based on predicting coding (PC). The authors connect DeltaNet-PC to existing works on linear attention and conduct experiments on sequential memory tasks.

Critical strengths
  1. The writing is clear and easy to follow.
  2. The derivations are clear and the motivation is well justified.
  3. The experiments are well-designed and the analysis is thorough.
Critical weaknesses
  1. The experiments are limited to synthetic datasets and there are no natural language or real sequential datasets.

  2. The external baselines include only DeltaNet. A more comprehensive comparison would strengthen the generalizability and overall usefulness of the method.

  3. The overall scale of the experiments is limited. Experiments on bigger models with a larger vocabulary size would show whether the method is scalable.

Questions for authors
  1. Can the authors explain whether DeltaNet-PC can be parallelizable during training for practical purposes?

  2. Can the authors compare DeltaNet-PC against additional BPTT baselines with reasonable memory consumption, such as the Mesa layer which also tries to find the exact solution of RLS?

2 comments

Thank you for your review, and for the specific requests. You asked for real data, a larger vocabulary, and the Mesa layer as a baseline. We ran all three and the results strengthen the manuscript.

Real text and a larger vocabulary. We evaluated on natural language with a 50k vocabulary (six times MQAR's) using the copy task from Jelassi et al., 2024, where the model must store and retrieve text. The metric is the fraction of strings copied exactly. DeltaNet-PC copies over 99% of strings, on a tenth of the data the baselines use.

MethodNatural textShuffled text
DeltaNet-PC (local updates)0.9950.992
DeltaNet (BPTT)0.8210.909
MesaNet (BPTT)0.8010.748

Fraction of 640 held-out strings copied with all 255 tokens correct, mean over 3 seeds. DeltaNet and MesaNet are 2-layer models with a 4-token convolution ahead of the memory layer, trained for 64 epochs (6.4M strings), with the learning rate tuned over 4 values and the better of two widths (256, 512) shown. For parity, DeltaNet-PC's key is computed from the last 4 tokens through a learned gate. DeltaNet-PC stops by 629k strings.

More baselines. We added two. MesaNet (the BPTT-trained RLS model, von Oswald et al., 2025) appears in the table above and on MQAR, where it achieved perfect recall on 2/3 seeds and required all 100,000 training samples to do so. DeltaNet-PC needs 32 samples and DeltaNet needs 1,024. We also ran BayesPCN (Yoo & Wood, 2022), a PC-based associative memory, on our Omniglot-MNIST key-value task. Both models reach 100% test accuracy on all three seeds. At test time BayesPCN takes about 19 minutes per episode and DeltaNet-PC takes 12 milliseconds, with a memory 24 times smaller.

Scale. The copy task above uses real text and a 50k vocabulary. The memory is a D×DD \times D matrix, and matrices of 10,000×10,00010{,}000 \times 10{,}000 are well within reach of current hardware, so DD can conceivably grow by another order of magnitude from the 1,0241{,}024 we tested (Section 4.1). Peak training memory is 0.8 GB at 131k tokens, where DeltaNet runs out of memory (Table 3). This is not a scaling paper, but it is one direction that could be scaled.

Parallel training. Yes. Training has no backward pass over time, so positions interact only through the memory state, and our method admits a parallel form. For both DeltaNet and MesaNet, the parallel form was a paper in its own right (Delta rule -> DeltaNet, Mesa layer -> MesaNet).

In light of the responses above, we hope you will consider raising your score.

I appreciate the author for the response. My major concerns have been addressed, and I am raising the score to 3.

Reviewer_XzF9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper derives a new (?) DeltaNet-like predictive coding-like memory and demonstrates it.

The contribution may be a DetaNet-PC architecture, but, as presented, it is difficult to say so.

The architecture/model/algorithm (?) is demonstrated on three recall tasks.

Critical strengths

I think the research area is extremely interesting and attempts to tie PC into the modern deep learning approaches is likely to be fruitful at some point.

Critical weaknesses

I think the most critical weakness it that the paper is that it is self-inconsistent and overclaims rather substantially, particularly relative to prior art.

As far as I can tell, eq. (7) and (8) are standard conjugate Bayesian updates for Bayesian linear regression. Why these were derived through a Taylor series expansion and other convoluted means I don't understand.

The claim against prior art, specifically the non-applicability of Yoo & Wood to sequence data, is simply not true. Hetero-associative recall is part of that work and is explicitly called out as such. AFAIK this paper is Yoo & Wood with key-value binding via heter-associative recall + ROPE.

The abstract claim that "by taking the log prob. of a standard PC-based probability model, we derive..." is not true; eqn. (9) corresponds to no graphical model (footnote 3 makes this clear in a way) and certainly is not "taking the log prob. of a standard PC-based probability model"

And finally, I frankly can't understand what the experiments are really trying to demonstrate.

Questions for authors

Clarification:

Are (7) and (8) standard conjugate Bayesian updates for Bayesian linear regression?

How does BayesPCN in hetero-associative recall mode (k/v) differ from what you have presented besides ROPE? I think at the very minimum this needs to be included as an experimental baseline. And ROPE ablations included as well.

Suggestions:

Reframe everything as an energy-based model.

Old school suggestions:

I believe that every paper should be complete (i.e readable) in its 10 pages without referencing the appendix, etc. It is impossible to understand the training and test regimes for this work without an algorithm or a full description of, say, how and when enc() and dec() parameters are trained.

1 comment

Thank you for your review. In this work, we show mathematical links between predictive coding and linear attention, thus enabling cross-pollination between the two communities. We also demonstrate a functioning test-time memory system for sequential data.

You raise two main concerns: our claims relative to BayesPCN (Yoo & Wood, 2022), and the abstract's description of our model. We believe the paper is accurate on both and explain why below.

Eqs. 7 and 8. Yes, they are the conjugate update for Bayesian linear regression, and we will say so in the text. Thank you for noting this. The Taylor expansion is exact because the energy is quadratic, so nothing is approximated. It gives the energy as a function of the update ΔSt\Delta S_t (Eq. 5), which is the latent that PC infers. The point of our derivation is what it connects. Given our energy function, a PC learning step on the memory gives the delta rule (DeltaNet), and PC inference on the memory update gives RLS (the Mesa layer). The link between predictive coding and linear attention in our work is new. BayesPCN does not make that link.

BayesPCN and sequences. Our manuscript states that Yoo & Wood "do not address sequential data" (Section 2.1). That is accurate. BayesPCN stores i.i.d. images. Its authors say sequences are future work: "extending BayesPCN beyond memorizing i.i.d. data to memorizing sequences like the human episodic memory would be of great interest." Hetero-associative recall on its own does not solve our sequence task. A memory keyed on the previous token can recall at most 21% of the tokens in our sequences (chance is 10%), because the same token is followed by different tokens at different positions.

How we differ from BayesPCN, besides RoPE. The write is shared, and Table 1 lists both as RLS. Everything around it differs. (1) BayesPCN has no training phase. We train an encoder, decoder and key space, and without that training the same write with RoPE recalls 29% of tokens against 100% trained (Table 2). (2) Our address is computed from the previous observation and the position, so it exists before the item arrives, which is what lets a cue retrieve the next item. BayesPCN infers its address from the item itself. (3) Our memory is a single D×DD \times D matrix that is written and read in one pass each. BayesPCN's memory is every weight in the network, written with 500 settling steps and read with up to 15,000.

BayesPCN as a baseline. We ran BayesPCN's public code on our Omniglot-MNIST key-value task, which is the i.i.d. setting it was designed for. It does not apply to our sequential setting for the reasons described above. Both models reach 100% test accuracy on all three seeds. At test time BayesPCN takes about 19 minutes per episode and DeltaNet-PC takes 12 milliseconds, with a memory 24 times smaller.

BayesPCNDeltaNet-PC
Test accuracy (3 seeds)100%100%
Test time per 128-pair episode19 min12 ms
Memory per episode (floats)3.2M131k

128 test episodes (16,384 probes) per seed, both timed on a V100 GPU.

RoPE ablations. These are in the appendix's ablation table (Table 7). Freezing RoPE drops recall on every task where it is used.

Abstract and Eq. 9. The abstract's sentence describes the derivation of the write, which uses Eq. 2, not Eq. 9. Eq. 2 is a directed Gaussian model, and as you note, its updates are the conjugate updates of Bayesian linear regression. So the write does follow from the log probability of a standard probabilistic model, as the abstract says. Eq. 9 is the full system, where the value latent receives two predictions. We write it with a proportionality sign, state in footnote 3 that it is a product of experts, and define the system by its energy (Figure 2B, Appendix B.4). We agree that the full system is an energy-based model and will make the abstract's scope explicit.

What the experiments demonstrate. Each one probes a property of a test-time memory: capacity (one sequence, up to L=DL = D), interference (several sequences in one memory, and inferring which context a cue belongs to), key-value binding (Omniglot-MNIST), and sample efficiency/memory efficiency against BPTT-trained models (MQAR).

Algorithm. Algorithm 1 (Appendix B.5, referenced in Section 3.3) states how and when the encoder and decoder are trained. We will move it into the main text.

Thank you again for helping us highlight the similarities and differences between BayesPCN and our work, which will improve the manuscript. We included the BayesPCN baseline, as you asked, and hope you will consider accepting our work.

Reviewer_1VJVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces DeltaNet-PC, which adds fixed-size fast-weight memory to predictive coding without BPTT by treating each memory update as an inferred latent variable (yielding closed-form RLS). It is evaluated on synthetic sequence recall, MQAR, and Omniglot-MNIST binding.

Critical strengths
  1. Treating the per-step memory change as an inferred latent rather than a slow weight is a clean way to derive RLS fast weights inside predictive coding, and pairing it with learnable RoPE lets the model adapt naturally between positional and content addressing.
  2. Avoiding BPTT keeps GPU memory nearly constant with sequence length, while test-time settling lets a single-layer model work backward to infer hidden episode context from ambiguous prefixes.
Critical weaknesses
  1. Even though the paper motivates itself via LLMs and linear attention, there is no real text experiment or standard next-token perplexity evaluation. Modern foundation models predict tokens sequentially on natural text, whereas synthetic probes like MQAR are just sanity checks with limited practical value on their own. Worse, it is unclear if DeltaNet-PC can actually model real text. Gaussian MSE against a fixed codebook averages ambiguous next tokens instead of modeling discrete token probabilities, unweighted RLS overloads once sequence length exceeds width (L > D, Figure 3B), and only 1-layer linear models are tested.

  2. Theoretically, the model sidesteps temporal credit assignment rather than solving it without BPTT. Writes do no PC settling at all (just closed-form RLS, as in Mesa layers), and during learning the memory S is frozen so zero gradient flows back to past writes. That only works because queries and keys are hard-tied (q_t = k_t) for verbatim lookup—there is no learning signal to train separate Q/K projections or selective forget gates for long contexts.

  3. The MQAR sample-efficiency claim (32 vs. 1,024 streams, Table 3) is not apples-to-apples, and GPU runtime goes the wrong way. DeltaNet-PC gets a frozen orthogonal codebook, second-order RLS, and 16 extra test-time settling steps, while DeltaNet uses NLMS and must learn 2.1M token embeddings from scratch (32 streams cannot even see half of the 8,192 vocabulary). Fair baselines (fixed-codebook DeltaNet, BPTT-trained RLS/MesaNet, and no test-time write settling) are missing, and losing chunkwise parallelism makes H100 training ~43x slower per stream (18 ms vs. 0.42 ms).

Questions for authors
  1. Can you evaluate DeltaNet-PC with cross-entropy on a real text language modeling benchmark (perplexity) where L >> D?
  2. With S frozen during learning, how could this framework learn asymmetric Q/K projections or a data-dependent forget gate?
  3. How do fixed-codebook DeltaNet, MesaNet, and DeltaNet-PC without the 16-step test-time settle compare on MQAR?
1 comment

Thank you for your review. In this work, we show mathematical links between predictive coding and linear attention, thus enabling cross-pollination between the two communities. We also demonstrate a functioning test-time memory system for sequential data, which provides a tool that may be useful for building a PC-based language model.

Language Modeling. We've built a memory model in the same spirit as past associative memory works, e.g., Associative Memories via Predictive Coding, Salvatori et al., 2021. As a result, a perplexity measurement is out of scope.

Real text. We evaluated on natural language with a large vocabulary size (50k) using the copy task from Jelassi et al., 2024, where the model must store and retrieve text. The metric is the fraction of strings copied exactly. The table below further demonstrates the strength of our memory module.

MethodNatural textShuffled text
DeltaNet-PC (local updates)0.9950.992
DeltaNet (BPTT)0.8210.909
MesaNet (BPTT)0.8010.748

Fraction of 640 held-out strings copied with all 255 tokens correct, mean over 3 seeds. DeltaNet and MesaNet are 2-layer models with a 4-token convolution ahead of the memory layer, trained for 64 epochs (6.4M strings), with the learning rate tuned over 4 values and the better of two widths (256, 512) shown. For parity, DeltaNet-PC's key is computed from the last 4 tokens through a learned gate. DeltaNet-PC stops by 629k strings.

Cross entropy. We used an MSE loss for consistency with the PC literature. At recall, each prediction is snapped to the nearest entry in the codebook before it is fed back in, so an averaged prediction is never reused. Nothing prevents us from using cross-entropy.

L>DL>D. Increasing DD increases memory capacity (Figure 3C). The memory is a D×DD \times D matrix, and matrices of 10,000×10,00010{,}000 \times 10{,}000 are well within reach of current hardware, so DD can conceivably grow by another order of magnitude. This is not a scaling paper, but it is one direction that could be scaled. For future work, we can make use of PC's error signal to decide what to store (a form of data-dependent gating), and we can transfer memories from fast weights to slower weights to further address L>DL>D.

More layers. Additional encoder or decoder layers are already available in the code. The context inference experiment also points to how to create hierarchical models.

Temporal credit assignment. It is correct that no gradient flows back to past writes. This is the constraint we work under (and achieve exact recall under). To clarify, the model must learn to make qt=ktq_t=k_t. The positional encoding makes a query at a new position differ from the write key, so the model must learn the positional code and how to organize the key space, something that typically requires BPTT. Our context inference experiment shows one way queries can differ from keys: the queries lack context information, which must be inferred to find the right key.

Settling writes. Earlier versions of this system wrote by settling, and it worked. We use RLS because it's faster and more stable.

Sample efficiency. We ran DeltaNet with our frozen codebook and retuned DeltaNet-PC to work without the 16 extra settling steps. The results remain the same: 32 samples to exact recall for DeltaNet-PC, 1,024 samples for DeltaNet. We also evaluated MesaNet (BPTT-trained RLS model) on MQAR (with a learned embedding, since it stays at chance with our fixed codebook), which only achieved perfect recall on 2/3 seeds and required all 100,000 training samples to achieve perfect recall.

GPU Runtimes. The runtime is our measurement of a sequential implementation, reported for transparency. We make no claims about GPU runtimes at this stage. For both DeltaNet and MesaNet, the parallel form was a paper in its own right (Delta rule -> DeltaNet, Mesa layer -> MesaNet). Our method admits a parallel implementation.

We believe this work will be of interest to the ICLR community and hope you will consider accepting it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.