- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors claim that the memory requirements of BPTT in training transformers and linear attention variants is limiting. In order to address this problem, the authors present DeltaNet-PC, a method for sequential modeling based on predicting coding (PC). The authors connect DeltaNet-PC to existing works on linear attention and conduct experiments on sequential memory tasks.
Critical strengths
- The writing is clear and easy to follow.
- The derivations are clear and the motivation is well justified.
- The experiments are well-designed and the analysis is thorough.
Critical weaknesses
-
The experiments are limited to synthetic datasets and there are no natural language or real sequential datasets.
-
The external baselines include only DeltaNet. A more comprehensive comparison would strengthen the generalizability and overall usefulness of the method.
-
The overall scale of the experiments is limited. Experiments on bigger models with a larger vocabulary size would show whether the method is scalable.
Questions for authors
-
Can the authors explain whether DeltaNet-PC can be parallelizable during training for practical purposes?
-
Can the authors compare DeltaNet-PC against additional BPTT baselines with reasonable memory consumption, such as the Mesa layer which also tries to find the exact solution of RLS?
2 comments
I appreciate the author for the response. My major concerns have been addressed, and I am raising the score to 3.
Thank you for your review, and for the specific requests. You asked for real data, a larger vocabulary, and the Mesa layer as a baseline. We ran all three and the results strengthen the manuscript.
Real text and a larger vocabulary. We evaluated on natural language with a 50k vocabulary (six times MQAR's) using the copy task from Jelassi et al., 2024, where the model must store and retrieve text. The metric is the fraction of strings copied exactly. DeltaNet-PC copies over 99% of strings, on a tenth of the data the baselines use.
Fraction of 640 held-out strings copied with all 255 tokens correct, mean over 3 seeds. DeltaNet and MesaNet are 2-layer models with a 4-token convolution ahead of the memory layer, trained for 64 epochs (6.4M strings), with the learning rate tuned over 4 values and the better of two widths (256, 512) shown. For parity, DeltaNet-PC's key is computed from the last 4 tokens through a learned gate. DeltaNet-PC stops by 629k strings.
More baselines. We added two. MesaNet (the BPTT-trained RLS model, von Oswald et al., 2025) appears in the table above and on MQAR, where it achieved perfect recall on 2/3 seeds and required all 100,000 training samples to do so. DeltaNet-PC needs 32 samples and DeltaNet needs 1,024. We also ran BayesPCN (Yoo & Wood, 2022), a PC-based associative memory, on our Omniglot-MNIST key-value task. Both models reach 100% test accuracy on all three seeds. At test time BayesPCN takes about 19 minutes per episode and DeltaNet-PC takes 12 milliseconds, with a memory 24 times smaller.
Scale. The copy task above uses real text and a 50k vocabulary. The memory is a D×D matrix, and matrices of 10,000×10,000 are well within reach of current hardware, so D can conceivably grow by another order of magnitude from the 1,024 we tested (Section 4.1). Peak training memory is 0.8 GB at 131k tokens, where DeltaNet runs out of memory (Table 3). This is not a scaling paper, but it is one direction that could be scaled.
Parallel training. Yes. Training has no backward pass over time, so positions interact only through the memory state, and our method admits a parallel form. For both DeltaNet and MesaNet, the parallel form was a paper in its own right (Delta rule -> DeltaNet, Mesa layer -> MesaNet).
In light of the responses above, we hope you will consider raising your score.