- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces elastic memory, a fixed-size long-term memory for Transformer models, by compressing past key and values into a fixed-size set of HiPPO polynomial coefficients. Sequences are processed in blocks for efficiency and each block reconstructs approximations for past key values at chosen positions. The LLM requires training to adapt to new architecture, but the method does not introduce additional trainable parameters. Experiments cover from-scratch pretraining, continued pretraining, and fine-tuning evaluation on downstream tasks.
Critical strengths
A wide range of experiments are done and the evidence show that the compressed memory can preserve long-range information. Elastic memory beats the baselines at equal total state budgets, showing that a fixed analytic compressor can be a viable alternative to learned compressive memories. Memory scaling by raising the polynomial order , improves PPL and LongPPL with no additional parameters and an acceptable efficiency tradeoff. Even with a larger , training throughput reaims higher that Melodi, and in continued pretraining on LLama-2-7B, it is faster than FreqKV at comparable perplexity.
Critical weaknesses
- The experiments do not isolate the contribution of the proposed mechanism. In the from-scratch experiments, Elastic Memory augments only the ninth layer, concentrating the entire state budget there (Appendix A.1), while Infini-Transformer and Melodi augment every layer. This difference can confound both the quality and throughput comparisons, making it difficult to determine how much of the improvement comes from the proposed memory representation rather than from layer placement and budget allocation. A comparison using the same augmented layers and matched total state budgets would help clarify this.
- Scalability to longer contexts and long generation remains insufficiently validated. Evaluations extend only to 32K tokens, so quality at longer context lengths, where the paper’s motivation of an unbounded context horizon matters most, is untested. As history grows, the same N coefficients must cover more positions, so reconstruction fidelity per position likely decreases. Moreover, the paper does not specify how Elastic Memory operates during autoregressive decoding. Under the block-recurrent design, generated blocks are written into the fixed-size state, so exact access to earlier output may be lost as generation proceeds. Since memory matters most for long contexts and long generation, evaluation beyond 32K and on long-output tasks would strengthen the paper’s claims.
Questions for authors
-
Why was the ninth layer chosen for memory injection, and how do results change with a different layer or with several augmented layers?
-
How sensitive are the results to the block length L? Tokens early in a block have almost no exact preceding context and must rely on the reconstruction.
-
During decoding, when are completed blocks incorporated into the coefficient state and their exact KV discarded?
2 comments
Dear Reviewer vR1d,
Thank you for your valuable comments. As the discussion period closes today, we would appreciate knowing whether these address your concerns, and we are happy to answer any further questions in the remaining time.
Best regards, The Authors
Response to Reviewer vR1d
We thank the reviewer for recognizing the breadth of the experiments and the viability of a fixed analytic compressor as an alternative to learned compressive memory.
The main comparison already includes a placement-matched baseline: Memorizing Transformer and Elastic Memory both augment layer 9 with the same 0.8M budget, and Elastic Memory leads (PPL/LongPPL, exp sampling):
For the all-layer baselines we also ran a control placing Melodi's memory in the same layer 9 (Proof-Pile, 100M): PPL is on par (2.918 vs. 2.916) and LongPPL is about 7% lower for Elastic Memory (2.586 vs. 2.779), while Elastic Memory adds no trainable parameters, unlike Melodi's summary module. The advantage appears on the predictions that depend on distant history. We will add the control to the appendix and list layers, state size, and reader width in every main table.
Beyond 32K. One PG-19 100M checkpoint (N=M=540, L=2048), evaluated without retraining on non-overlapping windows, reaches PPL 10.7550 at 64K and 10.8816 at 128K, with a constant-size state of 1,555,200 bytes per sequence.
Decoding. Generated tokens follow the same memory path as input tokens: the current block keeps exact KV and each completed block is written into the state (steps in Q3). Earlier tokens keep informing later predictions through the compressed state, and NIAH tests this directly, since facts in earlier blocks remain retrievable after compression.
Long-output evaluation. It needs purpose-built benchmarks and matched SFT data, which exceeded the time and compute available during the rebuttal. We view it as a natural follow-up, since the 64K/128K, NIAH, and memory-scaling results already support the central claims.
Proof-Pile, 100M (PPL / LongPPL):
Quality improves with depth under both policies and layer 9 is the best single layer, hence our choice. Memory also composes across layers, and a second layer lowers LongPPL further.
(PPL / LongPPL.) Both metrics improve monotonically with L under both policies, so we use L=2048. Block-initial tokens read history through the M reconstructed entries, which is what the design is for. With 50% of local context corrupted in all layers, Elastic Memory still beats Memorizing Transformer on PPL and LongPPL (Appendix B.3), so the model uses the reconstructed history effectively.
We will add this to the method section next to Algorithm 1. We hope these additions address your concerns, and we would be grateful if you would consider raising your assessment.