acceptodds
Under review as a conference paper at ICLR 2027

Towards compressive and scalable recurrent memory

Abstract

Recurrent memory unlocks an unbounded context horizon for language models within a finite memory footprint. We introduce Elastic Memory, a memory architecture that compresses historical context into polynomial states and reconstructs these features for attention. By leveraging HiPPO for parallelized online compression and polynomial sampling for flexible retrieval, our approach operates without additional trainable parameters. This design decouples memory capacity from the backbone model, maintaining a fixed-size recurrent state regardless of sequence length and model size. In from-scratch pretraining, Elastic Memory outperforms strong baselines on PPL and LongPPL across three long-document domains. Notably, it achieves lower perplexity than Memorizing Transformer with a 16 smaller memory footprint, and scales robustly with model size and memory size. During continued pretraining (CPT) on Llama-2 7B, it matches FreqKV's quality while delivering a 1.83 throughput speedup. For supervised fine-tuning (SFT), Elastic Memory achieves strong results on LongBench, including a 5.31-point gain over FreqKV on RepoBench-P. Elastic Memory provide a principled, scalable and parameter-free recurrent memory for LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, long context, memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vR1dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces elastic memory, a fixed-size long-term memory for Transformer models, by compressing past key and values into a fixed-size set of HiPPO polynomial coefficients. Sequences are processed in blocks for efficiency and each block reconstructs approximations for past key values at chosen positions. The LLM requires training to adapt to new architecture, but the method does not introduce additional trainable parameters. Experiments cover from-scratch pretraining, continued pretraining, and fine-tuning evaluation on downstream tasks.

Critical strengths

A wide range of experiments are done and the evidence show that the compressed memory can preserve long-range information. Elastic memory beats the baselines at equal total state budgets, showing that a fixed analytic compressor can be a viable alternative to learned compressive memories. Memory scaling by raising the polynomial order NN, improves PPL and LongPPL with no additional parameters and an acceptable efficiency tradeoff. Even with a larger NN, training throughput reaims higher that Melodi, and in continued pretraining on LLama-2-7B, it is faster than FreqKV at comparable perplexity.

Critical weaknesses
  1. The experiments do not isolate the contribution of the proposed mechanism. In the from-scratch experiments, Elastic Memory augments only the ninth layer, concentrating the entire state budget there (Appendix A.1), while Infini-Transformer and Melodi augment every layer. This difference can confound both the quality and throughput comparisons, making it difficult to determine how much of the improvement comes from the proposed memory representation rather than from layer placement and budget allocation. A comparison using the same augmented layers and matched total state budgets would help clarify this.
  2. Scalability to longer contexts and long generation remains insufficiently validated. Evaluations extend only to 32K tokens, so quality at longer context lengths, where the paper’s motivation of an unbounded context horizon matters most, is untested. As history grows, the same N coefficients must cover more positions, so reconstruction fidelity per position likely decreases. Moreover, the paper does not specify how Elastic Memory operates during autoregressive decoding. Under the block-recurrent design, generated blocks are written into the fixed-size state, so exact access to earlier output may be lost as generation proceeds. Since memory matters most for long contexts and long generation, evaluation beyond 32K and on long-output tasks would strengthen the paper’s claims.
Questions for authors
  1. Why was the ninth layer chosen for memory injection, and how do results change with a different layer or with several augmented layers?

  2. How sensitive are the results to the block length L? Tokens early in a block have almost no exact preceding context and must rely on the reconstruction.

  3. During decoding, when are completed blocks incorporated into the coefficient state and their exact KV discarded?

2 comments
AuthorsOpenReview ↗

Response to Reviewer vR1d

We thank the reviewer for recognizing the breadth of the experiments and the viability of a fixed analytic compressor as an alternative to learned compressive memory.

W1. Elastic Memory augments only the ninth layer [...] while Infini-Transformer and Melodi augment every layer. This difference can confound both the quality and throughput comparisons [...]. A comparison using the same augmented layers and matched total state budgets would help clarify this.

The main comparison already includes a placement-matched baseline: Memorizing Transformer and Elastic Memory both augment layer 9 with the same 0.8M budget, and Elastic Memory leads (PPL/LongPPL, exp sampling):

Mem. TransformerElastic Memory
PG-1911.050 / 15.79410.651 / 8.885
Proof-Pile3.305 / 3.8432.915 / 2.586

For the all-layer baselines we also ran a control placing Melodi's memory in the same layer 9 (Proof-Pile, 100M): PPL is on par (2.9182.918 vs. 2.9162.916) and LongPPL is about 7%7\% lower for Elastic Memory (2.5862.586 vs. 2.7792.779), while Elastic Memory adds no trainable parameters, unlike Melodi's summary module. The advantage appears on the predictions that depend on distant history. We will add the control to the appendix and list layers, state size, and reader width in every main table.

W2. Evaluations extend only to 32K tokens [...] the paper does not specify how Elastic Memory operates during autoregressive decoding [...] exact access to earlier output may be lost as generation proceeds. [...] evaluation beyond 32K and on long-output tasks would strengthen the paper's claims.

Beyond 32K. One PG-19 100M checkpoint (N=M=540N=M=540, L=2048L=2048), evaluated without retraining on non-overlapping windows, reaches PPL 10.755010.7550 at 64K and 10.881610.8816 at 128K, with a constant-size state of 1,555,2001{,}555{,}200 bytes per sequence.

Decoding. Generated tokens follow the same memory path as input tokens: the current block keeps exact KV and each completed block is written into the state (steps in Q3). Earlier tokens keep informing later predictions through the compressed state, and NIAH tests this directly, since facts in earlier blocks remain retrievable after compression.

Long-output evaluation. It needs purpose-built benchmarks and matched SFT data, which exceeded the time and compute available during the rebuttal. We view it as a natural follow-up, since the 64K/128K, NIAH, and memory-scaling results already support the central claims.

Q1. Why was the ninth layer chosen for memory injection, and how do results change with a different layer or with several augmented layers?

Proof-Pile, 100M (PPL / LongPPL):

Layer(s)expuniform
32.961 / 2.7533.042 / 2.766
62.942 / 2.6322.986 / 2.586
92.915 / 2.5862.958 / 2.560
3+92.921 / 2.5492.951 / 2.537

Quality improves with depth under both policies and layer 9 is the best single layer, hence our choice. Memory also composes across layers, and a second layer lowers LongPPL further.

Q2. How sensitive are the results to the block length LL? Tokens early in a block have almost no exact preceding context and must rely on the reconstruction.

LLexpuniform
5123.105 / 2.7283.222 / 2.699
1,0243.024 / 2.6743.102 / 2.631
2,0482.915 / 2.5862.958 / 2.560

(PPL / LongPPL.) Both metrics improve monotonically with LL under both policies, so we use L=2048L=2048. Block-initial tokens read history through the MM reconstructed entries, which is what the design is for. With 50%50\% of local context corrupted in all layers, Elastic Memory still beats Memorizing Transformer on PPL and LongPPL (Appendix B.3), so the model uses the reconstructed history effectively.

Q3. During decoding, when are completed blocks incorporated into the coefficient state and their exact KV discarded?

  1. Reconstruct the historical KV visible to the current block from the state built from completed blocks.
  2. Within the block, attend over exact causal KV token by token.
  3. At LL tokens, write the block's pre-RoPE keys and values into the state and release its exact KV.
  4. A tail block shorter than LL stays in the local cache.

We will add this to the method section next to Algorithm 1. We hope these additions address your concerns, and we would be grateful if you would consider raising your assessment.

Dear Reviewer vR1d,

Thank you for your valuable comments. As the discussion period closes today, we would appreciate knowing whether these address your concerns, and we are happy to answer any further questions in the remaining time.

Best regards, The Authors

Reviewer_9nchOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Elastic Memory, a recurrent memory architecture for Transformer language models that compresses historical keys and values into High-Order Polynomial Projection Operators (HiPPO) Legendre (LegS) polynomial coefficients Ci∈RN×dC_i \in \mathbb{R}^{N \times d} in parallel blocks of length LL. Before causal attention in each block, a polynomial-sampling reader evaluates the basis at MM historical coordinates via Ri∈RM×NR_i \in \mathbb{R}^{M \times N} to reconstruct memory keys and values without introducing additional trainable parameters. The authors evaluate the method across from-scratch pretraining (100M to 400M parameters), continued pretraining (CPT) on Llama-2 7B and 13B, and supervised fine-tuning (SFT) on Llama-3-8B-Instruct.

Critical strengths
  1. Because the HiPPO-LegS block transition matrices Pi∈RN×NP_i \in \mathbb{R}^{N \times N} and polynomial reconstruction operators Ri∈RM×NR_i \in \mathbb{R}^{M \times N} are defined analytically, memory capacity can scale by increasing the polynomial order NN without adding trainable parameters (Section 2.4, Table 1). Furthermore, the sampling schedule (uniform versus exponential) can be switched at evaluation time without retraining (Table 5).
  2. In continued pretraining on Llama-2 7B and 13B (Section 3.2, Table 6), Elastic Memory matches the 32K PG-19 perplexity of FreqKV (7.08 versus 7.02 PPL at 7B, and 6.42 versus 6.41 PPL at 13B) while achieving 1.83 times higher training throughput at 7B (64,498 versus 35,211 tokens/s) and 1.71 times higher training throughput at 13B (38,578 versus 22,508 tokens/s).
Critical weaknesses
  1. Essential experimental hyperparameters and memory measurements needed to evaluate the efficiency claims are missing. Specifically, Section 2.3 (Equations 4 and 5) defines the reconstruction matrix Ri∈RM×NR_i \in \mathbb{R}^{M \times N} over MM sample points, and Appendix C.4 (Equation 19) defines the exponential sampling schedule with parameter α∈(0,1)\alpha \in (0, 1). Although Appendix A.1 reports the polynomial order N∈{540,1080,2160,4320,8640}N \in \{540, 1080, 2160, 4320, 8640\}, neither the reader width MM nor the parameter α\alpha is specified in Section 3 or Appendices A to C. Because reconstruction costs O(MNd)O(MNd) and augmented attention costs O(L(M+L)d)O(L(M+L)d) per block (Appendix C.5), the computational overhead cannot be verified without MM. Moreover, Appendix A.2 (page 14) states that Elastic Memory stores its recurrent state in 32-bit floating-point (FP32) precision during 16-bit brain floating-point (BF16) continued pretraining, whereas the CPT baselines are matched by cache slot count (4,096 slots) rather than byte count or peak GPU memory.
  2. The related-work discussion in Section 1 and Appendix D omits FourierAttention (Liu et al., 2025), which also compresses and reconstructs historical keys and values using HiPPO dynamics. A direct discussion is needed to clarify the specific contribution of the growing-history LegS state and polynomial-sampling reader relative to FourierAttention.
  3. [Post-rebuttal note] The authors did not submit a response during the discussion period. Consequently, all questions regarding the missing hyperparameters MM and α\alpha, peak GPU memory in bytes, and the comparison with FourierAttention remain unresolved.
Questions for authors
  1. What are the numerical values of the reader width MM and the exponential sampling parameter α\alpha in Section 3, and is MM held fixed when scaling the polynomial order NN from 540540 to 86408640 in Table 3?
  2. What is the peak GPU memory in GiB for Elastic Memory, FreqKV, and MemTransform under the same batch size in Table 6, including the FP32 recurrent states, precomputed operator banks, and temporary reconstruction buffers?
  3. How does Elastic Memory compare conceptually and empirically with FourierAttention (Liu et al., 2025)?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 9nch

We thank the reviewer for recognizing the parameter-free capacity scaling, the post-training sampler switching, and the CPT quality–speed trade-off.

W1. The experimental reader width (MM) and sampling parameter (α\alpha) are unspecified. Elastic Memory uses FP32 recurrent states, but the CPT methods are matched by cache slots rather than reported bytes. Peak memory, including cached operators and reconstruction buffers, would make the comparison more informative.

We now report all three: MM and α\alpha in Q1, and measured end-to-end peak memory in Q2. The CPT budget of 4,096 slots follows the released FreqKV configuration. We will add the configuration table and measurements to the paper.

W2. The related-work discussion misses FourierAttention (Liu et al., 2025) [...] their contribution relative to that method needs clearer discussion.

We will add the citation and a dedicated discussion (see Q3).

Q1. What are the values of MM and α\alpha, and is MM fixed during the memory-scaling experiments?

The main scratch configuration is N=M=540N=M=540 with L=2048L=2048. In the scaling experiments MM grows with NN:

Scale1×1\times2×2\times4×4\times8×8\times16×16\times
N=MN=M5401,0802,1604,3208,640

The study therefore scales the memory configuration (state order and reader width) with zero additional parameters, and we will report NN and MM in every table. The exponential schedule is

q=1−1Ms,xjt=1−q j1−q M−1,j=0,…,M−1,q=1-\frac{1}{Ms},\qquad \frac{x_j}{t}=\frac{1-q^{\,j}}{1-q^{\,M-1}},\qquad j=0,\dots,M-1,

with s=0.5s=0.5 (q≈0.9963q\approx0.9963 at M=540M=540). Uniform sampling uses xj/t=j/(M−1)x_j/t=j/(M-1). We will align Eq. 19 with this parameterization.

Q2. What is the peak GPU memory for Elastic Memory and FreqKV under the same batch size, including operators and temporary buffers?

We measured whole-process peak GPU memory on LLaMA-2-7B under the same 4,096-slot budget, covering weights, state and KV caches, operators, reconstruction buffers, and activations.

Inference on PG-19 (MiB)

ContextFreqKVElastic MemoryDiff.
2K14,42714,939+3.5%
4K15,93116,443+3.2%
8K18,45318,957+2.7%
16K24,00924,513+2.1%
32K34,34734,339parity

CPT training peak (MiB): FreqKV 26,07926{,}079; Elastic Memory 26,21926{,}219 (compact RoPE) and 26,22926{,}229 (native RoPE), a 0.5%0.5\% difference.

Even with FP32 recurrent state, end-to-end memory stays within 3.5%3.5\% of FreqKV at inference, and the gap closes with length. The "16×16\times" comparison in the paper refers to persistent state elements (Appendix C.5). In bytes,

Sstate=B∑ℓNℓHKV,ℓ (dk,ℓ+dv,ℓ) bℓ,S_{\text{state}}=B\sum_{\ell}N_\ell H_{\mathrm{KV},\ell}\,(d_{k,\ell}+d_{v,\ell})\,b_\ell,

with bℓb_\ell the bytes per element. We will add both tables and this formula to the appendix.

Q3. Which capabilities or benefits distinguish the method from FourierAttention?

FourierAttention is a training-free KV-cache compressor that projects only long-context-insensitive head dimensions onto Fourier bases. Elastic Memory differs in three ways:

  • State: a LegS projection of the complete history with position-aware reconstruction.
  • Training: the state is updated in parallel per block, and the model is trained to read from it.
  • Reader: MM positions of our choosing are sampled, and the policy can change at evaluation time without retraining (Table 5).

The two works use analytic bases for KV history in complementary ways, and we will discuss them in related work.

We hope these measurements and clarifications address your concerns, and we would be grateful if you would consider raising your assessment.

Dear Reviewer 9nch,

Thank you for your valuable comments. As the discussion period closes today, we would appreciate knowing whether these address your concerns, and we are happy to answer any further questions in the remaining time.

Best regards, The Authors

Reviewer_wYhmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Elastic Memory encodes historical keys and values into HiPPO-LegS polynomial coefficients. From these coefficients, a collection of pseudo-KV entries is rebuilt at each block, which the active block utilizes. The memory system is devoid of trainable parameters, and its capacity can be adjusted separately from the model's size. The study assesses its performance when trained from the ground up (100M-400M, PPL/LongPPL), alongside continued pretraining of Llama-2 in comparison with FreqKV, and includes SFT evaluations on LongBench and NIAH.

Critical strengths
  1. This design is straightforward, does not rely on parameters, is founded on principles, and is cost-effective in application, achieving 1.83 times the throughput of FreqKV during the 7B continued pretraining.

  2. The memory-scaling experiment is impressive. The quality consistently enhances as N increases without any increase in parameters, outperforming Melodi despite Melodi having greater memory capacity.

  3. The evaluation occurs in three settings (scratch, CPT, SFT), and the ability to change the sampler at evaluation time without the need for retraining is an interesting feature.

Critical weaknesses
  1. LegS's uniform measure indicates that as the context increases, the per-token resolution of a fixed-size state decreases, resulting in reconstructed keys that are averaged out. The testing for precise recall is limited to NIAH up to 16K, which does not meet the expectations set by the "unbounded context" concept.

  2. The baseline performance is inadequate (the Transformer++ baseline only processes the current block), with a lack of SWA, SSM/linear-attention memory, and RMT-style baselines. In comparison to FreqKV, the method appears to perform similarly, yet the significant LongPPL discrepancy on FineWeb-Edu requires clarification.

  3. The improvement over HiPPO + FreqKV is minimal. The operators are calculated for a defined context range, leaving the behavior outside of that range uncertain.

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer wYhm

We thank the reviewer for recognizing the simplicity of the design, the memory-scaling results, and the evaluation-time sampler switching.

W1. [...] the per-token resolution of a fixed-size state decreases [...] The testing for precise recall is limited to NIAH up to 16K, which does not meet the expectations set by the "unbounded context" concept.

"Unbounded" refers to the recurrent formulation: the per-sequence state has constant size NH(dk+dv)NH(d_k+d_v) per augmented layer, and for fixed N,M,LN,M,L the per-block cost does not grow with the number of processed blocks (Appendix C.5). Fidelity within this state is set by the memory configuration, which scales without adding parameters:

  • Going from N=M=540N=M=540 to 8,6408{,}640 reduces key/value reconstruction MSE from 0.450.45 to 0.270.27 (Appendix B.1), PPL from 2.922.92 to 2.752.75, and LongPPL from 2.592.59 to 2.152.15, with the backbone unchanged.
  • One PG-19 100M checkpoint (N=M=540N=M=540, L=2048L=2048), evaluated without retraining on non-overlapping windows, reaches PPL 10.755010.7550 at 64K (173 windows) and 10.881610.8816 at 128K (86 windows). Doubling the length changes PPL by about 1.2%1.2\%, and the state stays at 1,555,2001{,}555{,}200 bytes per sequence.

For retrieval, the NIAH grid covers 31 lengths (1K–16K, matching the SFT length) and 10 depths. A needle must survive compression across many blocks, and Elastic Memory retains broad coverage at longer lengths where the window and raw-memory baselines lose many needles (Figure 2).

W2. The baseline performance is inadequate [...] with a lack of SWA, SSM/linear-attention memory, and RMT-style baselines. In comparison to FreqKV, the method appears to perform similarly, yet the significant LongPPL discrepancy on FineWeb-Edu requires clarification.

Baselines. All methods share one Llama-style backbone, data order, and training budget, and cover the main memory families: Memorizing Transformer (KV retrieval), Infini-Transformer (linear-attention associative memory), and Melodi (learned recurrent summaries, the RMT family). CPT adds a sink+sliding-window Transformer++ (4 sink and 4,092 recent tokens), MemTransform, and FreqKV. In scratch training, Transformer++ is the memory-free reference, and we will label each baseline's history access in the tables. SSMs replace attention itself and are complementary to attention-based memory.

FreqKV. At 7B, PPL is within 1%1\% of FreqKV (7.087.08 vs. 7.027.02) and 8.8%8.8\% below the window baseline at 7B (7.087.08 vs. 7.767.76) and 13B (6.426.42 vs. 7.047.04). End-to-end peak memory is also comparable (our reply to Reviewer 9nch, Q2), and the sampled reader adds evaluation-time control over the history operating point (Table 5).

FineWeb-Edu. LongPPL 10.91510.915/11.93211.932 comes from the larger-capacity configuration (all 12 layers, N=M=2048N=M=2048), consistent with the Proof-Pile scaling trend (2.59→2.152.59\to2.15), and we will label it in Table 2. Under the common 0.8M budget, Elastic Memory leads all baselines on PG-19 (LongPPL 8.6468.646 vs. 9.5889.588) and Proof-Pile (LongPPL 2.5602.560 vs. 2.7362.736).

W3. The improvement over HiPPO + FreqKV is minimal. The operators are calculated for a defined context range, leaving the behavior outside of that range uncertain.

Elastic Memory builds on HiPPO dynamics and turns them into a trainable Transformer memory: a full-history LegS projection, parallel block updates, and a position-selectable reader. The reader gives a measurable benefit: on one PG-19 checkpoint, switching only the sampling policy moves LongPPL from 8.898.89 to 8.608.60 (Table 5), with consistent directional gains on all three datasets, so one model serves different history preferences without retraining.

All operators are analytic and shared across sequences. Covering a longer range is a one-time offline computation that neither enlarges the per-sequence state nor adds learned weights (Appendix C.5). With fixed normalized coordinates, the reconstruction matrix is shared across blocks (O(MN)O(MN)).

Questions. See weaknesses

Answered above. We hope these additions address your concerns, and we would be grateful if you would consider raising your assessment.

Dear Reviewer wYhm,

Thank you for your valuable comments. As the discussion period closes today, we would appreciate knowing whether these address your concerns, and we are happy to answer any further questions in the remaining time.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.