acceptodds
Under review as a conference paper at ICLR 2027

PUM-Net: Dual-Memory Retrieval for Long-Context State Space Models

Abstract

Long-context language modeling requires access to details whose relevance may emerge thousands of tokens after they appear. Selective state space models make long-sequence computation efficient, but recurrent compression offers no explicit route to revisit information discarded from the state. The challenge is to retain access to input-specific evidence while enabling its use alongside related corpus knowledge. We propose PUM-Net, a dual-memory extension of Mamba that sepa- rates recurrent computation, causal context retention, and offline corpus reuse. In- ternal retrieval accesses completed input chunks, while external retrieval accesses corpus representations encoded offline. Bidirectional attention updates each re- trieved set using candidates from the other source before pooling and gated inte- gration. This design preserves Mamba’s parallel scan and incorporates retrieved evidence without appending passages to the input. After fine-tuning with 4k-token windows, the corpus-enabled configuration achieves the lowest perplexity in all twelve evaluated domain–length settings through 32k. At 130M, internal retrieval recovers passkeys in 18 of 20 single-trial settings beyond the training length, com- pared with 10 for Mamba. The results support explicitly retained memory as a complement to recurrent compression for prediction and recall over long contexts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
State Space Models, Long-Context Modeling, Memory-Augmented Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W1jmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the difficulty selective state space models have in recovering details that were compressed out of the recurrent state. It proposes PUM-Net, which augments selected Mamba blocks with two retrieval memories. An internal memory stores representations of completed input chunks and is causally masked, so each prediction only sees earlier context. An external memory stores corpus passages that are encoded offline with frozen models. Each augmented block retrieves a small set of entries from both memories, lets the two retrieved sets attend to each other, and adds the pooled results to the block output through learned gates. Experiments on long-context language modeling, passkey recall, and question answering are conducted for evaluation. The paper reports lower perplexity than the baselines and better passkey recall than Mamba.

Critical strengths
  • Adding causal chunk retrieval to SSM blocks is a reasonable way to access earlier context while keeping the parallel scan.

  • The paper describes the causal availability rule and boundary cases carefully, so the training mask and the streaming insertion rule are consistent.

  • The controlled comparisons that fix the retrieved evidence are a helpful attempt to separate evidence access from the integration mechanism.

Critical weaknesses
  • The technical contribution appears to be mostly a combination of known components. Retrieval of stored representations from earlier context and from an external corpus are both established ideas in retrieval-augmented language models. The interaction module is standard cross-attention followed by gated residual addition, and the paper itself notes that co-attention and gated cross-attention are established mechanisms. The paper does not clearly discuss what is new beyond placing these pieces inside a Mamba block with two separate stores. The main language modeling comparison also includes only plain and attention-augmented Mamba baselines, not existing retrieval-augmented methods or long-context SSM methods.

  • The perplexity gains from external memory may not reflect the proposed mechanism. For math and code, the external stores come from sources that likely overlap with the evaluation data. For example, most of the ProofPile evaluation text likely comes from arXiv mathematics papers, and the external store for ProofPile also includes arXiv mathematics papers. The same test documents may therefore appear in the store. The paper states that document-level overlap is not quantified. These two domains also show the largest gains from external memory, while the gain on books with a Wikipedia store is small. The external values are also produced by a frozen model much larger than the backbone. For math and code, no control uses the same store and encoder with a different integration, and no experiment appends the retrieved text to the context. So it is unclear whether the gains come from the proposed retrieval and interaction design, from data overlap, or from the larger encoder.

  • The ablations and controls appear to rely on weak baselines. The state-addition controls add retrieved states without learned fusion, so they seem easy to beat. In the cross-attention ablation, the simple-gating variant with both memories is worse than the internal-only model at every length, and worse than plain Mamba at some lengths. Its gap to the full model is also much larger than the total gain from adding external memory on the same dataset. It is therefore unclear if the variant is well developed enough to isolate the effect of cross-attention.

  • The evidence for long-context behavior is limited. Perplexity rises sharply at the longest evaluated length for all models, including PUM-Net, so strong degradation at long lengths remains for all configurations. On passkey retrieval beyond the training length, internal retrieval does not outperform an existing Mamba length-extension baseline at the smaller scale. The question-answering scores seem low for all models compared to scores of small models reported in the original benchmark, which may make the differences hard to interpret. The online timing covers a single block, and the paper places external lookup and autoregressive decoding outside the benchmark scope. During decoding, internal search cost grows with the number of stored chunks, so the end-to-end cost remains unclear.

Questions for authors
  • How were the external stores for ProofPile and CodeParrot separated from their test sets?

  • How much of the external-memory gain on ProofPile and CodeParrot does the paper attribute to the retrieval and interaction design, and how much to the larger frozen encoder?

3 comments
AuthorsOpenReview ↗

R1&R2: Memory design, baseline comparisons, and the CodeParrot result

Vanilla Mamba keeps a fixed-size recurrent state and a short convolution buffer during decoding. It has no per-token KV cache to revisit details that have dropped out of that state. PUM-Net stores representations of completed input chunks for causal retrieval. A separate corpus store, encoded offline, supplies evidence that may be absent from the input.

An earlier import can tell the model which API passage applies. AIM lets chunks from each source attend to individual chunks from the other before pooling, so the import can inform each passage separately.

With external memory disabled, PUM-Int lowers PPL relative to vanilla Mamba in 12/12 dataset–length settings under the same fine-tuning budget. Mamba+SWA and Mamba+NSA test local and sparse attention on the same 130M backbone.

At 4k on CodeParrot, PPL drops from 4.47 to 1.11 (75.2%) when moving from PUM-Int to PUM-Full. This measures the combined addition of corpus access, frozen 2.7B value representations, and learned fusion. Related Python code and exact matches to the target are both plausible sources of improvement.

Each store has about 100M tokens, drawn from The Stack's Python files for CodeParrot and arXiv and Wikipedia mathematics for ProofPile. We found no exact full-document matches even after Unicode and whitespace normalization. We found near duplicates for 13/2,000 (0.65%) CodeParrot files and 1/1,000 (0.10%) ProofPile documents, using Jaccard similarity of at least 0.80 over sets of lexical 5-grams.

We checked the final 100 tokens of each audit window to mirror the scoring protocol. A window matches if at least 32 consecutive token IDs also occur in one 64-token external chunk.

ContextCodeParrot matched / eligibleProofPile matched / eligible
4k146/3,380 (4.32%)17/1,430 (1.19%)
8k57/1,221 (4.67%)13/1,221 (1.06%)
16k34/370 (9.19%)2/870 (0.23%)
32k32/130 (24.62%)1/300 (0.33%)

The 32k windows come from 13 CodeParrot files and 30 ProofPile documents. At 4k, 31 CodeParrot windows also match all 100 tail tokens in one external file. These counts show where matching text was available. Its effect on PPL depends on which chunks the model retrieved and how much the loss changed on those targets.

On PG-19 at 4k (Table 11), PPL is 16.69 for Mamba, 17.85 for direct state addition without joint fusion training, and 15.96 for PUM-Full. The latter two share a backbone and encoded store. PUM-Full also adds internal retrieval and learned fusion. The 2.7B encoder's separate contribution on CodeParrot remains unmeasured.

AuthorsOpenReview ↗

R3&R4: Fusion ablation, long-context performance, and decoding cost

Simple gating has higher PPL than PUM-Int at all four PG-19 lengths. Table 12 isolates the effect of adding cross-attention before pooling with both memories, the frozen encoder, and learned gates in place.

ContextPUM-Int PPLSimple gating PPLPUM-Full PPL
4k16.1216.6315.96
8k16.7718.3316.63
16k16.9819.5716.86
32k22.3225.1622.23

At 32k, AIM lowers PPL by 2.93 points relative to simple gating. Of that difference, 2.84 points close the gap to PUM-Int, and 0.09 points (0.4%) are the improvement over internal retrieval alone.

From 16k to 32k, PUM-Full PPL rises from 4.40 to 26.50 on ProofPile and from 4.25 to 22.50 on CodeParrot. In the 130M passkey test, PUM-Int recalls the planted string in 18/20 settings beyond the 4k training length, compared with 19/20 for MambaExtend and 10/20 for Mamba. Across 32k and 64k combined, PUM-Int gets 8/10 and MambaExtend gets 9/10.

Starting from the same instruction-tuned Mamba-2.8B checkpoint, mean F1 across seven LongBench tasks rises from 4.81 for Mamba to 8.12 for PUM-Int and 10.14 for PUM-Full.

We measured decoding on one A100 with inputs of at least 8k tokens before adding evidence. All retrieval variants used the same five 256-token Wikipedia passages per question. Timing starts after prefill, with the selected passages available, and covers generation, including the memory lookups and fusion performed during decoding.

2.8B model2WikiMultihopQA F1HotpotQA F1Decode (ms/token)Peak GPU memory (GB)
Mamba1.180.5312.58.2
Mamba + Text RAG2.521.8514.88.5
PUM-Int + Text RAG3.822.6022.420.6
PUM-Full4.153.1218.18.8

Compared with PUM-Int + Text RAG, which also uses internal retrieval and receives the same external passages, PUM-Full decodes 4.3 ms faster per token, uses 11.8 GB less peak GPU memory, and improves F1 by 0.33 points on 2WikiMultihopQA and 0.52 points on HotpotQA.

Thank you for your response and additional information. However, I still have concerns about the limited novelty of the method and the uneven improvements from external memory across domains. Given the measured overlap and larger frozen encoder, it's still unclear on why we have larger external-memory gains on code. I will thus keep my current scores.

Reviewer_UJQFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Selective state space models' recurrent compression gives no way to revisit information once it has been discarded from the state. The authors propose PUM-Net, a dual-memory extension of Mamba that separates recurrent computation, causal context retention, and offline corpus reuse. Internal retrieval pulls from completed input chunks, while external retrieval pulls from offline-encoded corpus representations, and bidirectional attention refines each set before pooling and gated integration. After fine-tuning with 4k-token windows, the corpus-enabled configuration achieves the lowest perplexity in all twelve evaluated domain–length settings through 32k

Critical strengths
  1. The problem being addressed is clear and meaningful, and the approach using internal and external memory is interesting.
  2. Experimental results are presented across various datasets (PG-19, ProofPile, CodeParrot) and various context length.
  3. Overall, it is written in an easy-to-understand manner.
Critical weaknesses
  1. The justification for going from the problem to the solution is lacking: more reasoning is needed for "why the authors adopted this design.” In particular, is there a reason for using frozen MiniLM embeddings as keys even for internal memory?
  2. Design choices are not supported by ablations. For example, the rationale for choices such as 64-token chunks and k=8 is not justified through ablation studies.
  3. Although long-context is the target, long-context experiments are insufficient (32K is the maximum).
  4. Notation needs to be cleaned up. The same symbol is used with multiple meanings.
  5. Need more concrete efficiency analysis (decoding latency, end-to-end throughput)
Questions for authors
  1. Given that this targets long-context, the experimental context lengths are short. Can Table 1 provide results for lengths >32K (64K, 128K)? And can Table 2 also provide 64K and 128K results?
  2. The chunk encoder and direct state addition do not appear to be defined in the main text. Could the authors clarify the definitions of these two?
7 comments
AuthorsOpenReview ↗

R1 · Why use frozen MiniLM keys for internal memory?

We use frozen MiniLM keys so that every augmented Mamba block can search the same text-based representation of a completed chunk. For each 64-token chunk, MiniLM encodes the chunk’s text into a key used for search. Each Mamba block computes its own value from the chunk’s block-input representations. This value is what the block reads when it retrieves the chunk.

For example, when a later token depends on a definition given earlier in the input, the block forms a query from its current hidden state, scores the keys of completed chunks, and reads its own stored value for the selected chunk. The query projections and value encoders are trainable, so different blocks and token positions can retrieve different chunks and use the retrieved information differently. We use the same frozen encoder for external keys, which also lets us build the corpus index once and reuse it throughout fine-tuning.

AuthorsOpenReview ↗

R2: Chunk length and retrieval width

Shorter chunks let the model retrieve more specific pieces of text, but they also give it more keys to search. A 32,768-token input contains 1,024 chunks at 32 tokens per chunk, 512 at 64, and 256 at 128. We chose 64 tokens to keep the chunks small enough for fine-grained retrieval without doubling the number of keys to score.

In our preliminary runs, fewer than four retrieved entries gave the model too little variety in the evidence. Retrieving more than eight slowed decoding, with no clear gain in F1. Eight entries from each memory give AIM up to 64 candidate pairs per head, per query, in each direction. With 16 entries per memory, that rises to 256. The retrieval width limits how many entries AIM processes after selection. Exact internal search still scores all eligible keys.

Our preliminary tuning guided the choice of 64-token chunks and up to eight retrieved entries per memory. We used this configuration throughout the main PUM-Net experiments.

AuthorsOpenReview ↗

R3: Evaluation at 64k and 128k

Table 1 stops at 32k because perplexity on the final 100 tokens rose sharply at 64k, including for PUM-Net. We do not have language-modeling results at 128k.

We did test passkey recall at 64k. We ran five prompts with the passkey at a different position in each prompt and kept external memory off. Here is how many keys each model recovered.

Model sizeMambaDeciMambaMambaExtendPUM-Int
130M0/52/54/54/5
1.4B0/50/52/53/5
Reviewer_JoPiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds two retrieval memories to selected Mamba blocks. One is an internal store of completed 64-token input chunks, causally masked. The other is an external store of corpus passages pre-encoded by a frozen Mamba-2.7B. The two retrieved sets attend to each other through AIM before pooling and gated addition to the block output. Experiments cover 130M fine-tuning on PG-19, ProofPile, and CodeParrot up to 32k, passkey retrieval up to 64k, task-specific LongBench fine-tuning at 2.8B, and single-block timing.

Critical strengths

The causal rule is handled with care. Section 3.1 spells out exactly when a chunk becomes eligible, and the paper admits exact internal search is not strictly linear. I also appreciate that the authors ran a matched-evidence control: in Table 7, with the same retrieved page and top-8 width, PUM-Full beats direct state addition on all three LB tasks (e.g. 5.18 to 11.46 on 2WikiMultihopQA). This is the remarkable part, because it separates the integration mechanism from simply having more evidence.

Internal-only memory also gives a consistent, if modest, gain over plain Mamba in all twelve Table 1 settings at the same backbone, and the block costs about 1.8x Mamba at 64k (Table 10).

Critical weaknesses

My main concern is the very large external-memory gain on CodeParrot: at 4k, perplexity drops from 4.47 with PUM-INT to 1.11 with PUM-FULL. Since both the external store and CodeParrot contain GitHub code, exact or near-duplicate retrieval is a serious possibility. Without evaluation-set overlap analysis, it is difficult to separate long-context improvement from retrieval of similar code. ProofPile raises a related, though weaker, concern.

The resource comparison is also incomplete. PUM-FULL uses a frozen 2.7B encoder, a roughly 100M-token store, and ANN retrieval, yet efficiency is measured only at the block level. End-to-end latency and memory would be needed to support the efficiency claim.

Finally, the results lack seed variance, and each passkey cell contains only one trial. The QA evaluation also lacks a standard text-concatenation RAG baseline using the same retrieved evidence.

Questions for authors

Clarification: What fraction of the CodeParrot and ProofPile evaluation documents have exact or near-duplicate matches in the external stores?

Clarification: What are the end-to-end latency and memory costs when the encoder, ANN retrieval, and external store are included?

Suggestion: Could you report seed variance and run multiple passkey trials per cell?

Suggestion: Could you compare against a text-concatenation RAG baseline given the same retrieved evidence?

4 comments
AuthorsOpenReview ↗

R1: Matched-evidence Text RAG and decoding measurements

Experimental setup. We evaluate on the 8k+ subsets of 2WikiMultihopQA and HotpotQA. All configurations receive the same original context and question. For each question, every retrieval configuration uses the same five externally retrieved 256-token Wikipedia passages.

  • Mamba-2.8B processes the original input without retrieval, providing a reference for QA performance and decoding cost.
  • Mamba-2.8B + Text RAG appends the five passages as raw text. This is the standard text-concatenation baseline on the same backbone as PUM-Net.
  • Pythia-2.8B + Text RAG appends the same passages and serves as a Transformer reference.
  • PUM-Int-2.8B + Text RAG retains causal retrieval over the original context and appends the five external passages as raw text. This lets us compare the two external evidence interfaces with internal retrieval enabled in both.
  • PUM-Full-2.8B also retains causal internal retrieval. It uses AIM to fuse the retrieved internal representations with the precomputed representations of the same external passages, without appending their text.
Configuration2WikiMultihopQA F1 ↑HotpotQA F1 ↑Decode (ms/token) ↓Peak GPU memory (GB) ↓
Mamba-2.8B1.180.5312.58.2
Mamba-2.8B + Text RAG2.521.8514.88.5
Pythia-2.8B + Text RAG2.851.9529.424.8
PUM-Int-2.8B + Text RAG3.822.6022.420.6
PUM-Full-2.8B (latent fusion)4.153.1218.18.8

Compared with standard Text RAG. With the same Mamba backbone and external passages, PUM-Full improves over Mamba + Text RAG by 1.63 F1 points on 2WikiMultihopQA and 1.27 points on HotpotQA, at a cost of 3.3 ms per generated token and 0.3 GB of peak GPU memory.

With internal retrieval enabled in both systems. PUM-Full improves over PUM-Int + Text RAG by 0.33 and 0.52 F1 points on the two tasks, respectively. It also reduces decoding latency by 4.3 ms/token and peak GPU memory by 11.8 GB. In this setting, integrating the external passages as latent states improves QA scores on both tasks while reducing the measured decoding and memory costs of text concatenation.

AuthorsOpenReview ↗

R2: External-memory construction, block latency, and peak memory

Offline encoding and index construction. We encode the external corpus and build the ANN index before model training. For the approximately 100M-token corpus, Table 8 reports about 6.5 minutes for MiniLM key encoding on one H100, about 10 minutes for value encoding with the frozen Mamba-2.7B encoder on eight H100s, and less than one minute for ANN index construction on eight H100s. These one-time preparation costs are measured separately from model training.

Block timing and peak memory. We apologize that Section 4.4 mentioned internal retrieval and AIM without explicitly stating that the forward timing also includes external retrieval. Within the timed forward pass, the block computes queries for both memories, retrieves the internal candidates, and uses ANN to look up and fetch the selected precomputed external values. AIM then fuses the two sets of retrieved representations. At 64k on one H100, Table 10 and Figure 5(c) report:

BlockForward (ms)Backward (ms)Peak allocated GPU memory (relative to Mamba)
Vanilla Mamba6.0237.771.00×
PUM-Net10.8369.11≈1.30×

Scope of the requested measurement. Could you clarify whether by “end-to-end” you mean the total wall-clock time from offline corpus encoding and ANN index construction through the completion of model training, or the cost of executing a block with retrieval from the prebuilt external store? If you mean the latter, Table 10 and Figure 5(c), reproduced above, already report the block timing and peak GPU memory for that setting.

AuthorsOpenReview ↗

R3: Test-set overlap with the external corpora

We checked each test set against the external store used in our experiments. Each store contains about 100M tokens. We found no exact full-document matches in any of the three datasets.

DatasetExact matches / test documentsDetected near duplicates / test documents
CodeParrot0/2,000 (0%)13/2,000 (0.65%)
ProofPile0/1,000 (0%)1/1,000 (0.10%)
PG-190/100 (0%)0/100 (0%)

For exact matches, we compared SHA-256 hashes of the original text and of a normalized version. For normalization, we applied Unicode NFC, collapsed consecutive whitespace to one space, and trimmed whitespace at both ends. For the remaining documents, we used hashes of five consecutive lexical units (words, code identifiers, numbers, or punctuation) to find candidate pairs. We then computed case-sensitive Jaccard similarity over the full sets of distinct five-unit sequences, ignoring whitespace. We counted a test document once if any candidate reached 0.80.

A document can contain a matching passage even if the rest of it is different. We therefore also checked the final 100 tokens of each window, which are the tokens used to compute perplexity. We tokenized both corpora with the same GPT-NeoX tokenizer and split the external documents into nonoverlapping 64-token passages. We counted a window as a match if at least 32 consecutive token IDs from its scored suffix appeared in a single external passage.

Context length (tokens)CodeParrot: matched / eligible windowsProofPile: matched / eligible windowsPG-19: matched / eligible windows
4,096146/3,380 (4.32%)17/1,430 (1.19%)0/990 (0%)
8,19257/1,221 (4.67%)13/1,221 (1.06%)0/950 (0%)
16,38434/370 (9.19%)2/870 (0.23%)0/940 (0%)
32,76832/130 (24.62%)1/300 (0.33%)0/780 (0%)

At each context length, we skipped documents shorter than the window. We used ten maximally spaced windows per eligible document when there were enough distinct start positions, and one window otherwise. At 32,768 tokens, the windows came from 13 CodeParrot files, 30 ProofPile documents, and 78 PG-19 books. The counts show how often matching text was present in the store. How often the model actually retrieved those passages during evaluation remains an open question.

Thanks for the additional experiments. The Text RAG and efficiency results are helpful. However, the large CodeParrot gain is still difficult to interpret because matched text reaches 24.62% at 32k, and the effect of those matches was not measured. I will retain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.