- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the difficulty selective state space models have in recovering details that were compressed out of the recurrent state. It proposes PUM-Net, which augments selected Mamba blocks with two retrieval memories. An internal memory stores representations of completed input chunks and is causally masked, so each prediction only sees earlier context. An external memory stores corpus passages that are encoded offline with frozen models. Each augmented block retrieves a small set of entries from both memories, lets the two retrieved sets attend to each other, and adds the pooled results to the block output through learned gates. Experiments on long-context language modeling, passkey recall, and question answering are conducted for evaluation. The paper reports lower perplexity than the baselines and better passkey recall than Mamba.
Critical strengths
-
Adding causal chunk retrieval to SSM blocks is a reasonable way to access earlier context while keeping the parallel scan.
-
The paper describes the causal availability rule and boundary cases carefully, so the training mask and the streaming insertion rule are consistent.
-
The controlled comparisons that fix the retrieved evidence are a helpful attempt to separate evidence access from the integration mechanism.
Critical weaknesses
-
The technical contribution appears to be mostly a combination of known components. Retrieval of stored representations from earlier context and from an external corpus are both established ideas in retrieval-augmented language models. The interaction module is standard cross-attention followed by gated residual addition, and the paper itself notes that co-attention and gated cross-attention are established mechanisms. The paper does not clearly discuss what is new beyond placing these pieces inside a Mamba block with two separate stores. The main language modeling comparison also includes only plain and attention-augmented Mamba baselines, not existing retrieval-augmented methods or long-context SSM methods.
-
The perplexity gains from external memory may not reflect the proposed mechanism. For math and code, the external stores come from sources that likely overlap with the evaluation data. For example, most of the ProofPile evaluation text likely comes from arXiv mathematics papers, and the external store for ProofPile also includes arXiv mathematics papers. The same test documents may therefore appear in the store. The paper states that document-level overlap is not quantified. These two domains also show the largest gains from external memory, while the gain on books with a Wikipedia store is small. The external values are also produced by a frozen model much larger than the backbone. For math and code, no control uses the same store and encoder with a different integration, and no experiment appends the retrieved text to the context. So it is unclear whether the gains come from the proposed retrieval and interaction design, from data overlap, or from the larger encoder.
-
The ablations and controls appear to rely on weak baselines. The state-addition controls add retrieved states without learned fusion, so they seem easy to beat. In the cross-attention ablation, the simple-gating variant with both memories is worse than the internal-only model at every length, and worse than plain Mamba at some lengths. Its gap to the full model is also much larger than the total gain from adding external memory on the same dataset. It is therefore unclear if the variant is well developed enough to isolate the effect of cross-attention.
-
The evidence for long-context behavior is limited. Perplexity rises sharply at the longest evaluated length for all models, including PUM-Net, so strong degradation at long lengths remains for all configurations. On passkey retrieval beyond the training length, internal retrieval does not outperform an existing Mamba length-extension baseline at the smaller scale. The question-answering scores seem low for all models compared to scores of small models reported in the original benchmark, which may make the differences hard to interpret. The online timing covers a single block, and the paper places external lookup and autoregressive decoding outside the benchmark scope. During decoding, internal search cost grows with the number of stored chunks, so the end-to-end cost remains unclear.
Questions for authors
-
How were the external stores for ProofPile and CodeParrot separated from their test sets?
-
How much of the external-memory gain on ProofPile and CodeParrot does the paper attribute to the retrieval and interaction design, and how much to the larger frozen encoder?
3 comments
R3&R4: Fusion ablation, long-context performance, and decoding cost
Simple gating has higher PPL than PUM-Int at all four PG-19 lengths. Table 12 isolates the effect of adding cross-attention before pooling with both memories, the frozen encoder, and learned gates in place.
| Context | PUM-Int PPL | Simple gating PPL | PUM-Full PPL |
|---|---|---|---|
| 4k | 16.12 | 16.63 | 15.96 |
| 8k | 16.77 | 18.33 | 16.63 |
| 16k | 16.98 | 19.57 | 16.86 |
| 32k | 22.32 | 25.16 | 22.23 |
At 32k, AIM lowers PPL by 2.93 points relative to simple gating. Of that difference, 2.84 points close the gap to PUM-Int, and 0.09 points (0.4%) are the improvement over internal retrieval alone.
From 16k to 32k, PUM-Full PPL rises from 4.40 to 26.50 on ProofPile and from 4.25 to 22.50 on CodeParrot. In the 130M passkey test, PUM-Int recalls the planted string in 18/20 settings beyond the 4k training length, compared with 19/20 for MambaExtend and 10/20 for Mamba. Across 32k and 64k combined, PUM-Int gets 8/10 and MambaExtend gets 9/10.
Starting from the same instruction-tuned Mamba-2.8B checkpoint, mean F1 across seven LongBench tasks rises from 4.81 for Mamba to 8.12 for PUM-Int and 10.14 for PUM-Full.
We measured decoding on one A100 with inputs of at least 8k tokens before adding evidence. All retrieval variants used the same five 256-token Wikipedia passages per question. Timing starts after prefill, with the selected passages available, and covers generation, including the memory lookups and fusion performed during decoding.
| 2.8B model | 2WikiMultihopQA F1 | HotpotQA F1 | Decode (ms/token) | Peak GPU memory (GB) |
|---|---|---|---|---|
| Mamba | 1.18 | 0.53 | 12.5 | 8.2 |
| Mamba + Text RAG | 2.52 | 1.85 | 14.8 | 8.5 |
| PUM-Int + Text RAG | 3.82 | 2.60 | 22.4 | 20.6 |
| PUM-Full | 4.15 | 3.12 | 18.1 | 8.8 |
Compared with PUM-Int + Text RAG, which also uses internal retrieval and receives the same external passages, PUM-Full decodes 4.3 ms faster per token, uses 11.8 GB less peak GPU memory, and improves F1 by 0.33 points on 2WikiMultihopQA and 0.52 points on HotpotQA.
Thank you for your response and additional information. However, I still have concerns about the limited novelty of the method and the uneven improvements from external memory across domains. Given the measured overlap and larger frozen encoder, it's still unclear on why we have larger external-memory gains on code. I will thus keep my current scores.
R1&R2: Memory design, baseline comparisons, and the CodeParrot result
Vanilla Mamba keeps a fixed-size recurrent state and a short convolution buffer during decoding. It has no per-token KV cache to revisit details that have dropped out of that state. PUM-Net stores representations of completed input chunks for causal retrieval. A separate corpus store, encoded offline, supplies evidence that may be absent from the input.
An earlier import can tell the model which API passage applies. AIM lets chunks from each source attend to individual chunks from the other before pooling, so the import can inform each passage separately.
With external memory disabled, PUM-Int lowers PPL relative to vanilla Mamba in 12/12 dataset–length settings under the same fine-tuning budget. Mamba+SWA and Mamba+NSA test local and sparse attention on the same 130M backbone.
At 4k on CodeParrot, PPL drops from 4.47 to 1.11 (75.2%) when moving from PUM-Int to PUM-Full. This measures the combined addition of corpus access, frozen 2.7B value representations, and learned fusion. Related Python code and exact matches to the target are both plausible sources of improvement.
Each store has about 100M tokens, drawn from The Stack's Python files for CodeParrot and arXiv and Wikipedia mathematics for ProofPile. We found no exact full-document matches even after Unicode and whitespace normalization. We found near duplicates for 13/2,000 (0.65%) CodeParrot files and 1/1,000 (0.10%) ProofPile documents, using Jaccard similarity of at least 0.80 over sets of lexical 5-grams.
We checked the final 100 tokens of each audit window to mirror the scoring protocol. A window matches if at least 32 consecutive token IDs also occur in one 64-token external chunk.
The 32k windows come from 13 CodeParrot files and 30 ProofPile documents. At 4k, 31 CodeParrot windows also match all 100 tail tokens in one external file. These counts show where matching text was available. Its effect on PPL depends on which chunks the model retrieved and how much the loss changed on those targets.
On PG-19 at 4k (Table 11), PPL is 16.69 for Mamba, 17.85 for direct state addition without joint fusion training, and 15.96 for PUM-Full. The latter two share a backbone and encoded store. PUM-Full also adds internal retrieval and learned fusion. The 2.7B encoder's separate contribution on CodeParrot remains unmeasured.