- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DART, which augments Mamba-2 with state-memory attention over retained chunk state contributions. For each input token, DART decodes token-conditioned keys and values from these compressed memories, attends over historical chunks, and integrates the retrieved output through a gated residual connection. The method reuses Mamba-2’s chunkwise computation and implements retrieval with a FlashAttention-style kernel. Experiments include MQAR, 100B-token Pile pretraining at three model scales up to approximately 795M parameters, downstream evaluations, retrieval benchmarks, and kernel/cache measurements. The results demonstrate substantial improvements over Mamba-2 on associative recall and several extraction tasks while largely preserving language-modeling quality. The paper also establishes a 75% reduction in the length-dependent cache relative to its matched attention baseline under the default configuration.
Critical strengths
1. A coherent integration of recurrent compression and attention-based retrieval. DART uses Mamba-2’s chunk state contributions as a shared memory representation from which both keys and values are decoded. This provides a concrete mechanism for making compressed recurrent memories explicitly retrievable. Reusing the existing chunkwise computation and implementing SMA with an IO-aware kernel makes the proposal more substantial than an architectural sketch.
2. Meaningful retrieval improvements beyond synthetic associative recall. The gains are not limited to MQAR. At the largest evaluated scale and the default chunk size, DART improves SWDE accuracy from 39.9% to 56.5% and FDA accuracy from 17.2% to 28.9% over Mamba-2, while largely maintaining Pile perplexity and general downstream performance. The SMA ablations further show that the trained model relies on the retrieval branch for these capabilities. These results support the usefulness of the proposed memory-access mechanism.
Critical weaknesses
1. The kernel benchmarks do not establish end-to-end efficiency. Figure 5 measures isolated forward kernels with precomputed inputs, including chunk memories for SMA. Full DART execution also includes the recurrent path, projections, and state construction and management. It remains unclear how the kernel results translate into prefill and decoding performance. In particular, the additional cost over Mamba-2 needs to be measured to judge whether the retrieval gains justify it.
2. The memory cost of the chunk-size improvements is not sufficiently characterized. The default cache reduction is supported, but several higher retrieval scores use smaller chunks. Under the paper’s matched attention accounting, with , the length-dependent cache ratio increases from 25% at to 100% at and 200% at . These ratios do not represent total GPU memory usage. The approximately cache-matched Attamba comparison is useful. However, the absence of total inference memory measurements across chunk settings makes it difficult to judge the practical cost of the additional retrieval gains. It leaves the choice between higher accuracy and lower memory insufficiently evaluated.
3. The retrieval advantage does not persist on the harder 4K tasks. For DART 795M at , NIAH-Single-2 accuracy falls from 70.0% at 2K to 1.4% at 4K, and Single-3 falls from 43.4% to 2.6%. Its advantage over Mamba-2 disappears in these settings, and smaller chunks do not resolve the failure. Single-1 remains strong, so this is a task-dependent limitation. Since training uses 2K contexts, the results do not distinguish length-extrapolation failure from difficulty retrieving over more chunk memories. Understanding this distinction matters for the paper’s long-context motivation. The kernel benchmarks extending to 64K provide runtime evidence, but do not establish retrieval quality at those lengths.
Questions for authors
1. End-to-end execution cost: Could you report complete-model prefill time and decoding latency/throughput across several context lengths and batch sizes, with hardware and precision held fixed? I would particularly like to understand the additional cost over Mamba-2 and whether the kernel-level advantage over attention remains in full-model execution.
2. Retrieval quality versus memory: Could you report retrieval accuracy alongside measured peak inference memory for the existing (S=256,64,32) configurations, separating persistent cache from other memory costs? Which settings provide better retrieval at comparable memory budgets? This would make the practical implications of the chunk-size experiments clearer.
3. The 4K degradation: What explains the sharp decline on NIAH-Single-2/3 at 4K? Could a length sweep with controlled needle placement and distractor construction show whether the advantage decreases gradually or fails abruptly beyond the training length? Results with an extended training context, if available, would help distinguish the causes. Comparisons with attention-based models should also account for the positional-extrapolation limitations discussed in Appendix D.3.
4. Larger model: Evaluating a larger DART model, for example above 1B parameters, would help establish whether the retrieval–resource tradeoff persists with scale. I view this as a useful extension rather than a required new pretraining run for the rebuttal.
3 comments
Response to Reviewer TxJw (2/2)
What explains the sharp decline on NIAH-Single-2/3 at 4K? Could a length sweep with controlled needle placement and distractor construction show whether the advantage decreases gradually or fails abruptly beyond the training length?
To examine this decline, we randomly select 100 examples per task that DART answered correctly in the 2K evaluations of NIAH-S-2 and NIAH-S-3. We insert specified numbers of natural tokens or repeated background tokens before the question, keeping the original context and question unchanged. All entries report accuracy (%).
Table 13. NIAH accuracy under context extension (%).
| Task | Appended tokens | Natural tokens (%) | Repeated background tokens (%) |
|---|---|---|---|
| NIAH-Single-2 | 512 | 97 | 100 |
| NIAH-Single-2 | 640 | 0 | 100 |
| NIAH-Single-2 | 1024 | 0 | 99 |
| NIAH-Single-2 | 1536 | 0 | 61 |
| NIAH-Single-3 | 512 | 87 | 98 |
| NIAH-Single-3 | 640 | 0 | 98 |
| NIAH-Single-3 | 1024 | 0 | 91 |
| NIAH-Single-3 | 1536 | 0 | 51 |
Under the tested natural continuations, retention falls sharply between +512 and +640 tokens. Repeated background substantially mitigates this decline, indicating a strong dependence on background content.
We further ablate the SSM and SMA branches by replacing either branch’s output with its corresponding output before context extension, across all 48 layers and at every token position from the start of the question through autoregressive generation.
Table 14. NIAH accuracy with restored branch outputs (%).
| Task | Appended natural tokens | Replace SSM output | Replace SMA output |
|---|---|---|---|
| NIAH-Single-2 | 512 | 98 | 98 |
| NIAH-Single-2 | 640 | 99 | 0 |
| NIAH-Single-2 | 1024 | 98 | 0 |
| NIAH-Single-2 | 1536 | 96 | 0 |
| NIAH-Single-3 | 512 | 98 | 81 |
| NIAH-Single-3 | 640 | 98 | 0 |
| NIAH-Single-3 | 1024 | 98 | 0 |
| NIAH-Single-3 | 1536 | 97 | 0 |
These results suggest that the SSM and SMA branches play asymmetric roles in retrieval degradation under context extension. Restoring the SSM branch output largely recovers retrieval accuracy (96–99% at +640 tokens and above), while restoring the SMA branch output alone yields 0%. Prior work also reports substantial performance degradation in Mamba-2 beyond its training context length [1]. We consider improving DART's length generalization an important direction for future work, including exploring its combination with Gated DeltaNet (GDN) [2]. We thank the reviewer for this important observation.
Evaluating a larger DART model, for example above 1B parameters, would help establish whether the retrieval–resource tradeoff persists with scale.
We thank the reviewer for this suggestion and agree that evaluating DART beyond 1B parameters would provide valuable evidence on how the retrieval–resource tradeoff scales with model size. We consider this an important direction for future work.
Reference
[1] Ricardo Buitrago Ruiz and Albert Gu. Understanding and Improving Length Generalization in Recurrent Models. 2025.
[2] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025.
Thanks for the detailed response and additional experiments. The efficiency results are helpful, and I appreciate the follow-up analysis of why retrieval performance drops at longer contexts. With these clarifications, I’m raising my score from 2 to 3.
Response to Reviewer TxJw (1/2)
We sincerely appreciate the reviewer’s time and constructive suggestions on complete-model efficiency and retrieval capabilities. We now address the questions and suggestions below.
We thank the reviewer for this important question. Quantifying the additional cost over Mamba-2 and establishing whether DART’s advantage over attention persists in full-model execution are essential for assessing its practical efficiency.
In the General Response, Tables 2–3 report full-model latency across seven context-length and batch-size settings, and Table 4 reports persistent cache and peak memory. At 64K, DART achieves 1.44× prefill and 2.42× prefill + decode speedups over Transformer. DART incurs additional cost over Mamba-2, but the increases are substantially smaller than the corresponding gaps between Mamba-2 and Transformer (Tables 3–4). We will include these results in the revised paper and hope that these experiments address the reviewer’s concerns about complete-model efficiency.
We measured inference memory for all three pretrained checkpoints at S=256,64,32. We report memory measurements with a context length of 2048 tokens and batch size 1. Here "cache" denotes historical KV cache, "weights" denotes weight memory, and "peak" denotes peak allocated GPU memory including KV historical cache, model weights, and other persistent cache. All measurements are in GiB.
Table 12. Retrieval accuracy and inference memory across chunk sizes.
At a fixed context length, historical cache scales approximately as 1/S. Reducing S from 256 to 64 substantially improves both SWDE and FDA accuracy with moderate increases in peak memory. Further reducing S to 32 yields smaller, less consistent gains on SWDE but more substantial gains on FDA, suggesting that tasks benefit differently from finer-grained retrieval. This second reduction also incurs a larger absolute increase in peak memory. These results illustrate the retrieval–memory trade-off in DART's combination of recurrent compression and attention-style retrieval.