acceptodds
Under review as a conference paper at ICLR 2027

DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling

Abstract

Modern language models are built primarily from Transformers, recurrent models, and their hybrid architectures. Transformers rely on token-level attention memories, while recurrent models such as state space models (SSMs) and linear attention maintain compact recurrent states. These architectures are typically instantiated separately or interleaved at the layer level, leaving open whether a shared memory representation can support both recurrent compression and attention-style retrieval. We study this question through the state space duality (SSD) view of Mamba-2, where the SSM state can be interpreted as a compressed associative key–value (KV) cache. We observe that Mamba-2 decodes token-conditioned values from this state but does not decode token-conditioned keys. Based on this observation, we propose DART (Decoded Attention over Recurrent sTates), which retains the chunk state contributions produced by Mamba-2's chunkwise SSD computation as chunk state memories, decodes token-conditioned keys and values from these memories, and performs state-memory attention (SMA) over the resulting KV pairs. The retrieved output is then combined with the native Mamba-2 output through a gated residual connection. DART supports practical training by reusing the Mamba-2 chunkwise SSD computation and implementing SMA as a FlashAttention-style computation. Our analysis and experiments show that DART substantially reduces the inference cache compared with a matched attention baseline (e.g., savings when the chunk size is and the state size is ). Compared with Mamba-2, DART substantially improves associative recall and retrieval while preserving general language-modeling quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hybrid Models, Language Modeling, Recurrent Models, State Space Models, Long-Context Modeling, Attention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TxJwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DART, which augments Mamba-2 with state-memory attention over retained chunk state contributions. For each input token, DART decodes token-conditioned keys and values from these compressed memories, attends over historical chunks, and integrates the retrieved output through a gated residual connection. The method reuses Mamba-2’s chunkwise computation and implements retrieval with a FlashAttention-style kernel. Experiments include MQAR, 100B-token Pile pretraining at three model scales up to approximately 795M parameters, downstream evaluations, retrieval benchmarks, and kernel/cache measurements. The results demonstrate substantial improvements over Mamba-2 on associative recall and several extraction tasks while largely preserving language-modeling quality. The paper also establishes a 75% reduction in the length-dependent cache relative to its matched attention baseline under the default configuration.

Critical strengths

1. A coherent integration of recurrent compression and attention-based retrieval. DART uses Mamba-2’s chunk state contributions as a shared memory representation from which both keys and values are decoded. This provides a concrete mechanism for making compressed recurrent memories explicitly retrievable. Reusing the existing chunkwise computation and implementing SMA with an IO-aware kernel makes the proposal more substantial than an architectural sketch.

2. Meaningful retrieval improvements beyond synthetic associative recall. The gains are not limited to MQAR. At the largest evaluated scale and the default chunk size, DART improves SWDE accuracy from 39.9% to 56.5% and FDA accuracy from 17.2% to 28.9% over Mamba-2, while largely maintaining Pile perplexity and general downstream performance. The SMA ablations further show that the trained model relies on the retrieval branch for these capabilities. These results support the usefulness of the proposed memory-access mechanism.

Critical weaknesses

1. The kernel benchmarks do not establish end-to-end efficiency. Figure 5 measures isolated forward kernels with precomputed inputs, including chunk memories for SMA. Full DART execution also includes the recurrent path, projections, and state construction and management. It remains unclear how the kernel results translate into prefill and decoding performance. In particular, the additional cost over Mamba-2 needs to be measured to judge whether the retrieval gains justify it.

2. The memory cost of the chunk-size improvements is not sufficiently characterized. The default cache reduction is supported, but several higher retrieval scores use smaller chunks. Under the paper’s matched attention accounting, with N=128N=128, the length-dependent cache ratio increases from 25% at S=256S=256 to 100% at S=64S=64 and 200% at S=32S=32. These ratios do not represent total GPU memory usage. The approximately cache-matched Attamba comparison is useful. However, the absence of total inference memory measurements across chunk settings makes it difficult to judge the practical cost of the additional retrieval gains. It leaves the choice between higher accuracy and lower memory insufficiently evaluated.

3. The retrieval advantage does not persist on the harder 4K tasks. For DART 795M at S=256S=256, NIAH-Single-2 accuracy falls from 70.0% at 2K to 1.4% at 4K, and Single-3 falls from 43.4% to 2.6%. Its advantage over Mamba-2 disappears in these settings, and smaller chunks do not resolve the failure. Single-1 remains strong, so this is a task-dependent limitation. Since training uses 2K contexts, the results do not distinguish length-extrapolation failure from difficulty retrieving over more chunk memories. Understanding this distinction matters for the paper’s long-context motivation. The kernel benchmarks extending to 64K provide runtime evidence, but do not establish retrieval quality at those lengths.

Questions for authors

1. End-to-end execution cost: Could you report complete-model prefill time and decoding latency/throughput across several context lengths and batch sizes, with hardware and precision held fixed? I would particularly like to understand the additional cost over Mamba-2 and whether the kernel-level advantage over attention remains in full-model execution.

2. Retrieval quality versus memory: Could you report retrieval accuracy alongside measured peak inference memory for the existing (S=256,64,32) configurations, separating persistent cache from other memory costs? Which settings provide better retrieval at comparable memory budgets? This would make the practical implications of the chunk-size experiments clearer.

3. The 4K degradation: What explains the sharp decline on NIAH-Single-2/3 at 4K? Could a length sweep with controlled needle placement and distractor construction show whether the advantage decreases gradually or fails abruptly beyond the training length? Results with an extended training context, if available, would help distinguish the causes. Comparisons with attention-based models should also account for the positional-extrapolation limitations discussed in Appendix D.3.

4. Larger model: Evaluating a larger DART model, for example above 1B parameters, would help establish whether the retrieval–resource tradeoff persists with scale. I view this as a useful extension rather than a required new pretraining run for the rebuttal.

3 comments
AuthorsOpenReview ↗

Response to Reviewer TxJw (1/2)

We sincerely appreciate the reviewer’s time and constructive suggestions on complete-model efficiency and retrieval capabilities. We now address the questions and suggestions below.

Could you report complete-model prefill time and decoding latency/throughput across several context lengths and batch sizes, with hardware and precision held fixed? I would particularly like to understand the additional cost over Mamba-2 and whether the kernel-level advantage over attention remains in full-model execution.

We thank the reviewer for this important question. Quantifying the additional cost over Mamba-2 and establishing whether DART’s advantage over attention persists in full-model execution are essential for assessing its practical efficiency.

In the General Response, Tables 2–3 report full-model latency across seven context-length and batch-size settings, and Table 4 reports persistent cache and peak memory. At 64K, DART achieves 1.44× prefill and 2.42× prefill + decode speedups over Transformer. DART incurs additional cost over Mamba-2, but the increases are substantially smaller than the corresponding gaps between Mamba-2 and Transformer (Tables 3–4). We will include these results in the revised paper and hope that these experiments address the reviewer’s concerns about complete-model efficiency.

Could you report retrieval accuracy alongside measured peak inference memory for the existing (S=256,64,32) configurations, separating persistent cache from other memory costs?

We measured inference memory for all three pretrained checkpoints at S=256,64,32S=256,64,32. We report memory measurements with a context length of 2048 tokens and batch size 1. Here "cache" denotes historical KV cache, "weights" denotes weight memory, and "peak" denotes peak allocated GPU memory including KV historical cache, model weights, and other persistent cache. All measurements are in GiB.

Table 12. Retrieval accuracy and inference memory across chunk sizes.

ModelSSSWDE (%)FDA (%)CacheWeightsPeak
DART 133M25634.9219.690.070.250.41
DART 133M6444.8226.230.290.250.64
DART 133M3246.0829.040.580.250.95
DART 380M25646.8931.310.190.701.04
DART 380M6456.7145.830.770.701.63
DART 380M3256.1750.361.550.702.43
DART 795M25656.5328.860.291.482.00
DART 795M6463.7341.381.161.482.89
DART 795M3265.9847.732.321.484.08

At a fixed context length, historical cache scales approximately as 1/S1/S. Reducing SS from 256 to 64 substantially improves both SWDE and FDA accuracy with moderate increases in peak memory. Further reducing SS to 32 yields smaller, less consistent gains on SWDE but more substantial gains on FDA, suggesting that tasks benefit differently from finer-grained retrieval. This second reduction also incurs a larger absolute increase in peak memory. These results illustrate the retrieval–memory trade-off in DART's combination of recurrent compression and attention-style retrieval.

AuthorsOpenReview ↗

Response to Reviewer TxJw (2/2)

What explains the sharp decline on NIAH-Single-2/3 at 4K? Could a length sweep with controlled needle placement and distractor construction show whether the advantage decreases gradually or fails abruptly beyond the training length?

To examine this decline, we randomly select 100 examples per task that DART answered correctly in the 2K evaluations of NIAH-S-2 and NIAH-S-3. We insert specified numbers of natural tokens or repeated background tokens before the question, keeping the original context and question unchanged. All entries report accuracy (%).

Table 13. NIAH accuracy under context extension (%).

TaskAppended tokensNatural tokens (%)Repeated background tokens (%)
NIAH-Single-251297100
NIAH-Single-26400100
NIAH-Single-21024099
NIAH-Single-21536061
NIAH-Single-35128798
NIAH-Single-3640098
NIAH-Single-31024091
NIAH-Single-31536051

Under the tested natural continuations, retention falls sharply between +512 and +640 tokens. Repeated background substantially mitigates this decline, indicating a strong dependence on background content.

We further ablate the SSM and SMA branches by replacing either branch’s output with its corresponding output before context extension, across all 48 layers and at every token position from the start of the question through autoregressive generation.

Table 14. NIAH accuracy with restored branch outputs (%).

TaskAppended natural tokensReplace SSM outputReplace SMA output
NIAH-Single-25129898
NIAH-Single-2640990
NIAH-Single-21024980
NIAH-Single-21536960
NIAH-Single-35129881
NIAH-Single-3640980
NIAH-Single-31024980
NIAH-Single-31536970

These results suggest that the SSM and SMA branches play asymmetric roles in retrieval degradation under context extension. Restoring the SSM branch output largely recovers retrieval accuracy (96–99% at +640 tokens and above), while restoring the SMA branch output alone yields 0%. Prior work also reports substantial performance degradation in Mamba-2 beyond its training context length [1]. We consider improving DART's length generalization an important direction for future work, including exploring its combination with Gated DeltaNet (GDN) [2]. We thank the reviewer for this important observation.

Evaluating a larger DART model, for example above 1B parameters, would help establish whether the retrieval–resource tradeoff persists with scale.

We thank the reviewer for this suggestion and agree that evaluating DART beyond 1B parameters would provide valuable evidence on how the retrieval–resource tradeoff scales with model size. We consider this an important direction for future work.

Reference

[1] Ricardo Buitrago Ruiz and Albert Gu. Understanding and Improving Length Generalization in Recurrent Models. 2025.

[2] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025.

Thanks for the detailed response and additional experiments. The efficiency results are helpful, and I appreciate the follow-up analysis of why retrieval performance drops at longer contexts. With these clarifications, I’m raising my score from 2 to 3.

Reviewer_MhDdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DART, which augments Mamba-2 with an attention-style branch that lets each token retrieve information from compressed chunk-level states. The retrieved information is added to the original Mamba-2 output through a gate. The method is evaluated on MQAR, 100B-token Pile pretraining at three model scales, and retrieval benchmarks. The results show substantial retrieval improvements over Mamba-2 while preserving language-modeling performance, along with lower cache usage than token-level attention.

Critical strengths

The paper presents an improvement to Mamba-2 that addresses the limited recall of its compressed recurrent state. DART keeps chunk-level states from Mamba-2 and lets each token retrieve from them using keys and values decoded for that token. The design builds on Mamba-2's chunkwise computation and is straightforward to understand. Empirically, DART improves SWDE and FDA over Mamba-2 across three model scales. It also achieves strong MQAR results while largely preserving language-modeling performance.

Critical weaknesses
  1. The ablation studies do not fully establish the contribution of dynamic key decoding. More targeted ablations would help clarify the effectiveness of this design.

  2. The paper provides cache analysis and kernel-level benchmarks, but lacks end-to-end inference speed and full-model memory comparisons to demonstrate its practical efficiency.

Questions for authors

Could the authors provide additional ablation studies to clarify the contribution of the proposed key-decoding mechanism?

2 comments
AuthorsOpenReview ↗

Response to Reviewer MhDd

We are grateful for the reviewer’s positive evaluation of our work and constructive suggestions. We hope to answer the questions and suggestions posed by the reviewer below.

The ablation studies do not fully establish the contribution of dynamic key decoding. More targeted ablations would help clarify the effectiveness of this design.

Could the authors provide additional ablation studies to clarify the contribution of the proposed key-decoding mechanism?

We appreciate the reviewer’s important question about the effectiveness of DART’s dynamic key decoding. We added two targeted MQAR ablations: Uniform assigns equal weights to all completed historical chunks, and Shared E replaces the token-dependent key-decoding vector EtE_t with a learnable vector shared across tokens within each layer. All runs follow the original 32-epoch protocol with L=1024L=1024, N=S=16N=S=16, and learning rate 10−310^{-3}. The table reports test accuracy (%).

Table 11. Dynamic key-decoding ablations: MQAR test accuracy (%).

Model widthOriginal DARTUniformShared E
3295.948.9865.69
6499.8816.3596.26
12899.9812.5789.61

Both original DART and Shared E substantially outperform Uniform, highlighting the importance of selectively retrieving relevant historical chunks. Original DART also consistently outperforms Shared E, with a particularly large gain of 30.25 percentage points at dmodel=32d_{\mathrm{model}}=32. At larger widths, DART achieves near-perfect accuracy, supporting the contribution of dynamic key decoding across model widths.

The paper provides cache analysis and kernel-level benchmarks, but lacks end-to-end inference speed and full-model memory comparisons to demonstrate its practical efficiency.

We thank the reviewer for this important suggestion. We agree that end-to-end inference speed and full-model memory comparisons are essential for assessing DART’s practical efficiency. In the General Response, Tables 2–3 report full-model prefill and prefill + decode latency, while Table 4 reports persistent cache and peak memory. At 64K, DART achieves 1.44× prefill and 2.42× prefill + decode speedups over Transformer, with peak memory of 10.36 GiB versus 22.22 GiB. Persistent cache is reduced by 67.85–74.12% across the tested settings. These results demonstrate that DART's kernel-level efficiency advantages can also translate into substantial full-model gains. We will include these results into the revised manuscript.

The authors' response helps clarify the benefit of dynamic key decoding, and the full-model benchmarks provide the practical efficiency evidence I was looking for. I appreciate the authors' efforts to address these points. My main concerns are now resolved, and I will keep my rating at 3.

Reviewer_Ddm7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to add a state-memory attention (SMA) branch to every Mamba-2 block. DART keeps the per-chunk state contributions, which chunkwise SSD already computes, and treats them as memories. SMA is implemented as a FlashAttention-style kernel with on-chip recomputation. Overall i think it is a simple, well-motivated design, but lack important baselines in the experimental results.

Critical strengths
  • DART proposed in the paper is simple and well-motivated. Using chunk state contribution from SSM branch, DART adds attention style access to history with under 3% extra parameters. And the zero-initialized gate makes SMA a clean residual on top of an unmodified Mamba-2 block.
  • The paper proposes a reasonable remedy to address the recall issue of SSMs. SMA reads chunk contributions without the inter-chunk decay , so old chunks do not fade.
  • The paper pretrains at three scales on 100B tokens, reimplements an Attamba baseline, and evaluates on a broad retrieval suite.
  • The authors proposed a practical kernel reported measured runtime and cache allocation.
  • I generally find the paper's writing is clear and easy to follow. At the same time being candid about some limitations, e.g. that most zero-shot inputs are shorter than one chunk.
Critical weaknesses
  • in the abstract, the paper claims 75% cache savings when the chunk size is S = 256 and the state size is N = 128. However, this is only comparing 1 layer. From table 8 and table 9, with comparable parameter amount, DART uses twice number of layers. Hence the cache savings should be presented at the model level not the per layer level. It would only be convincing if the authors report total cache, FLOPs and decode throughput against a parameter-matched full attn Transformer.
  • Same as the previous point, the author should consider report model level speed and inference cache for figure 5.
  • Also for figure 5, if i am reading correctly the SMA kernel takes >10× long as SSD, which essentially means DART almost gives up Mamba-2's linear-time benefit.
  • the paper lacks proper comparison to hybrid architecture such as JAMBA, SAMBA, including retrieval performance, latency, and inference cache.
Questions for authors
  1. How does DART compare with hybrid architecture models, whose total KV cache equals DART's?
  2. Can you report model-level cache, FLOPs and decode throughput against a parameter-matched transformer, and revise the 75% claim if needed?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Ddm7 (1/2)

We thank the reviewer for recognizing DART as "simple and well-motivated" and for the insightful comments and suggestions. The reviewer raises important questions about model-level efficiency, KV-cache comparisons, and the relationship between DART and hybrid architectures. We hope the following responses clarify these points and address the concerns.

in the abstract, the paper claims 75% cache savings when the chunk size is S=256S=256 and the state size is N=128N=128. However, this is only comparing 1 layer.

We agree that the scope of the 75% cache saving should be clarified. DART is proposed as a sequence modeling mechanism that can be incorporated into different model architectures. For each head, DART uses recurrent compression to reduce the historical KV cache to (N/(2S)=25%)(N/(2S)=25\%) of standard attention with the same head dimension, corresponding to a 75% saving at S=256S=256 and N=128N=128. This ratio is not limited to a single layer and also holds for multi-layer models with the same number of layers, the same number of heads per layer, and the same head dimension.

However, actual full-model cache usage depends on architectural choices, including the number of layers, heads, and their dimensions. We thank the reviewer for highlighting this ambiguity and will clarify these conditions in the paper. We also provide full-model efficiency comparisons against a Transformer baseline using a unified Transformer block design, with the same number of layers, the same number of heads per layer, the same head dimension, and comparable total parameter counts.

From table 8 and table 9, with comparable parameter amount, DART uses twice number of layers.

We agree that the pretrained DART models have more layers than their Pythia counterparts. Pythia serves as a publicly available full-attention reference for evaluation, and the 75% cache-saving claim assumes matched layer and head configurations. These experiments compare performance across architectures (including DART, Pythia, Attamba) with comparable parameter counts and historical cache budgets. The results illustrate that full attention remains stronger on some retrieval tasks under the comparable cache budget, highlighting an important direction for further research and improvement of DART. We hope this clarifies the comparison settings and the basis for fair evaluation in our experiments.

Can you report model-level cache, FLOPs and decode throughput against a parameter-matched transformer, and revise the 75% claim if needed?

We thank the reviewer for raising this important question about full-model efficiency. In the General Response, Table 1 specifies the matched layer and head configurations and comparable parameter counts, Table 3 reports prefill + decode latency, Table 4 reports model-level cache and peak memory, and Tables 5–6 provide the leading-order FLOPs analysis. DART reduces total persistent cache by 67.85–74.12% relative to Transformer, while Tables 2–3 show a 1.67–2.42× speedup in prefill + decode. We will clarify that the theoretical 75% saving concerns historical KV cache under matched configurations.

Also for figure 5, if i am reading correctly the SMA kernel takes >10× long as SSD, which essentially means DART almost gives up Mamba-2's linear-time benefit.

We agree that SMA introduces additional computation compared with SSD, and DART does not retain the strictly linear-time complexity of pure recurrent models. However, this is a deliberate design trade-off. We believe that, as sequence length increases, models should maintain more memory or cache capacity to retain more information from the growing history.

Fixed-size recurrent states have bounded capacity and cannot losslessly retain an arbitrarily long history. Token-level attention, on the other hand, preserves access to the full history but incurs substantial memory costs. DART aims to combine the strengths of both approaches by compressing tokens into chunk-level memories while retaining these memories for attention-style retrieval. Figure 5 shows that SMA achieves 37.50% lower runtime than FlashAttention-2 at 64K tokens, demonstrating that retrieval over compressed states can achieve lower kernel runtime than token-level attention under the evaluated conditions.

AuthorsOpenReview ↗

Response to Reviewer Ddm7 (2/2)

the paper lacks proper comparison to hybrid architecture such as JAMBA, SAMBA, including retrieval performance, latency, and inference cache.

How does DART compare with hybrid architecture models, whose total KV cache equals DART's?

We agree that comparisons with hybrid architectures would help characterize DART’s performance. To address the question, we added an MQAR comparison against Mamba-2–attention hybrid baselines with matched KV cache budgets and comparable parameter counts. Specifically, we set N=S=16N=S=16 for DART and halve the number of heads per layer relative to the original MQAR configuration. For the hybrid baselines, we replace the attention module in one of the two Transformer++ blocks with Mamba-2 and evaluate both layer orders. All models have two layers and are trained on sequences of length L=1024L=1024 under the same 32-epoch training protocol. The configurations and test accuracies are summarized below.

Table 7. MQAR configurations for DART and hybrid baselines.

Model widthLayer sequenceHeads per layerTotal parametersKV cache
32DART → DART4535.59 K128 KiB
32Attention → Mamba-24551.56 K128 KiB
32Mamba-2 → Attention4551.56 K128 KiB
64DART → DART81.08 M256 KiB
64Attention → Mamba-281.15 M256 KiB
64Mamba-2 → Attention81.15 M256 KiB
128DART → DART162.22 M512 KiB
128Attention → Mamba-2162.50 M512 KiB
128Mamba-2 → Attention162.50 M512 KiB

The following table reports MQAR test accuracy (%).

Table 8. MQAR test accuracy for DART and hybrid baselines (%).

Model widthDART → DARTAttention → Mamba-2Mamba-2 → Attention
3272.930.0814.29
6497.180.1099.99
12899.870.23100.00

The results show that (1) at dmodel=32d_{\mathrm{model}}=32, DART substantially outperforms both hybrid baselines. At larger widths, DART consistently outperforms Attention → Mamba-2 and achieves comparable accuracy to Mamba-2 → Attention. (2) These two-layer hybrid baselines are highly sensitive to layer order. Attention → Mamba-2 remains below 0.24% accuracy across all three widths, whereas Mamba-2 → Attention achieves near-perfect recall at widths 64 and 128. In comparison, DART eliminates the need to choose an ordering of separate attention and recurrent layers. This provides an additional architectural advantage over these interleaved baselines.

We also benchmark the dmodel=128d_{\mathrm{model}}=128 architectures on RTX 4090 GPUs. Each run processes TT input tokens and produces TT output tokens. The tables report the latency.

Table 9. DART and hybrid baseline prefill latency in milliseconds.

Model1K × 642K × 324K × 168K × 816K × 432K × 264K × 1
DART8.128.2210.4312.6118.6632.4858.74
Attention → Mamba-25.746.339.389.4413.5519.6836.10
Mamba-2 → Attention6.116.988.8610.8013.7421.4539.28

Table 10. DART and hybrid baseline prefill + decode latency in seconds.

Model1K × 642K × 324K × 168K × 816K × 432K × 264K × 1
DART0.660.730.951.602.995.5210.29
Attention → Mamba-20.640.711.051.963.316.2811.81
Mamba-2 → Attention0.580.641.051.843.286.1811.80

DART has higher prefill latency than both hybrid baselines, but achieves lower prefill + decode latency at 4K and above.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank all reviewers for their careful reading and constructive suggestions. We would like to clarify the motivation and scope of DART again. DART is proposed as a sequence modeling mechanism instead of a fixed model architecture. The key motivation is that fixed-size recurrent states have bounded memory capacity, while token-level attention retains the full history at substantial cache cost. DART explores an intermediate design by combining recurrent compression with attention-style retrieval over shared memory representations.

All three reviewers requested full-model efficiency measurements. So we report the experimental setup and complete results below. We benchmark 1B DART, Transformer, and Mamba-2 models using a unified Transformer block design with 16 layers, 32 heads per layer, and a head dimension of 64. For DART, we set the chunk size to S=256S=256 and the state size to N=128N=128, and Mamba-2 has the same state size. All measurements are performed in BF16 on RTX 4090 GPUs. The prefill and decode sequence lengths are equal. Columns in the latency tables are labeled as "sequence length × batch size".

Table 1. Model configurations. The MLP widths are adjusted to match total parameter counts.

ParameterMamba-2TransformerDART
Number of layers161616
Hidden dimension204820482048
Number of heads323232
Head dimension646464
MLP intermediate dimension588853125824
State size NN128—128
Total parameters995.91 M996.74 M996.19 M

Table 2. Full-model prefill latency in seconds. Compared with Transformer, DART's prefill is slightly slower at 1K and 2K, approximately equal at 4K, and faster at longer contexts.

Model1K × 642K × 324K × 168K × 816K × 432K × 264K × 1
Mamba-20.820.820.820.820.880.820.82
Transformer0.810.850.891.011.231.672.54
DART0.860.870.900.961.071.291.76
DART speedup over Transformer0.94×0.97×0.99×1.05×1.15×1.30×1.44×

Table 3. Full-model prefill + decode latency in seconds. Prefill + decode is 1.67–2.42× faster than Transformer across these settings.

Model1K × 642K × 324K × 168K × 816K × 432K × 264K × 1
Mamba-24.827.2412.3222.1741.9981.11140.65
Transformer18.3035.4368.21132.04277.21542.601078.67
DART10.9817.4332.4860.75119.74237.84445.55
DART speedup over Transformer1.67×2.03×2.10×2.17×2.32×2.28×2.42×

Table 4. Full-model inference memory in GiB. The measured cache reduction relative to Transformer is 67.85–74.12%.

Input length × batch sizeMamba-2 cacheTransformer cacheDART cacheMamba-2 peak memoryTransformer peak memoryDART peak memory
1K × 640.5216.005.146.7721.9811.37
2K × 320.2616.004.636.5121.9810.86
4K × 160.1316.004.386.3821.9910.60
8K × 80.0616.004.256.3222.0110.48
16K × 40.0316.004.196.2922.0410.41
32K × 20.0216.004.166.2622.1010.38
64K × 10.0116.004.146.2522.2210.36
AuthorsOpenReview ↗

General Response (2/2)

Table 5. Sequence-modeling core decode FLOPs and arithmetic intensity. Autoregressive decoding is often memory-bound because it reads large KV caches while performing relatively little computation. We analyze the sequence-modeling core per decoding step, layer, head, and sequence. Let TT be the cached history length, SS the chunk size, M=⌊T/S⌋M=\lfloor T/S\rfloor the number of completed historical chunks, NN the state size, and dd the head dimension. States and KV caches are BF16, and a multiply-add counts as two FLOPs. We retain dominant state-matrix terms. Arithmetic intensity is FLOPs divided by input plus output bytes.

ModelInput bytesOutput bytesFLOPsArithmetic intensity in FLOPs/byte
Transformer4Td4TdO(d)\mathcal O(d)4Td4Td≈1.00\approx1.00
Mamba-22Nd2Nd2Nd+O(d)2Nd+\mathcal O(d)5Nd5Nd≈1.25\approx1.25
DART2MNd2MNd2Nd+O(d)2Nd + \mathcal O(d)4MNd4MNd≈2.00\approx2.00

DART decodes a key and value from each historical state at 4Nd4Nd FLOPs per chunk. Its dominant FLOP and traffic ratios relative to Transformer are N/SN/S and N/(2S)N/(2S), respectively (50.00% and 25.00% at N=128N=128, S=256S=256).

Table 6. Full-model decode FLOPs. For one decoding step and one sequence, let LL be the number of layers, D=HdD=Hd the hidden dimension, HH the number of heads, and IT,IDI_T,I_D the Transformer and DART MLP widths. T,M,N,dT,M,N,d are defined above. PDP_D is DART's input-projection output width in the benchmark configuration, PD=2D+3N+d+H+1P_D=2D+3N+d+H+1.

ComponentTransformerDART
Mixer projectionsQ, K, V, O: 8D28D^2Input D→PDD\to P_D, output D→DD\to D: 2DPD+2D22DP_D+2D^2
SwiGLU MLPGate, up, down: 6DIT6DI_TGate, up, down: 6DID6DI_D
Sequence modelingQK and AV over TT tokens: 4HTd4HTdKey/value decoding over MM chunk states: 4HMNd4HMNd
Total across LL layersL(8D2+6DIT+4HTd)\displaystyle L(8D^2+6DI_T+4HTd)L(2DPD+2D2+6DID+4HMNd)\displaystyle L(2DP_D+2D^2+6DI_D+4HMNd)

These leading-order estimates omit other minor operations. Under the benchmark configuration, the model blocks require approximately 10.17 GFLOPs for Transformer and 5.87 GFLOPs for DART per decoding step and sequence at a cached history length of T=64KT=64K.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.