acceptodds
Withdrawn from ICLR 2027

Differential Attention for State Space Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The Transformer architecture has emerged as the dominant paradigm for sequence modeling; however, it suffers from quadratic complexity in sequence length. In parallel, state space models (SSMs), such as Mamba, have been proposed as efficient alternatives, offering sub-quadratic complexity while maintaining competitive performance. The Differential Transformer demonstrated that transformers tend toward attention over-allocation, which can impair retrieval and long-context reasoning. They address over-allocation by subtracting attention maps, thereby improving signal selectivity and mitigating noise. Existing attempts to transfer this idea to SSMs rely on parallel reduced Mamba blocks that incur limited scalability, and significant computational overhead. We introduce a generalized formulation of differential computation for state space models derived directly from structured state space duality (SSD). Rather than constructing parallel SSM branches, our formulation shows that differential computation naturally emerges as an operation on state-space readout. This perspective enables both an exact single-pass formulation and an efficient single differential circuit implementation without modifying the underlying SSM kernels. We instantiate the proposed framework in both Mamba-2 and Mamba-3, yielding Diff-Mamba-2 and Diff-Mamba-3. Across language modeling benchmarks, our models consistently outperform their corresponding baselines by up to 1.5 average points, while preserving the computational efficiency of modern SSM architectures. We further demonstrate improved retrieval on real-world long-context benchmarks, and sharper token-level interaction patterns through interpretability analysis. These results establish differential computation as a principled and effective design paradigm for state space models. Our code is publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mamba, Space State Models, Differential Attention, Differential Mamba, LLMs, Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_xAVpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates differential computation for state space models through structured state space duality (SSD). For two branches that share temporal dynamics, the authors propose a single-pass computation method using an augmented state of dimension 2N. They further propose C-only SDC, which adjusts the readout through input-dependent subtraction and normalization. This design is applied to Mamba-2 and Mamba-3. Experiments show that the differential variants improve average accuracy on downstream language modeling tasks and achieve gains on several retrieval tasks.

Critical strengths
  1. The C-only variant shares the recurrent state and preserves the state dimension. Compared with duplicating the entire Mamba module, this design reduces the additional structure required to implement differential computation.

  2. The language modeling experiments cover multiple model sizes and use 100B-token pretraining. In all comparisons reported in Table 1, the differential variants achieve higher average downstream scores than their corresponding baselines.

Critical weaknesses
  1. The justification for the connection between differential readout from a shared state and selective denoising remains insufficient. The paper interprets this design as more precise common-mode noise cancellation, but does not sufficiently explain why the components jointly represented by the two branches mainly correspond to irrelevant information, or why subtraction can suppress these components while preserving the target signal.

  2. The gains and regressions in retrieval performance lack explanation. For Mamba-3 MIMO, the score on NIAH-Single-3 at 1024 tokens drops from 95.8 to 61.4. This is a substantial decline that occurs within the training length, especially considering that improving memory utilization is an important motivation of the paper.

Questions for authors
  1. What direct evidence shows that the two branches’ contributions from irrelevant information are similar enough to cancel out, while the target information is preserved? Existing experiments provide only limited direct support for this mechanistic explanation. For example, could the authors, as in Differential Transformer, separately quantify how C-only changes the contributions of target and irrelevant information in tasks with clearly defined targets and distractors?

  2. Table 4(a) provides evidence that subtraction brings gains in the full construction. Could the authors explain whether this conclusion also holds for the C-only + RMSNorm variant used in the main experiments?

Reviewer_MMT1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a differential computation framework for state-space models (SSMs) based on Structured State Space Duality (SSD). It derives an exact single-pass formulation and introduces a lightweight Single Differential Circuit (SDC) that preserves the original recurrent state dimension. The method is applied to Mamba-2 and Mamba-3, with experiments showing improvements in downstream language modeling tasks, alongside mixed retrieval results and inference efficiency trade-offs.

Critical strengths
  1. Principled and efficient architectural design. The paper derives differential computation through SSD and proposes a lightweight implementation that avoids duplicating recurrent states and remains compatible with existing SSM kernels.

  2. Consistent downstream improvements. Experiments across multiple model scales and Mamba architectures show recurring improvements in downstream accuracy, supporting the practical potential of the proposed design.

Critical weaknesses
  1. Insufficient evidence for noise suppression in SSMs.

Unlike Differential Transformer, which quantitatively demonstrates excessive attention to irrelevant tokens and its reduction, this paper lacks comparable evidence of irrelevant-information interference in SSMs or its suppression by differential computation. The qualitative interaction heatmaps alone do not establish the proposed noise-cancellation mechanism.

  1. Inconsistent retrieval improvements and substantial inference overhead.

The claimed improvements in long-context retrieval and efficiency are not consistently supported. Several NIAH settings exhibit substantial regressions at 4K (Table 3), while autoregressive inference throughput drops by 25–51% compared to vanilla Mamba (Appendix E, Figure 4), despite comparable full-sequence forward latency in Table 6(b).

Questions for authors
  1. Noise suppression (additional experiment): Could the authors provide a controlled experiment, analogous to Differential Transformer's attention analysis, quantifying the contributions of relevant versus irrelevant tokens in SSMs and demonstrating whether differential computation selectively suppresses irrelevant information?

  2. Retrieval performance (clarification): Could the authors explain the substantial NIAH regressions and clarify under which conditions differential computation improves retrieval?

  3. Inference efficiency (clarification and additional experiment): Since the C-only design preserves Mamba's recurrent state update, could the authors clarify whether cached-state decoding is properly implemented? Could they also provide profiling or a minimally optimized implementation (even a proof of concept) to investigate whether the throughput degradation can be reduced below approximately 20%?

Reviewer_V5BAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary
  • The paper introduces Diff-Mamba-2 and Diff-Mamba-3, adapting differential attention to state-space models in a principled manner.
  • The main variant (called "C only") uses a gated, normalized difference between two readout projections while keeping the recurrent-state size unchanged
  • The "C only variant" consistently improves average short-context language-modeling scores across several model scales, but the recall results are mixed.
Critical strengths
  • The formulation the authors propose (at least the B-and-C formulation) seems much closer to the original differential attention proposed in Ye et al. (2024) than existing approaches using two independent parallel blocks – the authors managed to incorporate the differential attention operation within a single linear recurrence block.
  • The paper conducts extensive evaluations on RULER single needle tasks, real-world recall tasks, and common language-modeling benchmarks, and reports results at multiple model scales.
Critical weaknesses
  • I'm mainly concerned regarding the motivation of the work: differential attention is used to reduce noise from other competing keys -- this makes sense in dense attention where there can be lots of keys, but in linear recurrence, there is only a limited number of "keys" / slots / "directions" one is competing with. So not sure whether importing differential attention in a linear recurrence is fixing any problem as such.
  • differential attention does not seem to help Mamba3 especially in recall tasks where the haystack is real world essays containing "natural" noise (S-NIAH-2 and S-NIAH-3 in Table 3). differential attention does seem to help in Mamba2 a bit, but again I am not sure what is it fixing in the first place.
  • There is no comparison with methods that do the "naive" way of differential attention that authors mention in related works
Questions for authors
  • Can the authors provide a concrete demonstration of the problem of competing keys in linear recurrences? I am asking because this would help establish differential attention as a motivated solution; otherwise, I am not sure what it is fixing.
  • How does this principled approach compare with existing work that simulates differential attention using two independent parallel recurrence blocks on downstream and recall tasks? Authors can reduce head dims to match on state size here – or increase head dim in their approach
  • Can the authors conduct experiments using two more seeds on Mamba-2 and Mamba-3 at an intermediate scale, such as 370M, to check whether the reported gains are consistent?
  • The interpretability results currently show the token interactions after "suppression". Can the authors also show the subtraction mask, so we can see what is being suppressed? It could be the case that the model is just predicting the gtg_t values to be zero – which would mean nothing is being suppressed, the subtraction mask is just being ignored.
Reviewer_YW47OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an efficient implementation of differential state space models, focusing on Mamba 2 and 3. The authors first show that the standard two-branch differential mamba model can be represented exactly using a single SSM with a larger state dimension (double the size) by incorporating the subtraction directly into the readout operation. In practice, they use a ssm with the same state dimension as the original but segment the state into two halves. Furthermore, they show that you can further improve efficiency by sharing the state between the two halves of the ssm, sharing the write projections, decay, and initialization to make the two implicit recurrent states identical. They ablate this choice along with normalization of the differential readout in the experiments. The main results demonstrate lower perplexity and higher downstream accuracy on most benchmarks over the ordinary non differential mamba baselines. They show that their implementation of differntial mamba also achieves lower perplexity than the original differential baseline at three model scales. However, they clearly state that performance on needle in a haystack benchmarks are greatly degraded. The C-only variant (sharing the state and only using normalized differential readout) improves generation throughput over the differential baseline but is slower than the original ordinary mamba as expected

Critical strengths

The authors present a simple architecture change to implement differential transformers efficiently, demonstrating consistent gains on downstream benchmarks and on ppl. Thoroughly ablate different architecture choices and alternatives, showing the gain from B only and C only architecture.

Critical weaknesses

Why does the differential operation/readout help? if it is noise suppression or better use of memory, why does it significantly degrade the retrieval performance? Does this method outperform Mamba with normalization and other readout alterations? Are you able to isolate the effect of the subtraction?

Questions for authors

Why does the implementation outperform differential mamba on ppl? It can't be the shared state because C only is shown to do worse than B and C together? Is it the normalization?

Comments on the paper

1 comment

We sincerely thank the reviewers for their time, thoughtful evaluations, and constructive feedback. After carefully considering the reviews, we have decided to withdraw our submission to further investigate the concerns raised and strengthen the work. We greatly appreciate the reviewers' valuable suggestions, which will help guide our future research and revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.