acceptodds
Under review as a conference paper at ICLR 2027

State Space Duality Attention Non-Collapse

Abstract

State-space models are emerging as compelling linear-time alternatives to Transformers. Yet efficiency alone does not explain what fundamentally separates the two architectures: how do they organize and preserve information across a sequence? Through the *State Space Duality* (SSD) structured attention matrix, ( M = L odot QK^ top, ) we prove that bounded-memory SSD attention spreads information across a growing number of directions, and establish rank non-collapse for structured causal mixers. As a causal-softmax contrast, triangular masking alone does not force relative collapse, whereas a coherent early-token score sink can drive attention toward collapse. Together, these results separate relative rank collapse from absolute spectral degeneration and show that spectral concentration is governed jointly by mask structure and score geometry. Across context lengths, sink-dominated causal self-attention exhibits spectral *compression*, whereas SSD attention with decaying memory exhibits spectral *diffusion*.

Primary area:
learning theory
Keywords:
State Space Duality Model, Mamba, Linear Attention, Self-Attention, Spectral Analysis, Attention Collapse, Attention Sink

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6ucDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the spectrum of the token mixing matrix of SSD models such as Mamba-2 and RetNet and contrasts it with causal softmax attention. The authors prove that decaying masks spread spectral mass over a growing number of directions, while a strong attention sink drives softmax attention toward rank-one collapse. Experiments on GPT-2, Based, Mamba-Codestral-7B and RetNet-1.3B show growing ranks for Mamba-2 and RetNet and nearly constant energy-sensitive ranks in the deeper GPT-2 layers. Replacing only the decay mask with an all-one mask lowers the ranks.

Critical strengths

The paper includes causal tests at the level of the mixing operator. Keeping the learned query and key factors fixed and replacing only the decay mask with an all-one mask lowers the rank, which isolates the effect of decay. On the softmax side, removing the score advantage of the first token in GPT-2 broadens the spectrum, which supports the proposed sink mechanism.

Critical weaknesses

The functional relevance of the spectral measures is not shown. All measurements, including the interventions above, stay at the level of the mixing matrix. The modified operators are never passed through the rest of the model, and the value path, head outputs and residual stream are never examined, although the abstract frames the work around preserving information. The interventions therefore show what determines the spectrum, but not whether the spectrum affects the model's behavior. High rank under decay may simply reflect locality, since with short memory each token mostly mixes with its neighbors.

Conversely, the collapse in deeper GPT-2 layers may reflect dormant heads that send attention to the first token and write almost nothing. This might be because they have nothing to do in this context, or because the computation happened elsewhere.

While the paper is a theory paper, it has an important empirical side, and I think it can be improved significantly. The cross-architecture comparison is confounded. The softmax side is GPT-2 small (124M parameters, learned absolute positions), while the SSD side includes a 7B Mamba model, so scale, data, tokenizer and positional encoding all differ. In addition, the final Based layer, which has no sink mechanism, also keeps a stable rank close to one, which suggests that the relevant contrast is decay versus no decay rather than softmax versus SSD.

Minor

  • Short intuition for the theorems in the main text would help readers.
  • Example heatmaps of the mixing matrices for a GPT-2 sink head and a Mamba-2 head would make the results more concrete.
Questions for authors

Have the authors considered softmax models with ALiBi? ALiBi’s linear logit penalty is equivalent to multiplying the unnormalized attention weights by a RetNet-style decay mask, so these models combine softmax with decay and would test whether decay or the absence of softmax drives spectral diffusion. A RoPE model would also show whether the GPT-2 results, from a model with learned absolute position embeddings, hold for modern softmax attention. I think especially ALiBi would be very informative for the paper. Head ablations and a comparison with scale-matched ALiBi and RoPE softmax models would help me reassess my evaluation.

Reviewer_XU4YOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the spectral properties of the matrix operator form in scalar-decay linear recurrences (e.g. Mamba-2 and RetNet), contrasting their behavior with causal softmax attention. Under stylized low-rank and Gaussian query–key assumptions, the authors establish spectral non-collapse results for semiseparable linear attention (with and without decay), alongside conditions under which attention sinks induce spectral concentration.

Experiments on GPT-2, Based, Mamba-2, and RetNet show an increase in the effective spectral rank for the recurrent architectures and stronger spectral concentration in later Transformer attention layers. The authors interpret these findings as evidence of a distinction between spectral diffusion and compression across these sequence mixers.

Critical strengths
  1. Theoretical contribution: The paper provides a substantial theoretical analysis of the spectral behavior of structured attention operators.

  2. Breadth of empirics: The experiments examine multiple architectures and include controlled mask replacements, providing useful evidence that the observed spectral behavior depends on the underlying sequnece mixer structure.

Critical weaknesses
  1. Unclear significance of spectral non-collapse: The paper measures the effective rank of the full operator, but this quantity does not necessarily characterize information preservation or memory utilization. For example, the identity operator has maximal effective rank despite transmitting no information across time. Prior work [1] addresses this distinction by analyzing past-to-future operator submatrices. The absence of a comparison with this framework leaves the significance of the proposed metric unclear.

  2. Limited connection between theory and practical architectures: The theoretical analysis relies on restrictive assumptions about query–key geometry and memory decay that are not established for the trained models studied. The experiments measure spectral properties over relatively short contexts without demonstrating any relationship to downstream performance, long-range recall, or information retention.

[1] Quantifying Memory Utilization with Effective-State Size

Questions for authors
  1. What motivates analyzing the effective rank of the full sequence-mixing matrix rather than past-to-future submatrices, as considered in the effective state-size framework [1]?

  2. How should the spectral non-collapse results be interpreted in cases where the operator approaches the identity matrix? Such operators exhibit maximal effective rank despite transmitting little or no information across time.

  3. Can the authors demonstrate a relationship between the proposed spectral measurements and functional capabilities such as long-range recall or memory utilization?

  4. Given that the experiments do not verify the assumptions underlying the main theoretical results, what conclusions can be drawn about the practical architectures beyond the observed empirical spectral trends?

[1] Quantifying Memory Utilization with Effective-State Size

Reviewer_StfEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is mainly about theoretical analysis of different matrix mixers such as self-attention, retnet, and mamba2 (ssd). Specifically, the authors focus on the spectral propertices of these structured matrix mixers; the paper shows that ssd and retnet can keep effective ranks even if input sequence length increases, whereas self-attention exhibits rank collapse especially when there exists a strong attention sink. The authors use pretrained model weights to support their claim.

Critical strengths
  • The theoretical analysis is interesting and looks into an area that the community has not been focusing on. It presents an interesting perspective of understnading the difference between ssd and self-attneiton.
  • The authors provide experiments that show controlled mask replacements can help isolating the effect of forgetting&decay on spectral rank.
Critical weaknesses
  • The experiments do support the paper's main claims. However, it's not as straightforward if this is applicable to real-world settings. Also, there are many confounding factors when we use pretrained weights for this kind of analysis. For example, their pretrained weights use different training datasets, and there are so many different architectural details that might have led to nontrivial results.
  • The assumptions that the paper use are not feasible to be universally used.
  • The paper's experiments use relatively low seuqnce lengths, so its real effects on downstream performance could actually be subtle.
Questions for authors
  • An identity sequence mixer have high rank, but it does not mix information at all. This makes me question if ranks do actually have direct signals in terms of expressivity. How do the authors think about this kind of edge cases?
  • What are some practical results of avoiding collapse in real-world settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.