- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the spectrum of the token mixing matrix of SSD models such as Mamba-2 and RetNet and contrasts it with causal softmax attention. The authors prove that decaying masks spread spectral mass over a growing number of directions, while a strong attention sink drives softmax attention toward rank-one collapse. Experiments on GPT-2, Based, Mamba-Codestral-7B and RetNet-1.3B show growing ranks for Mamba-2 and RetNet and nearly constant energy-sensitive ranks in the deeper GPT-2 layers. Replacing only the decay mask with an all-one mask lowers the ranks.
Critical strengths
The paper includes causal tests at the level of the mixing operator. Keeping the learned query and key factors fixed and replacing only the decay mask with an all-one mask lowers the rank, which isolates the effect of decay. On the softmax side, removing the score advantage of the first token in GPT-2 broadens the spectrum, which supports the proposed sink mechanism.
Critical weaknesses
The functional relevance of the spectral measures is not shown. All measurements, including the interventions above, stay at the level of the mixing matrix. The modified operators are never passed through the rest of the model, and the value path, head outputs and residual stream are never examined, although the abstract frames the work around preserving information. The interventions therefore show what determines the spectrum, but not whether the spectrum affects the model's behavior. High rank under decay may simply reflect locality, since with short memory each token mostly mixes with its neighbors.
Conversely, the collapse in deeper GPT-2 layers may reflect dormant heads that send attention to the first token and write almost nothing. This might be because they have nothing to do in this context, or because the computation happened elsewhere.
While the paper is a theory paper, it has an important empirical side, and I think it can be improved significantly. The cross-architecture comparison is confounded. The softmax side is GPT-2 small (124M parameters, learned absolute positions), while the SSD side includes a 7B Mamba model, so scale, data, tokenizer and positional encoding all differ. In addition, the final Based layer, which has no sink mechanism, also keeps a stable rank close to one, which suggests that the relevant contrast is decay versus no decay rather than softmax versus SSD.
Minor
- Short intuition for the theorems in the main text would help readers.
- Example heatmaps of the mixing matrices for a GPT-2 sink head and a Mamba-2 head would make the results more concrete.
Questions for authors
Have the authors considered softmax models with ALiBi? ALiBi’s linear logit penalty is equivalent to multiplying the unnormalized attention weights by a RetNet-style decay mask, so these models combine softmax with decay and would test whether decay or the absence of softmax drives spectral diffusion. A RoPE model would also show whether the GPT-2 results, from a model with learned absolute position embeddings, hold for modern softmax attention. I think especially ALiBi would be very informative for the paper. Head ablations and a comparison with scale-matched ALiBi and RoPE softmax models would help me reassess my evaluation.