- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the use of fixed-size state caches for block diffusion language models. The authors train three roughly 3B-parameter models with attention, Mamba, and hybrid backbones on 300B tokens under a shared block-causal training objective. Through extensive experiments, they show that Mamba achieves constant cache memory and stable decoding latency as context length increases, while also exhibiting better length extrapolation than attention. The hybrid model offers a trade-off between efficiency and quality, largely matching attention on downstream tasks while reducing inference costs.
Critical strengths
The paper provides a substantial empirical evaluation, training attention, hybrid, and Mamba-based block diffusion models at approximately 3B parameters on 300B tokens each. Its coverage of inference efficiency, downstream performance, and length extrapolation provides useful evidence on the practical trade-offs between these architectures.
Critical weaknesses
-
Limited novelty. The paper largely builds upon established techniques, including block-causal training, exact caching, and fixed-size Mamba states. Its main contribution lies in scaling and systematically evaluating their combination, without demonstrating substantial methodological advances or novel insights beyond known properties of state-space models.
-
Limited explanatory insights beyond empirical validation. While the larger-scale evaluation provides useful evidence on the quality–efficiency trade-off, the additional insights beyond prior work appear limited. The constant-state efficiency advantage and the associated batching benefits are largely expected from the architecture. The quality and length-extrapolation results are more informative, but the paper provides limited analysis of the factors underlying these outcomes. Overall, the contribution is primarily empirical validation, with less explanatory insight than the breadth of experiments might suggest.
Questions for authors
Could the authors provide a deeper analysis of why Mamba and Hybrid exhibit better long-context generalization than Attention? Understanding the underlying factors behind these differences would provide more insights beyond the empirical observations.
3 comments
Thank you for the detailed rebuttal and additional experiments. The new analyses of attention behavior, SSM state decay, and hybrid ablations provide valuable insights and substantially address my concerns about the limited explanatory analysis. While the methodological novelty remains limited, I appreciate the more nuanced distinction between length robustness and long-range retrieval. Overall, I believe the additional evidence strengthens the paper sufficiently, and I am updating my rating from 2 to 3.
Thank you
Thank you for taking the time to review our responses and updated experiments. We are glad that our clarifications and additional results have addressed your concerns, and we truly appreciate your positive feedback increasing the score.
Rebuttal
We thank the reviewer for the careful reading and for recognising the scale and coverage of the study. The revised manuscript adds an analysis of why the backbones differ at long context (App. G), NIAH results with confidence intervals (App. H) and a numerical check of cache exactness (App. E); new material is in blue. We address each concern below.
1. Addressing the novelty
We agree that the ingredients (block-causal training, exact caching, fixed-size Mamba state) are known. Our contribution is to establish their consequences at scale in a controlled setting: three 3B backbones trained on 300B tokens that differ only in the sequence mixer, with cached decoding verified to match full recomputation up to floating-point error (App. E). At this scale the results show behaviour that known SSM properties do not predict:
2. Adding the explanatory analysis: why Mamba and the Hybrid generalize better
We add this as new App. G (Fig. 6, Tabs. 12–13): inference-only diagnostics on the 3B checkpoints (NIAH at fixed needle distance, attention statistics, SSM decay rates, hybrid ablations), with attention evaluated without RoPE rescaling, as in the paper. "Length-robust" here means that accuracy on a needle at a fixed distance does not change as the context grows; this is not the same as using distant context.
(a) Attention fails because the context is longer than training, not because the needle is far. At fixed distance it retrieves every needle at 1k and none at 2k, even 64 tokens back (Fig. 6). The 103 of 560 heads that attend to the needle at 1k lose it: their attention mass on it falls from 0.34 to 0.04 at 2k, and their entropy rises from 0.49 to 0.66 (0.81 at 16k). App. G.2 shows that the failure comes from the positional encoding used beyond its trained range, not from attention itself.
(b) Mamba's robustness and its short memory are the same property. Its state carries no positions and decays fast: the median head halves its state in 8.3 tokens, and no head keeps half its state over 8k tokens (Tab. 12). Tokens more than a few hundred back barely affect the state, whether the context is 2k or 16k. Accordingly, needles up to 256 tokens back are retrieved at 94–98% at every length to 16k, and needles 1k or more back at most 12% of the time. Its flat LongBench score shows that what it uses is not disturbed by longer inputs, not that it uses the extra context.
(c) The Hybrid inherits Mamba's robustness until its attention layers fail. Its five attention layers do its retrieval: at 1k, removing layer 12 or 18 drops exact match from 98% to 8% or 0%, while removing neighbouring Mamba layers leaves 97–100% (Tab. 13). It matches Mamba at short distances up to 4k, but at 16k loses even a needle 64 tokens back (0% vs 94%). Restricting its attention to the last 1,024 tokens, without retraining, restores in-block copying at 16k (0% → 100%) and brings its NIAH token accuracy from 1.7% to 23.0%, Mamba's level.
We have added the following in the revised manuscript:
We hope the new analysis provides the explanatory insight that you found missing, and addresses all your concerns. We would be glad to clarify any of your questions.