- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper aims to extend block diffusion LMs to Mamba–attention hybrid backbones. It restricts bidirectional Mamba (forward and reverse scans) to the active denoising block and uses standard cached Mamba states for completed blocks to enable block-wise caching. Training with these caches uses an AdaLN design. Experiments at 87M and 350M parameters show lower perplexity than attention-based BDLMs.
Critical strengths
-
This work explores a natural extension of block diffusion models to hybrid architectures. Combining block diffusion with linear attention is helpful for improving long-context generation efficiency.
-
The proposed design is simple and well motivated, providing a natural extension of bidirectional Mamba in diffusion models.
-
The experiments are controlled, and the long-context throughput gains are promising. Each architecture receives its own learning-rate sweep, and BDLM Mamba-H’s throughput advantage continues to grow with sequence length.
Critical weaknesses
-
The technical novelty is limited. The paper finds that conditioning the Mamba component in an active block on a causal prefix Mamba cache reduces perplexity compared with fully bidirectional models. However, this insight has been widely used in full-attention models, such as “Set Block Decoding is a Language Model Inference Accelerator” and “Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding.” Both use causal masks to encode context and find that this improves diffusion performance. This work essentially applies the same idea to Mamba layers.
-
Based on the insight in point 1, the most relevant baseline would also use causal context encoding for full-attention layers, but this baseline is missing. Directly comparing the proposed architecture with fully bidirectional diffusion models may not be an apples-to-apples comparison.
-
Another major concern is the practical value of the throughput results. First, it is unclear why randomly initialized models are used. More importantly, the paper does not demonstrate the potential for parallel sampling in diffusion LMs, so it is unclear why the proposed approach would be preferred over AR models. In addition, there is no comparison with autoregressive Mamba–attention hybrids.
-
The evidence for generation quality is weak. Experiments at larger scales and on more complex benchmarks would be desirable. The authors could consider converting an existing AR hybrid model into a diffusion model to demonstrate this at a reasonable cost.
Questions for authors
I have included my questions in the weakness section.
One particular question is that what is "random-initialization" for inference throughput measurement? Why use this strategy?
2 comments
Response to Reviewer eM7h (Part 2/2)
Evaluation benchmarks at 87M/350M and why AR-to-BDLM Mamba conversion is impractical here (W4). Our 87M and 350M checkpoints are base models pretrained from scratch on 5B–17.65B DCLM tokens without instruction tuning or RL post-training. More complex reasoning or free-form generation benchmarks would be ill-suited for unaligned base models at 87M–350M scale, where performance largely reflects prompt-format sensitivity and near-chance variance; following standard base-DLM practice (BD3-LM; DiffuMamba), we evaluate held-out C4-en and Paloma-C4 NELBO PPL/BPB alongside six zero-shot fixed-choice MCQA tasks (Sec. 5.1).
Converting an existing pretrained AR hybrid into a BDLM Mamba model is also impractical for two concrete reasons. First, open pretrained AR hybrids lack checkpoints matching our sequential sparse-attention Mamba-2 architecture at this scale. Sequentially interleaved Mamba-2 hybrids such as Nemotron-H (8B–56B), Bamba-9B (9B), and Jamba (52B MoE) are an order of magnitude larger, while smaller 0.5B–1.5B candidates use incompatible topologies (Falcon-H1-0.5B/1.5B and Hymba-1.5B, ICLR 2025 place attention heads in parallel inside every layer and retain an K/V cache in all layers, whereas Zamba2-1.2B uses a single weight-shared attention block with LoRA adapters). Second, efficient AR-to-BDLM conversion recipes such as Fast-dLLM v2 (Wu et al., ICLR 2026) rely on two Transformer properties that break down on Mamba-2. In a Transformer, replacing a causal mask with a block-bidirectional mask reuses pretrained Q/K/V weights directly, whereas an AR Mamba-2 layer contains only forward scan parameters () and lacks the reverse SSM branch (, Eq. 10) required for intra-block bidirectionality, meaning conversion would still have to train newly initialized reverse conv/SSM parameters across 83% of layers. Moreover, Fast-dLLM v2 relies on a hierarchical sub-block cache that stores K/V states for arbitrary unmasked positions inside a partially decoded block; in Mamba-2, the forward recurrent state depends contiguously on all preceding in-block tokens (including still-masked slots whose inputs change at the next step) and the reverse scan sweeps right-to-left from the block end, preventing sub-block recurrent state caching.
Response to Reviewer eM7h (Part 1/2)
Thank you for the clear feedback and for noting our controlled per-architecture learning-rate sweeps and long-context throughput scaling. Below we address each point and note the clarifications we have made in the revised draft.
Causal-context full-attention baseline and technical contribution on Mamba-2 (W1, W2). On the baseline comparison in W2, our primary baseline (BDLM attention) already uses causal context encoding across completed prefix blocks (BD3-LM, Arriola et al., ICLR 2025, Eq. 6), storing causal prefix representations in a reusable K/V cache and applying bidirectional attention only inside the active 256-token block. We compare BDLM Mamba-H head-to-head against BDLM attention across every experiment in the paper (the 87M sweep in Tables 1 and 5, the 350M scale-up in Tables 2–4, and long-context inference in Table 6), while full-sequence attention and full-sequence DiffuMamba-H serve only as non-cacheable full-window references. On novelty relative to full-attention causal-prefix designs (W1), in Transformer block models such as BD3-LM, Set Block Decoding (Gat et al., 2025), and Nemotron-Labs-Diffusion (Fu et al., EMNLP 2026), conditioning an active block on a causal prefix is purely a 2D attention-mask change over standard K/V projections. In bidirectional Mamba-2 (DiffuMamba, Singh et al., ICML 2026), a full-window reverse scan instead propagates noisy active-block tokens backward into every prefix state and prevents exact caching across blocks. Making Mamba-2 natively block-cacheable therefore requires decoupling the forward recurrent boundary cache (conv1d plus selective SSM state skℓ) from a strictly block-local reverse scan (Eqs. 9–11), training those recurrent prefix boundary states end-to-end through downstream block losses across all blocks in parallel (Sec. 4.1), and factoring AdaLN so prefix scans remain independent of the reverse timestep t (Eq. 7, Appendix E). We have cited Set Block Decoding and Nemotron-Labs-Diffusion in Section 2 and made BDLM attention's causal prefix mask explicit.
Random-init 700M throughput benchmarks, parallel sampling, and AR hybrids (W3, Q1). Regarding Question 1, "random initialization" means profiling the compiled model and CUDA/Triton kernels from freshly initialized weights without pretraining a 700M checkpoint. Our native BDLM sampler (Appendix B, Table 9) runs a fixed schedule (B=256, p=16) in which every block launches the exact same FlashAttention, Mamba-2, and GEMM kernels on identical tensor shapes regardless of parameter values, so FLOPs, HBM traffic, and tokens/s are identical to those of a trained checkpoint. Following DiffuMamba (which profiles ~1.3B–1.4B models on H100), we ran this benchmark at 700M to evaluate a model larger than 350M that better saturates an A100-80GB GPU and provides a clearer comparison across architectures up to 262K tokens, as we have clarified in Section 5.3. On parallel sampling and AR hybrids, within each 256-token block all 256 positions are denoised in parallel over p=16 reverse steps (plus 1 clean cache append), amortizing prefix-cache reads across B/p=16 generated tokens per denoising step rather than streaming the accumulated cache once per token as in AR decoding. Mamba-2 hybridization and block-parallel decoding address complementary bottlenecks in long-context serving, since hybridization at 350M reduces attention K/V cache state by 6× (retaining attention in only 3 of 18 layers) while block diffusion cuts sequential cache-reading passes per 256 emitted tokens from 256 down to p+1=17.