acceptodds
Under review as a conference paper at ICLR 2027

Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale

Abstract

Diffusion language models decode tokens in parallel, but their bidirectional de- noiser rules out the naive key–value (KV) cache behind fast autoregressive infer- ence. Block diffusion restores caching by decoding block-by-block, and the block caches deployed on it so far are tied to attention: O(L) in memory and, if used as training-free retrofits, only an approximation of the model’s computation. Both constraints can be overcome: sequence mixers that summarize finalized blocks into a reusable state support block caching, and the corresponding block-causal training objective makes the cache exact. We study this recipe at scale, pretraining three 3B block-diffusion denoisers (attention, mamba, and hybrid) on 300B tokens under one single-frontier objective and decoding all three through a single cached interface. Only the state-space cache is O(1) in sequence length: its memory and per-step latency stay constant at any context length, while an attention cache re- mains O(L). At 256k tokens (where attention has grown to 82 GB and 29 ms/step), the Mamba cache runs at 4.3× lower latency, 11× less memory, and 2.6× higher single-stream throughput; and because that footprint is constant it scales with batch as well, reaching 14× the aggregate throughput, where attention cannot run beyond a single stream. The same linear-state bias lets the Mamba and hybrid backbones keep retrieving out to 8–16× their training length, whereas attention’s retrieval collapses at 2×: the hybrid at no measured quality cost, the pure-SSM model for about one downstream point.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion language models, caching, block diffusion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9extOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the use of fixed-size state caches for block diffusion language models. The authors train three roughly 3B-parameter models with attention, Mamba, and hybrid backbones on 300B tokens under a shared block-causal training objective. Through extensive experiments, they show that Mamba achieves constant cache memory and stable decoding latency as context length increases, while also exhibiting better length extrapolation than attention. The hybrid model offers a trade-off between efficiency and quality, largely matching attention on downstream tasks while reducing inference costs.

Critical strengths

The paper provides a substantial empirical evaluation, training attention, hybrid, and Mamba-based block diffusion models at approximately 3B parameters on 300B tokens each. Its coverage of inference efficiency, downstream performance, and length extrapolation provides useful evidence on the practical trade-offs between these architectures.

Critical weaknesses
  • Limited novelty. The paper largely builds upon established techniques, including block-causal training, exact caching, and fixed-size Mamba states. Its main contribution lies in scaling and systematically evaluating their combination, without demonstrating substantial methodological advances or novel insights beyond known properties of state-space models.

  • Limited explanatory insights beyond empirical validation. While the larger-scale evaluation provides useful evidence on the quality–efficiency trade-off, the additional insights beyond prior work appear limited. The constant-state efficiency advantage and the associated batching benefits are largely expected from the architecture. The quality and length-extrapolation results are more informative, but the paper provides limited analysis of the factors underlying these outcomes. Overall, the contribution is primarily empirical validation, with less explanatory insight than the breadth of experiments might suggest.

Questions for authors

Could the authors provide a deeper analysis of why Mamba and Hybrid exhibit better long-context generalization than Attention? Understanding the underlying factors behind these differences would provide more insights beyond the empirical observations.

3 comments

We thank the reviewer for the careful reading and for recognising the scale and coverage of the study. The revised manuscript adds an analysis of why the backbones differ at long context (App. G), NIAH results with confidence intervals (App. H) and a numerical check of cache exactness (App. E); new material is in blue. We address each concern below.

1. Addressing the novelty

We agree that the ingredients (block-causal training, exact caching, fixed-size Mamba state) are known. Our contribution is to establish their consequences at scale in a controlled setting: three 3B backbones trained on 300B tokens that differ only in the sequence mixer, with cached decoding verified to match full recomputation up to floating-point error (App. E). At this scale the results show behaviour that known SSM properties do not predict:

  • The efficiency advantage begins late. Up to 16k tokens attention decodes 4–17% faster than Mamba (Tab. 8); the constant state pays off only at tens to hundreds of thousands of tokens, or earlier under batching (App. C).
  • SSM retrieval is limited by distance, not length (2b): a needle 256 tokens back is retrieved equally well at 1k and at 16k.
  • The hybrid's weak point at long context is its attention layers, not its recurrence (2c).

2. Adding the explanatory analysis: why Mamba and the Hybrid generalize better

We add this as new App. G (Fig. 6, Tabs. 12–13): inference-only diagnostics on the 3B checkpoints (NIAH at fixed needle distance, attention statistics, SSM decay rates, hybrid ablations), with attention evaluated without RoPE rescaling, as in the paper. "Length-robust" here means that accuracy on a needle at a fixed distance does not change as the context grows; this is not the same as using distant context.

(a) Attention fails because the context is longer than training, not because the needle is far. At fixed distance it retrieves every needle at 1k and none at 2k, even 64 tokens back (Fig. 6). The 103 of 560 heads that attend to the needle at 1k lose it: their attention mass on it falls from 0.34 to 0.04 at 2k, and their entropy rises from 0.49 to 0.66 (0.81 at 16k). App. G.2 shows that the failure comes from the positional encoding used beyond its trained range, not from attention itself.

(b) Mamba's robustness and its short memory are the same property. Its state carries no positions and decays fast: the median head halves its state in 8.3 tokens, and no head keeps half its state over 8k tokens (Tab. 12). Tokens more than a few hundred back barely affect the state, whether the context is 2k or 16k. Accordingly, needles up to 256 tokens back are retrieved at 94–98% at every length to 16k, and needles 1k or more back at most 12% of the time. Its flat LongBench score shows that what it uses is not disturbed by longer inputs, not that it uses the extra context.

(c) The Hybrid inherits Mamba's robustness until its attention layers fail. Its five attention layers do its retrieval: at 1k, removing layer 12 or 18 drops exact match from 98% to 8% or 0%, while removing neighbouring Mamba layers leaves 97–100% (Tab. 13). It matches Mamba at short distances up to 4k, but at 16k loses even a needle 64 tokens back (0% vs 94%). Restricting its attention to the last 1,024 tokens, without retraining, restores in-block copying at 16k (0% → 100%) and brings its NIAH token accuracy from 1.7% to 23.0%, Mamba's level.

We have added the following in the revised manuscript:

  • At depth 100% the needle lies inside the decoded block, so this column of Table 2 and Figure 3 measures in-block copying; Mamba's 22.3% at 16× comes from it. At depths 0–75%, a 100-trial rerun gives at most 1.8% exact match at 4k and none at 8k–16k (App. H).
  • Long-context claims are now stated for attention without RoPE rescaling, and give the Hybrid's range as 8× and Mamba's as 16× (abstract, contributions, Table 2, §5.2–5.3).

We hope the new analysis provides the explanatory insight that you found missing, and addresses all your concerns. We would be glad to clarify any of your questions.

Thank you for the detailed rebuttal and additional experiments. The new analyses of attention behavior, SSM state decay, and hybrid ablations provide valuable insights and substantially address my concerns about the limited explanatory analysis. While the methodological novelty remains limited, I appreciate the more nuanced distinction between length robustness and long-range retrieval. Overall, I believe the additional evidence strengthens the paper sufficiently, and I am updating my rating from 2 to 3.

Thank you for taking the time to review our responses and updated experiments. We are glad that our clarifications and additional results have addressed your concerns, and we truly appreciate your positive feedback increasing the score.

Reviewer_dqMHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether block-diffusion language models can exploit a fixed-size state-space cache rather than a growing attention KV-cache. It pretrains three approximately 3B-scale block-diffusion denoisers—full attention, bidirectional Mamba-2, and an attention/Mamba hybrid—on ~300B tokens with a single-frontier block-causal training objective. The resulting inference interface stores the clean prefix in attention KV tensors or a forward recurrent state, while denoising a frontier block with local bidirectional computation. On an H100, the paper reports that at 256k-token cache depth the pure Mamba model uses ~7.6 GB of memory and ~6.8 ms per denoising step, versus ~81.9 GB and ~29.3 ms for attention. It also compares length extrapolation and downstream quality

Critical strengths
  1. A substantial controlled pretraining experiment. Pretraining attention, pure Mamba, and hybrid block-diffusion models on 300B tokens provides valuable large-scale evidence beyond the small-scale hybrid studies. The authors explicitly acknowledge that the underlying block cache and partial bidirectionality are not invented here, and position their contribution as a scale and empirical-behavior study.

  2. Strong architectural intuition and measured systems benefits. The cache distinction is clear: attention requires an O(L) clean-prefix KV history; a recurrent SSM can maintain a fixed-size forward state while restricting its reverse scan to the active block. The H100 latency, memory, throughput, batch-size, and FLOP comparisons support a meaningful long-context efficiency advantage for the pure SSM model.

  3. More than a speed-only evaluation. The paper reports NIAH passkey retrieval by length and position, LongBench, validation NLL, several downstream tasks, generative perplexity, and a denoising-step quality/throughput trade-off. Showing recency effects rather than hiding them is a strength.

Critical weaknesses
  1. The claimed long-context retrieval generalization is substantially stronger than the actual absolute performance. At 16k tokens (16× the 1024-token training length), the pure Mamba model achieves 22.3% mean passkey retrieval, Hybrid+NTK achieves 14.3%, and attention is at 0% (Table 2). Figure 3 reveals that most of Mamba's surviving performance comes from recent needles: at 16k, the most recent position remains ~99% correct while older positions are near 1–6% correct. This is evidence for a long recency window, not reliable information retrieval throughout a 16k-token context. Similarly, LongBench macro scores around 10% for Mamba at 16k (versus ~3–5% for attention) are very low in absolute terms; Table 3 averages just 20 examples per task. The paper can correctly claim relative length-robustness over its attention baseline, but 'retrieve out to 8–16×' should not be understood as reliable long-context capability. Please emphasize absolute and position-stratified accuracies, uncertainty, and practical performance thresholds in the abstract and discussion.

  2. The attention reference is deliberately trained for only 1024 tokens and receives limited position-extrapolation adaptation. A model with RoPE calibrated to a 1024-token context is not a competitive long-context attention baseline at 8–16k; the resulting 12.3% NIAH at 2k is an especially weak reference. The +NTK adaptation in Table 2 is stated to be the identity through 4k, meaning it does not address the failure at 2k and 4k where the core extrapolation claim is established. Better-known inference-time RoPE/YaRN or position-interpolation adjustments should be tested at the relevant lengths, or the conclusion explicitly limited to unadapted short-context-trained attention. This concern is about interpreting the mechanism, not demanding entirely new 300B-token pretraining. The observed superiority may reflect position encoding and architecture choices jointly, rather than being a universal attention-versus-recurrence result.

  3. An AR state-space baseline is needed to isolate the value of the diffusion part of the design. Fixed-size recurrence and favorable length extrapolation already characterize autoregressive Mamba-type models, as the authors explicitly acknowledge. The paper's contribution is that such state can be used with block diffusion, but it never benchmarks an AR Mamba/Hybrid reference for throughput at comparable generation quality, context length, batch size, and hardware. This leaves open whether the observed speed/memory frontier is mainly inherited from the SSM mixer and whether multi-token diffusion improves end-to-end inference over a simpler AR recurrent model. A controlled inference comparison to an existing comparable AR SSM checkpoint—or at minimum a reasoned accounting of AR recurrent throughput, decode passes, and accuracy—would distinguish the block-diffusion benefit from the state-space benefit. Related partial-bidirectionality work from Chaturvedi et al. is already discussed, so novelty should continue to emphasize scale rather than architectural priority.

  4. The supposed 'identical' architectural and training controls have consequential differences. Table 5 gives 3.033B parameters for Attn, 3.360B for Hybrid (+11%), and 3.431B for Mamba (+13%). Appendix Table 6 also changes peak learning rate (1e-4 for attention/hybrid, 5e-5 for Mamba) and gradient clipping (1.0 versus 0.5). The authors correctly report these differences and show analytical FLOPs, but the data do not fully isolate the choice of mixer from parameter budget and optimization recipe. Importantly, Mamba's validation NLL is 1.134 versus 1.104/1.102 for attention/hybrid, and its eight-task macro-average is 0.421 versus 0.434/0.432. It would be useful to report learning curves, sensitivity, and at least one parameter- or short-context-FLOP-matched variant or clarify why the selected settings are the most appropriate architecture-specific optima. The hybrid results are promising, but near-equality on eight tasks is not a rigorous equivalence test without uncertainty intervals.

  5. The 256k systems headline is not paired with a 256k retrieval-quality test or a fully comparable batch setting. Systems latency/memory are measured to 256k, but NIAH and LongBench stop at 16k, and all models were trained at 1k. Thus the 256k demonstration establishes the engineering cache bound, not useful semantic access to a 256k context. At 256k, the '14× aggregate throughput' compares Mamba batch 8 against attention's largest feasible batch 1, whereas the fair batch-1 comparison is 297 versus 114 tokens/s (~2.6×). Both are legitimate system facts, but the distinction should be prominent. Furthermore, the latency method uses CUDA-graph step replay and the reported generation throughput sums block latencies across cache depths; please clarify whether initial prompt-cache construction, cache warm-up, allocation, host synchronization, and full end-to-end generation are included. A numeric cached-versus-uncached-logit equivalence test would also independently verify the 'exact' cache claim for all three architectures.

Questions for authors
  1. What is the NIAH retrieval accuracy at 8k/16k conditioned on older/middle needle positions, with uncertainty, and does attention remain this weak under position-rescaling configured for extrapolation beginning at 2k?

  2. Can the authors provide quality-matched throughput/latency for a comparable AR Mamba or Hybrid inference baseline? What percentage of the advantage comes from diffusion parallelism rather than simply constant-size SSM state?

  3. Are the cache-equivalence claims empirically exact (maximum absolute cached vs. full-recompute logits error) and does throughput include context prefill/cold-cache construction?

  4. How do the findings change when S, block size G, or architecture parameters are varied at an otherwise matched budget?

4 comments

We thank the reviewer for a careful review and for the precise questions. The revised manuscript adds needle-in-a-haystack (NIAH) and LongBench results with confidence intervals (App. H), RoPE rescaling anchored at the training length (App. I), a numerical check of cache exactness (App. E) and an analysis of the long-context mechanisms (App. G), and states exactly what the timings include (§4, Tab. 8); new material is in blue. We address each question below.

1. Adding position-stratified NIAH accuracy with uncertainty

We re-ran NIAH with 100 trials per (model, length, depth) and report exact-match passkey accuracy with Wilson 95% intervals (new App. H).

  • Older and middle needles at 8k–16k. At depths 0–0.75, exact match is 0/100 for all three models at every depth (Wilson upper bound 3.7%), with token accuracy at most 6.9%.
  • Depth 1.0 is in-block copying. At depth 1.0 the needle lies inside the 32-token block being decoded, so this column measures copying within the block, not retrieval from the cache. Mamba's 22% at 16k in Table 2 comes from this column. The captions and §5.2 now say so.
  • Effective context. Under exact match, the largest length at which depth-averaged accuracy is at least 50% is 1,024 tokens for all three models. At 2k, depths 0–0.75, exact match is 0.0% [0.0, 1.0] for attention, 23.2% [19.4, 27.6] for the Hybrid and 28.2% [24.1, 32.9] for Mamba. At 4k it is 0, 1.8 and 0%.

New App. G explains this profile. With the needle-to-query distance held fixed, Mamba retrieves needles up to 256 tokens back at 94–98% at every length to 16k, and needles 1k or more back at most 12% of the time. Its robustness to length is a recency window, not long-range recall. The abstract now says "length-robust" rather than "retrieve", and the conclusion states this limit.

2. Addressing RoPE rescaling for attention

We thank the reviewer for pressing this point: with rescaling anchored at the training length, attention does not stay this weak. The submission's +NTK rows were anchored at 4,096, not 1,024, so they were the identity up to 4k, as the reviewer noted. We re-ran NIAH with PI, NTK-aware scaling and YaRN (α = 1, β = 32, untuned), all anchored at 1,024 (new App. I). Exact match at depths 0–0.75 (80 needles per cell), with depth-averaged token accuracy in brackets:

2k4k8k16k
Attn, no rescaling0 (13.3)0 (1.1)0 (0)0 (0)
Attn + PI67.5 (90.9)0 (34.4)0 (9.4)0 (2.6)
Attn + NTK25.0 (63.3)0 (7.7)0 (1.1)0 (0)
Attn + YaRN100 (100)100 (100)100 (100)100 (100)
Hybrid, no rescaling21.2 (53.9)0 (36.7)0 (24.3)0 (1.7)
Hybrid + YaRN98.8 (99.9)98.8 (99.9)98.8 (99.9)98.8 (99.9)
Mamba (no RoPE)†28.2 (76.9)0 (32.5)0 (23.7)0 (24.0)

Wilson 95% intervals: 100% → [95.4, 100]; 0% → [0, 4.6]. †100-trial run (App. H); from 4k on, Mamba's token accuracy comes almost entirely from the in-block depth 1.0. Hybrid + PI/NTK are in App. I.

  • YaRN restores retrieval of older and middle needles to 16k, so the collapse in Table 2 is a property of unadapted RoPE, not of attention. Controls confirm the needle is read from the context: with it removed, exact match is 0; with a different value planted, the model returns that value (Attn 100%, Hybrid 97%).
  • PI and NTK recover attention only partly at 2k. Under them the Hybrid keeps higher token accuracy than attention at every length.
  • Mamba has no positions to rescale and stays limited to its recency window. Our NIAH is a copy task (the query repeats the needle sentence), so it is easy once positions are in range.

We restate the conclusion as the reviewer suggests: without rescaling, attention trained at 1k collapses past 2× while the linear-state backbones need no adjustment; with YaRN, attention and the Hybrid both retrieve almost every needle to 16k. The state then buys cost, not accuracy: Hybrid + YaRN matches Attn + YaRN on NIAH and decodes at 2.7× lower latency with 3.8× less memory at 256k (Tab. 8), which rescaling does not change. The abstract, contributions, §5.2, Tables 2–3 and the conclusion are revised accordingly.

3. Clarifying where the speedup comes from: constant-size state vs. diffusion parallelism

An AR Mamba model has the same O(1) state. The speedup therefore factors into two independent terms:

speedup over AR attention = (state factor: mixer cost) × (diffusion factor: tokens per forward and cost per forward).

State factor (measured). Every comparison of backbones in the paper uses the same decoder (same G = 32 and denoising budget S), so all three backbones have the same tokens per forward (TPF). Diffusion parallelism therefore cancels out of every Mamba- or Hybrid-vs-attention ratio. The full 2.6× at 256k (Tab. 8), and the 1.64× and 1.45× at 64k, come from the constant-size state. App. F (Tab. 11) supports this factorisation. As S varies 16-fold, throughput changes 12-fold, but the ratio to attention barely moves:

S (TPF incl. commit)2 (10.7)8 (3.56)16 (1.88)32 (0.97)
Mamba / Attn, 64k1.791.681.651.64
Hybrid / Attn, 64k1.561.491.461.45
Mamba / Attn, 4k0.950.890.880.87

(Directly wall-clocked tok/s including commit passes, batch 1, H100.) At 4k the state does not yet pay off, at any S.

Diffusion factor (vs. AR, by accounting). Relative to AR decoding on the same backbone, block diffusion changes two things: tokens per forward, from 1 to 32/(S+1) counting the commit pass, and the work per forward, from one token to a 32-token block. At the paper's default S = 32, TPF is 0.97 and each forward processes a full block, so block diffusion is not faster than AR recurrent decoding, and we make no such claim. A gain over AR requires small S (S ≤ 8, TPF ≥ 3.6), which costs generation quality (Gen-PPL 12.1–13.1 at S = 8 vs 6.4–6.5 at S = 32; Tab. 11).

Answer to "what percentage".

  • Relative to block-diffusion attention: all of the advantage comes from the state, because diffusion parallelism is common to all three backbones.
  • Relative to AR decoding: at quality-preserving S, none of it comes from diffusion parallelism. Diffusion trades quality for speed only at small S.

Public AR SSM checkpoints differ from ours in training data, tokenizer, size and context length, so they would not isolate the effect of the decoder. A matched comparison would need an AR 3B model trained on our data, which is outside the scope of this study. The claims of the paper concern the cache, and the state factor above is measured under a matched decoder.

4. Clarifying cache exactness and what the timings include

Exactness. The cache is exact up to floating-point error, which we now measure (new App. E, Tab. 10). On all 128 blocks of a 4k-token WikiText-103 sequence, we compare cached-decoding logits with a full block-causal recomputation over the same prefix. Maximum absolute logit error over all blocks and positions (logits reach magnitude 25–28):

fp32 max |Δ|fp32 arg-max agreementbf16 max |Δ|bf16 per-block medianbf16 arg-max agreement
Attn1.9e-4100%1.060.12599.1%
Hybrid9.0e-4100%0.250.12597.3%
Mamba7.6e-399.95%0.160.12597.3%
  • For Attn and Hybrid, fp32 cuts the bf16 gap by two to three orders of magnitude; Attn in fp64 gives 2.5e-6, so the residual is rounding. Mamba's fp32 gap is the largest (KL 3.8e-8), and its kernel has no fp64 path to test further.
  • The bf16 median does not grow with depth, so errors do not compound; the worst case grows mildly (Attn's 1.06 is one block at 2–4k). The check covers one sequence per model, up to 4k.

Prefill and cold-cache construction. Throughput excludes them, as §4 and the Tab. 8 caption now state.

  • Tab. 8 sums per-step latencies of cached denoising steps (CUDA-graph replay, median of 30), excluding the commit pass, prompt prefill and host synchronisation, on randomly initialised weights (latency does not depend on the weight values).
  • App. F wall-clocks generation with the trained checkpoints, including every denoising and commit forward and sampling, after an untimed prefill. At S = 16 and 64k, Mamba and the Hybrid decode 1.66× and 1.47× faster than attention.
  • We did not measure prefill or time-to-first-token; all reported throughputs are steady-state decode rates.

5. Clarifying how the findings depend on S, G and architecture

The paper has three findings: (i) a constant-cost state, (ii) length robustness without positional adjustment, and (iii) quality parity of the Hybrid with attention. Each rests on a different mechanism, and each mechanism tells us what should move with S, G and architecture. Where we have measurements we give them; where we do not, we say so.

S (measured, App. D and F, S = 2–32). S moves all three backbones together and leaves the comparison between them unchanged.

  • Each halving of S gives 1.7–2.0× more throughput and worse Gen-PPL, for every backbone.
  • The state advantage does not depend on S. At 64k, Mamba/Attn is 1.64–1.79× and Hybrid/Attn 1.45–1.56× at every S (table in point 3).
  • LongBench at 4k keeps the Hybrid and Mamba above attention at every S.
  • Quality is where S matters. At S ≥ 16 the backbones are within 0.5 Gen-PPL; at S ≤ 8 attention has the lowest Gen-PPL, by 0.75–3.86. Parallel decoding therefore costs attention the least quality.

Architecture: the attention fraction (measured across our three models, at the same budget, depth, width, G and data). The three backbones are a sweep of the number of attention layers (28, 5 and 0 of 28).

  • Cost scales with the number of attention layers. From 64 to 256k tokens, the Hybrid's latency grows by 4.2 ms against attention's 23.8 ms, and its memory by 13.7 GB against 75.5 GB. Both ratios are 0.18, against 5/28 = 0.179 (Tab. 8). The long-context cost of any other mixing ratio follows from this, and §5.4 gives the FLOPs added by each attention layer (Tab. 5).
  • Which layers carry attention also matters. Removing attention layer 12 or 18 drops the Hybrid's short-range retrieval from 98% to 8% or 0%, while removing the neighbouring Mamba layers leaves it at 97–100%. Restricting all five attention layers to a 1,024-token window restores in-block copying at 16k (App. G.4, Tab. 13).
  • A few attention layers do the retrieval, and they are also the part that breaks at length.

G (not varied; the models are trained only at G = 32). We did not retrain at other G. Decoding a G = 32 model at another G would measure the mismatch with training, not the effect of G. What the mechanisms imply:

  • Cost. Attention reads its whole KV-cache at every step whatever G is, while Mamba's step touches only the G current tokens. G and S set the tokens per forward, and so the throughput of all three backbones, but not the O(L) vs O(1) difference. G can move the length at which the state starts to pay off (16k–64k here), but it cannot change the ordering at long context.
  • Length robustness. Every query attends over the full clean prefix for any G, so attention's positional failure past the training length does not depend on G (§5.2). Mamba's recency window is set by its decay half-lives, which are measured in tokens (App. G.3), not by G.
  • The one G-dependent effect in our evaluation. NIAH depth 1.0 places the needle inside the decoded block, so the share of in-block copying in that column grows with G. This is why we report depths 0–0.75 separately (App. H).
  • Quality. Block diffusion approaches AR as G shrinks, and smaller blocks give better likelihood (Arriola et al., 2025). We did not measure whether this trend differs between backbones.

We hope these answers address your concerns, and we would be glad to clarify any further questions.

We thank the reviewer again for the positive assessment and the detailed, constructive feedback. We have incorporated their suggestions into the revised manuscript (marked in blue), including the position-stratified NIAH results, the RoPE-rescaling comparison, the cache-exactness check and the clarified timing scope. We would be glad to answer any further questions.

Reviewer_ZVKNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the use of state-space models to improve the efficiency and long-context capabilities of block diffusion language models. The authors compare three backbones (Transformer attention, bidirectional Mamba-2, and a Mamba–Attention hybrid), pretrained on 300B tokens under a unified training objective. By replacing the growing KV cache with a fixed-size recurrent state, the Mamba-based model achieves constant-memory caching and length-independent per-step decoding latency.

Critical strengths
  1. The paper focuses on an important problem: how to make diffusion language models more efficient and more effective on long-context tasks.
  2. The experimental section reports results across three backbones, a unified training budget of 300B tokens, and long-context latency/memory/throughput measurements.
Critical weaknesses
  1. The novelty of the paper is very limited. SSM states, Mamba–Attention hybrids, and block diffusion have all been proposed in prior work, and the conclusion that applying linear-state or recurrent architectures to block diffusion yields speedups is rather obvious and has already been comprehensively validated in previous works such as FLARE [1] and DiffuMamba [2].
  2. The paper mainly focuses on training linear-attention-based block diffusion from scratch. Could the approach instead be adapted from an AR-pretrained LLM? This would likely achieve better results and be more practically valuable, especially since nearly all advanced dLLMs in the past year have been trained from AR-pretrained LLMs.
  3. Compared with the related work FLARE [1] and DiffuMamba [2], what are the main methodological differences and advantages of this paper?
  4. Although the paper reports decoding throughput under different denoising budgets, it does not explicitly analyze tokens per forward (TPF) or provide a comprehensive accuracy–TPF–throughput trade-off. Since parallel token generation is a key advantage of diffusion LMs over AR models, the authors should evaluate whether the proposed architectures can achieve competitive TPF at comparable generation quality, rather than focusing primarily on reducing the cost of each forward pass.

[1] FLARE: Diffusion for Hybrid Language Model
[2] DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone

Questions for authors

See Weaknesses

4 comments

We thank the reviewer for the constructive review. The revised manuscript adds the accuracy–TPF–throughput trade-off (App. F), a numerical check of cache exactness (App. E) and a comparison with FLARE in the related work (§2); new material is in blue. We address each concern below.

1. Addressing the novelty

We agree that the ingredients (block diffusion, Mamba denoisers, block caching) are known, and that FLARE and DiffuMamba point in the same direction. Our primary contribution is to establish these gains at scale: three 3B-parameter backbones trained on 300B tokens under identical conditions, measured from 64 to 256k tokens of context. Training with the block-causal objective makes the block cache exact up to floating-point error, so these measurements reflect the trained model itself rather than an approximation of it. DiffuMamba trained full-sequence denoisers up to 1.3B and used a block cache only to benchmark speed, on models not trained for it. FLARE reports no cache-exactness check, and neither work evaluates long-context retrieval (point 3 below).

At this scale, the measurements add specifics that the asymptotic argument leaves open:

  • Where the speed-up begins. Up to 16k tokens attention decodes 4–17% faster than Mamba (§5.1, Tab. 8); Mamba leads only at tens to hundreds of thousands of tokens. Batching moves this point: attention's last feasible length falls from 256k at batch 1 to 32k at batch 8, while Mamba stays at ~8 GB (App. C, Fig. 4).
  • Which backbone keeps quality. The Hybrid has the best validation NLL (1.102 vs 1.104, Tab. 1) and ties attention downstream (macro 0.432 vs 0.434, Tab. 4); pure Mamba trails by about one point (0.421).
  • That the cache is exact (new App. E, Tab. 10). On all 128 blocks of a 4k-token WikiText-103 sequence, cached decoding in fp32 matches the full block-causal forward: max |Δlogit| is 1.9e-4 (attention), 9.0e-4 (Hybrid) and 7.6e-3 (Mamba) against a logit scale of 25–28, with top-1 agreement of 100% / 100% / 99.95%. Attention in fp64 gives 2.5e-6; the Mamba-2 kernel does not support fp64. In bf16 the per-block median error is 0.125 for all three models with no trend over depth, so errors do not compound across blocks.

2. Addressing AR-initialization

Yes, in principle. The block-causal objective, and the exact cache it yields, do not depend on how the weights are initialized, so an AR-initialized model trained with this objective would have the same cache.

How best to convert an AR model into a block-diffusion model is, however, a separate research problem, with its own design choices (attention masks, adaptation data and budget, the move from causal to bidirectional attention). It is the focus of dedicated work such as Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed (arXiv 2512.14067, ICML 2026) and Nemotron-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context (arXiv 2606.26493), both from NVIDIA, and earlier DiffuLLaMA (Gong et al., ICLR 2025). Our question is different: what the sequence mixer itself buys under block diffusion.

We trained from scratch because this is a controlled comparison of mixers: the three backbones share data, tokenizer, objective, schedule and token budget, and differ only in the mixer (Section 4, Tab. 1). With AR parents, differences in the results could also come from the parents' pretraining. In addition, an AR Mamba model has only a forward scan; the reverse scan of a bidirectional denoiser has no AR counterpart and would be trained in any case.

3. Clarifying the differences from FLARE and DiffuMamba

DiffuMambaFLAREThis paper
Trainingfrom scratch, full-sequence masked diffusion, up to 1.3B / 120B tokensSFT conversion of AR Qwen3.5 (2B/4B/9B), ~10B tokensfrom scratch, block-causal, 3B / 300B tokens
MixersTransformer, Mamba-2, hybridGated DeltaNet–attention hybrid onlyattention, hybrid, pure Mamba-2, identical otherwise
Block cachethroughput benchmarks only, on models not trained for itrecurrent state reused; no exactness check reportedtrained for; exact up to floating-point error, measured (App. E)
Evaluationquality at 1k context; throughput to 260kreasoning, knowledge, math, codequality, NIAH and LongBench to 16k; throughput to 256k

The main differences are two. DiffuMamba trains full-sequence denoisers and uses a block cache only to benchmark speed, naming joint training with a block cache as the next step; we take that step at 3B scale, which makes the cache exact up to floating-point error and lets us measure quality and long-context behaviour under it. FLARE converts AR attention hybrids for reasoning tasks; it has no attention-free model, reports no cache-exactness check, and does not evaluate long-context retrieval.

4. Adding the accuracy–TPF–throughput analysis

We agree, and we add it as new Appendix F (Tab. 11, Fig. 5). Each 32-token block takes S denoising forwards plus one commit forward, so TPF = 32/S, or 32/(S+1) with the commit pass. Throughput is wall-clocked end to end, including every commit pass and sampling step, at batch 1 on one H100 with CUDA graphs; values are the median over 3 jobs × 5 repeats. LongBench is at 4k context, without RoPE rescaling. Values are Attention / Hybrid / Mamba:

TPF (S)16 (2)8 (4)4 (8)2 (16)1 (32)
Gen-PPL ↓17.6 / 20.4 / 21.415.3 / 17.1 / 18.512.1 / 12.9 / 13.19.2 / 9.1 / 9.56.4 / 6.5 / 6.5
LongBench@4k ↑3.8 / 6.3 / 7.04.3 / 7.3 / 7.44.5 / 9.4 / 8.34.9 / 9.7 / 9.44.6 / 10.2 / 10.0
tok/s, 4k context1661 / 1591 / 1578993 / 923 / 908551 / 501 / 491291 / 262 / 256150 / 134 / 131
tok/s, 64k context880 / 1370 / 1578528 / 797 / 907292 / 434 / 491155 / 227 / 25680 / 116 / 131
  • TPF trades speed for quality on every backbone. Each halving of S gives 1.7–2.0× tok/s, and Gen-PPL worsens monotonically as S falls. At TPF ≤ 2 the backbones are within 0.5 Gen-PPL of one another; at TPF ≥ 4 attention has the lowest Gen-PPL, by 0.75–3.86. The Hybrid and Mamba score above attention on LongBench at every TPF.
  • Context length is where the backbones differ. At 4k attention is the fastest at every TPF. At 64k it falls to 53% of its 4k rate, the Hybrid keeps 86–87%, and Mamba is unchanged. At TPF = 2 and 64k, Mamba is 1.66× and the Hybrid 1.47× faster than attention.
  • TPF and per-forward cost are separate levers. Throughput is TPF times forwards per second: the sampler sets TPF, and the mixer sets the cost of each forward, which for Mamba does not grow with context. Confidence-based adaptive decoding changes only how many positions are revealed per forward, so it can run inside the same cached generator; we have not measured it and note this in App. F.

We hope this clarifies the overall contribution: the gains of a constant-memory block cache for state-space diffusion denoisers, established at 3B scale with a cache verified to be exact up to floating-point error, and measured across long contexts, including where it does not help.

We thank the reviewer again for their review. As the discussion period nears its end, we would like to check whether our responses have addressed your concerns. In the revised manuscript (marked in blue), we added the accuracy TPF throughput trade-off (App. F), a numerical check that the cache is exact up to floating-point error (App. E), and the comparison with FLARE (section 2). Our response also clarifies the differences from FLARE and DiffuMamba and our choice to train from scratch.

If any concern remains, please let us know what we could clarify further to improve the assessment, and we would be glad to address it.

Thanks for the detailed response and it solves my concern. I would increase my score to weak accept

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback and the AC for their time. The reviewers value the controlled 3B / 300B-token comparison of attention, Hybrid and Mamba (dqMH, 9ext, ZVKN) and the measured latency, memory and throughput benefits of the constant-size cache (dqMH, ZVKN).

We addressed all concerns; changes are in blue in the revised manuscript. Reviewer 9ext found that the new analysis "substantially address" their concerns, moving the scores from (3, 2, 2) to (3, 3, 2).

Concern (reviewers)What we addedMain result
Why Mamba and the Hybrid generalize better (9ext, dqMH)App. G: fixed-distance NIAH, attention statistics, SSM decay, Hybrid ablationsAttention fails from positions beyond training, not needle distance. Mamba's retrieval depends on distance, not length (94–98% at ≤256 tokens, to 16k). The Hybrid retrieves through its attention layers.
Strength of long-context claims (dqMH)App. H: 100 trials, Wilson CIs, by depthClaims restated as "length-robust" (Mamba 16×, Hybrid 8×) throughout.
RoPE rescaling (dqMH)App. I: PI, NTK, YaRN anchored at 1kClaims scoped to unadapted RoPE. With YaRN, the Hybrid matches attention at 2.7× lower latency and 3.8× less memory at 256k.
Novelty vs. FLARE, DiffuMamba (ZVKN, 9ext)§2; App. EA controlled study at scale, with an exact block cache and long-context retrieval evaluation.
Cache exactness and timing scope (dqMH, ZVKN)App. E; §4, Tab. 8fp32 max logit error ≤ 7.6e-3 vs. logits of 25–28; no compounding over blocks. Timing scope stated.
TPF–quality–throughput (ZVKN, dqMH)App. FThe state's advantage holds at every TPF (Mamba 1.64–1.79×, Hybrid 1.45–1.56× at 64k).

We thank the reviewers and the AC again, and hope these additions help in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.