acceptodds
Under review as a conference paper at ICLR 2027

Training Hybrid Block Diffusion Language Models with Partial Bidirectionality

Abstract

High-throughput long-context generation is one of the central challenges for large language models. Generation is typically memory-bandwidth-bound rather than compute-bound: each decoding step must stream the accumulated key/value (KV) cache from memory, so bandwidth demand grows with context length while only one token is emitted. Two parallel approaches have therefore emerged: reducing memory access with efficient attention variants and linear-time mixers such as Mamba, or increasing parallel computation by generating blocks of tokens at once. However, technical challenges arise when combining these two ideas. Earlier hybrid diffusion models such as DiffuMamba singh2026diffumamba use bidirectional Mamba mixing, including a reverse-direction scan relative to causal generation. This reverse scan needs to scan the entire sequence, so its states are not prefix-only and cannot be precisely reused as a cache even when diffusion is performed block by block. We propose a BDLM Mamba–attention hybrid that addresses this challenge by restricting the reverse Mamba scan to the active denoising block, which enables exact caching across blocks. In an 87M-parameter DCLM sweep, BDLM Mamba-H achieves the best C4-en validation perplexity compared to BDLM attention and full-sequence baselines. At 350M parameters, it remains competitive with BDLM attention. For long-context inference, BDLM Mamba-H reaches 19.7x the throughput of full-sequence DiffuMamba-H at 65K tokens and 3.7x the throughput of BDLM attention at 262K, showing that Mamba hybrids are a potential long-context diffusion architecture.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Diffusion Language Models, dLLM, Block Diffusion, Discrete Diffusion, Hybrid Language Models, Training, Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_eM7hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims to extend block diffusion LMs to Mamba–attention hybrid backbones. It restricts bidirectional Mamba (forward and reverse scans) to the active denoising block and uses standard cached Mamba states for completed blocks to enable block-wise caching. Training with these caches uses an AdaLN design. Experiments at 87M and 350M parameters show lower perplexity than attention-based BDLMs.

Critical strengths
  1. This work explores a natural extension of block diffusion models to hybrid architectures. Combining block diffusion with linear attention is helpful for improving long-context generation efficiency.

  2. The proposed design is simple and well motivated, providing a natural extension of bidirectional Mamba in diffusion models.

  3. The experiments are controlled, and the long-context throughput gains are promising. Each architecture receives its own learning-rate sweep, and BDLM Mamba-H’s throughput advantage continues to grow with sequence length.

Critical weaknesses
  1. The technical novelty is limited. The paper finds that conditioning the Mamba component in an active block on a causal prefix Mamba cache reduces perplexity compared with fully bidirectional models. However, this insight has been widely used in full-attention models, such as “Set Block Decoding is a Language Model Inference Accelerator” and “Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding.” Both use causal masks to encode context and find that this improves diffusion performance. This work essentially applies the same idea to Mamba layers.

  2. Based on the insight in point 1, the most relevant baseline would also use causal context encoding for full-attention layers, but this baseline is missing. Directly comparing the proposed architecture with fully bidirectional diffusion models may not be an apples-to-apples comparison.

  3. Another major concern is the practical value of the throughput results. First, it is unclear why randomly initialized models are used. More importantly, the paper does not demonstrate the potential for parallel sampling in diffusion LMs, so it is unclear why the proposed approach would be preferred over AR models. In addition, there is no comparison with autoregressive Mamba–attention hybrids.

  4. The evidence for generation quality is weak. Experiments at larger scales and on more complex benchmarks would be desirable. The authors could consider converting an existing AR hybrid model into a diffusion model to demonstrate this at a reasonable cost.

Questions for authors

I have included my questions in the weakness section.

One particular question is that what is "random-initialization" for inference throughput measurement? Why use this strategy?

2 comments

Response to Reviewer eM7h (Part 1/2)

Thank you for the clear feedback and for noting our controlled per-architecture learning-rate sweeps and long-context throughput scaling. Below we address each point and note the clarifications we have made in the revised draft.

Causal-context full-attention baseline and technical contribution on Mamba-2 (W1, W2). On the baseline comparison in W2, our primary baseline (BDLM attention) already uses causal context encoding across completed prefix blocks (BD3-LM, Arriola et al., ICLR 2025, Eq. 6), storing causal prefix representations in a reusable K/V cache and applying bidirectional attention only inside the active 256-token block. We compare BDLM Mamba-H head-to-head against BDLM attention across every experiment in the paper (the 87M sweep in Tables 1 and 5, the 350M scale-up in Tables 2–4, and long-context inference in Table 6), while full-sequence attention and full-sequence DiffuMamba-H serve only as non-cacheable full-window references. On novelty relative to full-attention causal-prefix designs (W1), in Transformer block models such as BD3-LM, Set Block Decoding (Gat et al., 2025), and Nemotron-Labs-Diffusion (Fu et al., EMNLP 2026), conditioning an active block on a causal prefix is purely a 2D attention-mask change over standard K/V projections. In bidirectional Mamba-2 (DiffuMamba, Singh et al., ICML 2026), a full-window reverse scan instead propagates noisy active-block tokens backward into every prefix state and prevents exact caching across blocks. Making Mamba-2 natively block-cacheable therefore requires decoupling the forward recurrent boundary cache (conv1d plus selective SSM state skℓs_k^\ell) from a strictly block-local reverse scan (Eqs. 9–11), training those recurrent prefix boundary states end-to-end through downstream block losses across all blocks in parallel (Sec. 4.1), and factoring AdaLN so prefix scans remain independent of the reverse timestep tt (Eq. 7, Appendix E). We have cited Set Block Decoding and Nemotron-Labs-Diffusion in Section 2 and made BDLM attention's causal prefix mask explicit.

Random-init 700M throughput benchmarks, parallel sampling, and AR hybrids (W3, Q1). Regarding Question 1, "random initialization" means profiling the compiled model and CUDA/Triton kernels from freshly initialized weights without pretraining a 700M checkpoint. Our native BDLM sampler (Appendix B, Table 9) runs a fixed schedule (B=256B = 256, p=16p = 16) in which every block launches the exact same FlashAttention, Mamba-2, and GEMM kernels on identical tensor shapes regardless of parameter values, so FLOPs, HBM traffic, and tokens/s are identical to those of a trained checkpoint. Following DiffuMamba (which profiles ~1.3B–1.4B models on H100), we ran this benchmark at 700M to evaluate a model larger than 350M that better saturates an A100-80GB GPU and provides a clearer comparison across architectures up to 262K tokens, as we have clarified in Section 5.3. On parallel sampling and AR hybrids, within each 256-token block all 256 positions are denoised in parallel over p=16p = 16 reverse steps (plus 1 clean cache append), amortizing prefix-cache reads across B/p=16B / p = 16 generated tokens per denoising step rather than streaming the accumulated cache once per token as in AR decoding. Mamba-2 hybridization and block-parallel decoding address complementary bottlenecks in long-context serving, since hybridization at 350M reduces attention K/V cache state by 6×6\times (retaining attention in only 3 of 18 layers) while block diffusion cuts sequential cache-reading passes per 256 emitted tokens from 256 down to p+1=17p + 1 = 17.

Response to Reviewer eM7h (Part 2/2)

Evaluation benchmarks at 87M/350M and why AR-to-BDLM Mamba conversion is impractical here (W4). Our 87M and 350M checkpoints are base models pretrained from scratch on 5B–17.65B DCLM tokens without instruction tuning or RL post-training. More complex reasoning or free-form generation benchmarks would be ill-suited for unaligned base models at 87M–350M scale, where performance largely reflects prompt-format sensitivity and near-chance variance; following standard base-DLM practice (BD3-LM; DiffuMamba), we evaluate held-out C4-en and Paloma-C4 NELBO PPL/BPB alongside six zero-shot fixed-choice MCQA tasks (Sec. 5.1).

Converting an existing pretrained AR hybrid into a BDLM Mamba model is also impractical for two concrete reasons. First, open pretrained AR hybrids lack checkpoints matching our sequential sparse-attention Mamba-2 architecture at this scale. Sequentially interleaved Mamba-2 hybrids such as Nemotron-H (8B–56B), Bamba-9B (9B), and Jamba (52B MoE) are an order of magnitude larger, while smaller 0.5B–1.5B candidates use incompatible topologies (Falcon-H1-0.5B/1.5B and Hymba-1.5B, ICLR 2025 place attention heads in parallel inside every layer and retain an O(L)O(L) K/V cache in all layers, whereas Zamba2-1.2B uses a single weight-shared attention block with LoRA adapters). Second, efficient AR-to-BDLM conversion recipes such as Fast-dLLM v2 (Wu et al., ICLR 2026) rely on two Transformer properties that break down on Mamba-2. In a Transformer, replacing a causal mask with a block-bidirectional mask reuses pretrained Q/K/V weights directly, whereas an AR Mamba-2 layer contains only forward scan parameters (Fθ,→F_{\theta,\rightarrow}) and lacks the reverse SSM branch (Fθ,←F_{\theta,\leftarrow}, Eq. 10) required for intra-block bidirectionality, meaning conversion would still have to train newly initialized reverse conv/SSM parameters across 83% of layers. Moreover, Fast-dLLM v2 relies on a hierarchical sub-block cache that stores K/V states for arbitrary unmasked positions inside a partially decoded block; in Mamba-2, the forward recurrent state depends contiguously on all preceding in-block tokens (including still-masked slots whose inputs change at the next step) and the reverse scan sweeps right-to-left from the block end, preventing sub-block recurrent state caching.

Reviewer_f7wJOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies block diffusion language models with a hybrid Mamba–attention backbone. Its main architectural change restricts reverse-direction Mamba scans to the active denoising block, allowing completed prefixes to be represented by reusable forward recurrent states. The proposed training procedure constructs these prefix states and propagates downstream block losses through them, aligning training with the cache interface used during inference.

The paper evaluates language-modeling quality at approximately 87M and 350M parameters and benchmarks long-context inference using randomly initialized 700M models. It reports competitive validation quality and substantial throughput improvements at long generation lengths.

Critical strengths
  1. The authors addressed a concrete and relevant technical problem and solved it with a reasonable solution. Restricting reverse scans to the active block preserves local bidirectional mixing while enabling prefix reuse. Training through the same recurrent states consumed at inference is a sensible design. The experiments results also shows strong speedup on long sequence generation while maintaining the generation quality.
  2. Concrete systems measurements. The paper provides raw throughput measurements and implementation details. The observed transition from slower short-sequence inference to faster long-sequence inference is consistent with the intended reduction in prefix-processing and attention-cache costs.
Critical weaknesses
  1. The incremental contribution is not sufficiently isolated. The method combines existing block diffusion, Mamba-2, and sparse-attention hybrid components through a relatively focused change to scan directionality and prefix-state training. This can be valuable, but the experiments do not establish which proposed components are essential. In particular, a forward-only Mamba BDLM would help determine whether block-local reverse scans provide a meaningful quality benefit. A prefix-gradient ablation would similarly clarify the importance of training the cached representations through downstream blocks.
  2. Quality and efficiency are demonstrated in separate settings. Quality results use trained 87M-class and 350M models, whereas the headline inference results use randomly initialized 700M models. Random weights are useful for measuring execution costs, but these measurements do not establish that a trained model achieves the reported speedups while retaining comparable generation quality.
  3. The paper does not establish a quality–latency advantage. Block size and denoising budget are central to diffusion-model efficiency. Comparisons at selected settings do not reveal whether the proposed model offers a better quality–latency trade-off than BDLM attention. Sweeps over these settings, evaluated with trained models, would substantially strengthen the central claim.
Questions for authors
  1. Can the authors report quality–latency curves for trained models, varying block size and denoising budget for both BDLM attention and BDLM Mamba-H?
  2. Do trained models retain useful long-context behavior beyond the 8K training and validation setting? Long-range retrieval or long-context language-modeling evaluations would help assess this.
  3. Applicability of Complete(d)P. What justifies applying the Complete(d)P hidden-weight learning-rate transfer rule to this partially bidirectional Mamba–attention architecture? Please clarify how Mamba-specific parameters are treated and whether the implementation adopts the accompanying initialization, residual, and optimizer scaling rules, rather than only rescaling the global learning rate. Have you performed coordinate checks or learning-rate sweeps across model scales to verify that the proposed transfer remains effective under the block-diffusion objective?
2 comments

Response to Reviewer f7wJ (Part 1/2)

Thank you for the careful review and for noting both the cache-aligned formulation and our systems measurements. Below we address each concern and question, along with the clarifications we have made in the revised draft.

Isolating block-local reverse scans and prefix-state gradients (W1). Rather than acting as independent add-ons, both choices follow directly from instantiating block discrete diffusion (BD3-LM, Arriola et al., ICLR 2025) on a Mamba-attention hybrid. On scan directionality, our goal is to make bidirectional Mamba diffusion (DiffuMamba, Singh et al., ICML 2026) prefix-cacheable without turning the SSM stack into a causal denoiser. Under absorbing discrete diffusion (MDLM, Sahoo et al., NeurIPS 2024; BD3-LM), active-block tokens are masked uniformly at random and unmasked in arbitrary order, so early masked positions within a block routinely depend on unmasked tokens to their right. Because Mamba-2 accounts for 10 of 12 layers at 87M and 15 of 18 layers at 350M (Tables 7–8), making Mamba forward-only removes intra-block right context across nearly the entire network depth. Prior work shows that pairing uniform diffusion corruption with unidirectional mixers hurts reconstruction quality unless bidirectionality is retained or the corruption schedule itself is altered (e.g., DiffuLLaMA, Gong et al., ICLR 2025; Hwang et al., NeurIPS 2024; Oshima et al., 2024; and CARD, Ruan et al., 2026, Table 4, where random masking inside a causal model drops 8-task average accuracy from 53.21% to 51.62% and CSQA from 48.73% to 46.03%). As a caveat, those prior directionality ablations compare unidirectional mixers against full-sequence bidirectional baselines (or pure causal models), whereas BDLM Mamba-H is only partially bidirectional across the sequence, remaining causal over completed prefixes and fully bidirectional only within the active 256-token block alongside sparse attention layers. Restricting the reverse scan to the active block is the minimal change that preserves bidirectional intra-block Mamba mixing while keeping completed prefixes causal, and we have updated Sections 2 and 3 of the draft to include this context and distinction.

Similarly, we have updated Section 4.1 so that propagating gradients through cached prefix states does not read like a separate training heuristic. In standard attention BD3-LM (Sec. 3.2), clean prefix K/V projections are never detached, and active-block denoising losses backpropagate directly into clean prefix representations via the chain rule on ∑kL(b⋆(k),b⋆(<k);θ)\sum_k \mathcal{L}(b^{\star(k)}, b^{\star(<k)}; \theta) (Eq. 4). Our all-block objective preserves that same gradient path when the prefix summary is a recurrent Mamba boundary state skℓs_k^\ell instead of a K/V tensor. Because clean prefix blocks carry no standalone next-token loss in block diffusion, applying stop_gradient at block boundaries would truncate BPTT every 256 tokens and leave Mamba's state-retention parameters (A,Δ,BA, \Delta, B) without gradient signal for carrying information across blocks.

Connecting 87M/350M quality runs and 700M inference benchmarks (W2). Our native BDLM sampler (Appendix B, Table 9) executes a fixed decoding schedule (B=256B = 256, p=16p = 16). Every block launches the exact same sequence of FlashAttention, Mamba-2, and GEMM kernels on identical tensor shapes regardless of parameter values, meaning wall-clock throughput and HBM traffic are identical for random and trained weights at a given geometry. Following the long-context throughput protocol of DiffuMamba (which profiles ~1.3B–1.4B models on H100), we ran single-GPU inference at 700M to evaluate a model larger than 350M that better saturates an A100-80GB GPU and gives a clearer comparison across architectures up to 262K tokens, as we have clarified in Section 5.3. At 350M, where we evaluate trained checkpoints head-to-head, BDLM Mamba-H matches BDLM attention in validation quality (38.24 vs. 36.97 C4 PPL, 1.15 vs. 1.14 BPB, and 43.29% vs. 41.28% MCQA; Table 3) while already running 9.8% faster in 8K training with lower peak memory (Table 4).

Response to Reviewer f7wJ (Part 2/2)

Quality–latency trade-offs across block size and denoising budget (W3, Q1) and behavior beyond 8K (Q2). Because each block size BB defines a distinct BDLM training objective (Eq. 3), our pretraining fixed B=256B = 256 (matching BD3-LM) rather than retraining across multiple block sizes. At any matched (B,p)(B, p), however, varying the denoising budget pp scales per-block pass counts (p+1p + 1) identically for both architectures, rescaling throughput roughly as 1/p1/p without altering the per-step memory-traffic ratio, since BDLM attention streams growing K/V caches in every layer versus only 1/6 of layers (layers 0, 6, and 12 at 350M) alongside fixed-size recurrent states in the Mamba-2 layers. At B=256B = 256, BDLM Mamba-H outperforms BDLM attention across all five 87M learning rates (best 61.59 vs. 76.46 C4 PPL; Tables 1, 5) and matches 350M quality while cutting K/V cache state by 6×6\times, which drives the 1.9×1.9\times to 4.0×4.0\times speedup once prefix K/V bandwidth dominates past 16K–32K tokens (Table 6). Beyond 8K (Q2), although both models instantiate a 262K RoPE table (θ=10,000\theta = 10{,}000; Table 9) on sparse attention layers alongside position-free Mamba-2 updates, our 87M and 350M checkpoints were trained on packed 8,192-token sequences without continual long-context extension such as RoPE rescaling or >8K fine-tuning. Accordingly, the 32K–262K ladder in Section 5.3 specifically measures serving and cache-bandwidth scaling once operating at long context (278 vs. 74 tok/s over BDLM attention at 262K, and 35 tok/s for full-sequence at 65K; Table 6), as we have clarified in Sections 5.3 and 6.

Justification and implementation of Complete(d)P transfer (Q3). Recent scaling results on both discrete diffusion and hybrid SSMs support using AR-derived hidden-weight LR transfer as a practical scale-up rule here. On the objective side, Quokka (Ni et al., NeurIPS 2026) finds that AR learning-rate and batch-size scaling rules transfer directly to masked DLMs, von Rütte et al. (ICLR 2026) validate AR CompleteP transfer across width and depth in discrete DLMs up to 10B parameters, and Zheng et al. (NeurIPS 2025) prove that diffusion Transformer μ\muP aligns with vanilla Transformer μ\muP. BDLM sits closer still to AR optimization because its outer likelihood is autoregressive over blocks (BD3-LM, Eq. 2), so at B=256B = 256 on 8,192-token sequences, completed prefix blocks run causally while only the active 256-token block undergoes masked cross-entropy denoising. On the architectural side, Seo et al. (NeurIPS 2026) show on Transformer-Mamba2 hybrids that standard hidden-weight LR scaling yields a near-zero LR transfer gap across widths (256–2048), depths (4–32), batch sizes, and attention-to-SSM ratios, even though practical fixed-state Mamba layers fall outside Tensor Programs assumptions and fail formal coordinate checks, because global 1/width1/\text{width} LR scaling preserves update-to-weight ratios while AdamW's coordinate-wise normalization balances the SSM parameters (A,Δ,B,CA, \Delta, B, C) locally (see also Haas et al., NeurIPS 2025 on feature learning under standard init with global width-scaled LRs and cross-entropy loss). Summing our forward and block-local reverse scans (y→+y←y^{\rightarrow} + y^{\leftarrow}, Eq. 11) adds only a scale-free O(1)O(1) factor independent of width, depth, batch, and token horizon, leaving those scaling exponents unchanged.

In our implementation, we rescale only the global learning rate via Eq. 8 (Table 2) while keeping a single AdamW parameter group (betas=(0.9, 0.95), eps=1e-8, weight_decay=0.10) and standard initialization with zero-initialized output and AdaLN projections across all parameters, including Mamba-2 projections and SSM parameters, rather than applying per-module residual or optimizer rescaling rules or multi-scale coordinate checks. Both 350M runs trained smoothly without instability and preserved the 87M quality parity (Table 3), and we have added this justification and specification to Sections 3, 5.1, and Appendix C.

Comments on the paper

2 comments

Response to Reviewer pspA (Part 1/2)

Thank you for the positive assessment and for highlighting the prefix-cache dependency fix, cache-aligned training path, 350M scale-up, and systems reporting. Below we address each point and note the clarifications we have made in the revised draft.

Trained-checkpoint quality at 8K vs. 700M 262K speed, and scope beyond 8K (W1, Q1). Our native BDLM sampler (Appendix B, Table 9) executes a fixed schedule (B=256,p=16B = 256, p = 16) in which every block launches the exact same FlashAttention, Mamba-2, and GEMM kernels on identical shapes regardless of parameter values, so tokens/s and HBM traffic are identical for random and trained checkpoints at a given geometry. Following DiffuMamba (Singh et al., ICML 2026) (which profiles ~1.3B–1.4B models on H100), we benchmarked single-GPU inference at 700M to evaluate a model larger than 350M that better saturates an A100-80GB across architectures up to 262K tokens. At 350M, where we evaluate trained checkpoints directly under the same B=256B = 256 block formulation, BDLM Mamba-H matches BDLM attention quality (38.24 vs. 36.97 C4 PPL, 1.15 vs. 1.14 C4 BPB; 36.81 vs. 36.14 Paloma PPL, tied 1.13 BPB; and 43.29% vs. 41.28% MCQA; Table 3) while running 9.8% faster in 8K training with lower peak memory (Table 4). Beyond 8K, although both models instantiate a 262K RoPE table (θ=10,000\theta = 10{,}000; Table 9) alongside position-free Mamba-2 state updates, our 87M and 350M checkpoints were trained on packed 8,192-token sequences without continual long-context extension such as RoPE rescaling or >8K fine-tuning. Accordingly, the >8K ladder in Section 5.3 measures cache-bandwidth scaling once operating at long context, as we have clarified in Sections 5.3 and 6.

Timestep-conditioned Attention BDLM comparison in Table 10 (W2, Q2 part 1). We agree that Table 10 deserves direct discussion in Section 5.2 and have added it there alongside Appendix D. Even against timestep-conditioned BDLM attention at learning rate 4×10−34\times 10^{-3} (64.13 C4 PPL), no-timestep BDLM Mamba-H remains lower in C4 PPL both at the matched 4×10−34\times 10^{-3} learning rate (62.08) and at its sweep-selected 8×10−38\times 10^{-3} setting (61.59; Tables 1, 10), while timestep-conditioned BDLM attention achieves higher 87M MCQA on that fixed-LR slice (44.85% vs. 40.70%). Following BD3-LM (Arriola et al., ICLR 2025) and MDLM (Sahoo et al., NeurIPS 2024), our main sweep omitted explicit timestep conditioning because under absorbing masking the visible mask fraction already encodes noise level tt, and standard DDiT AdaLN modulates prefix tokens with ete_t, which breaks prefix-cache reuse across reverse steps unless refactored through our active-block-only AdaLN (Appendix E) while adding per-layer AdaLN parameters and compute. At 350M, our main claim is competitive quality parity (within 0.01 C4 BPB, tied 1.13 Paloma BPB, and +2.01 MCQA points; Table 3) paired with 6×6\times smaller K/V cache state (3/183/18 attention layers).

Response to Reviewer pspA (Part 2/2)

Forward-only Mamba and detached prefix-cache controls (W3, Q2 part 2). Both design choices follow directly from BD3-LM rather than acting as separable heuristics. On scan directionality, uniform intra-block masking means early masked positions routinely rely on unmasked right-context tokens within the same block, and because Mamba-2 forms 10 of 12 layers at 87M and 15 of 18 layers at 350M (Tables 7–8), forward-only Mamba removes intra-block right context across nearly the full network depth. Prior studies show that pairing uniform diffusion masking with unidirectional mixers hurts reconstruction quality unless bidirectionality is preserved or the masking schedule itself is altered (DiffuLLaMA, Gong et al., ICLR 2025; Hwang et al., NeurIPS 2024; Oshima et al., 2024; CARD, Ruan et al., 2026, Table 4), with the caveat that those prior ablations benchmark against full-sequence bidirectional baselines (or pure causal models) whereas BDLM Mamba-H is partially bidirectional (causal on completed prefixes and bidirectional only within the active block, alongside sparse attention). We have updated Sections 2 and 3 of the draft to add this context. Likewise, in standard attention BD3-LM (Sec. 3.2), clean prefix K/V projections are never detached, and active-block losses backpropagate through clean prefixes via the chain rule on ∑kL(b⋆(k),b⋆(<k);θ)\sum_k \mathcal{L}(b^{\star(k)}, b^{\star(<k)}; \theta) (Eq. 4). Detaching Mamba's recurrent boundary state skℓs_k^\ell (stop_gradient) would break parity with attention BDLM by truncating BPTT every 256 tokens, leaving Mamba's state-retention parameters (A,Δ,BA, \Delta, B) without gradient signal to carry information across blocks because clean prefix blocks carry no standalone next-token loss, as we have clarified in Section 4.1.

Validation sample count and uncertainty estimates (W4). Although each 8,192-token validation sequence uses one Monte Carlo corruption draw (t∼Uniform(10−3,1)t \sim \mathrm{Uniform}(10^{-3}, 1)), the all-block evaluator scores all 32 blocks per sequence, so 256 packed sequences at 350M cover 2.10M evaluated tokens per corpus (256×8,192=2,097,152256 \times 8{,}192 = 2{,}097{,}152 tokens each on C4-en and Paloma-C4) with deterministic seeds shared across models. Across those >2M-token evaluations, model rankings agree between C4-en and Paloma-C4 (Table 3) and vary smoothly across the five learning rates in our 20-run 87M sweep (Table 5). We treat the 0.01 BPB difference at 350M as practical parity rather than a meaningful gap, and we have updated Section 5.1 and Table 3 to state the 2.10M-token evaluation budget per corpus.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.