acceptodds
Under review as a conference paper at ICLR 2027

Block Parallelism for Efficient Distributed Long-Context Diffusion Language Model Training

Abstract

Block diffusion language models (BDLMs) combine autoregressive dependencies across blocks with parallel denoising within blocks, but long-context training is constrained by distributed attention communication and activation memory. Conventional context parallelism (CP) shards the combined clean-plus-corrupted sequence by position, communicating shared clean K/Vs together with block-specific corrupted K/Vs and their gradients. Our key observation is that the BDLM objective separates over target blocks and that diffusion-based speculative drafter losses have the same decomposition. We introduce block parallelism (BP), a new distributed parallelism dimension that assigns each corrupted-block computation to one rank. To scale BP to long contexts, we introduce context-sharded block parallelism (CSBP), which also shards the shared clean sequence across those ranks. CSBP keeps corrupted K/Vs and their gradients local, avoids replicated clean prefixes, and preserves BDLM training semantics. Applied to speculative drafters, it also avoids replicated block-local computation. On 16 H200 GPUs at 256K context, CSBP improves throughput over the best baseline by \textbf{1.18--1.45\times} for supervised fine-tuning on 3B–26B-A4B models and \textbf{1.27--1.33\times} for converting 27B autoregressive models to BDLMs, while matching or reducing peak HBM. For DiffusionGemma 26B-A4B, the fine-tuning speedup over the best baseline grows with context length, reaching \textbf{1.61\times} at 512K. On eight H100 GPUs, CSBP accelerates Qwen3.8-27B DFlash2 drafter training by \textbf{2.48\times} at 512K and \textbf{7.59\times} at 1M. In 12-hour DiffusionGemma 26B-A4B SFT, CSBP achieves higher pass rates at every trained checkpoint on SWE-bench Verified and Terminal-Bench Lite.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Diffusion LLMs, Long Context, Distributed Training

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_z3MBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies distributed training for block diffusion language models and diffusion-based speculative decoding drafters with long contexts. The authors exploit the independence of corrupted-block computations across target blocks and the reuse of shared clean context to introduce Block Parallelism (BP). They further propose Context-Sharded Block Parallelism (CSBP), which shards the shared clean context across the same group of ranks. This keeps corrupted-block K/V tensors and their gradients local while avoiding replicated clean prefixes. The paper provides a proof of computational and gradient equivalence and evaluates efficiency in supervised fine-tuning, autoregressive-to-block-diffusion model conversion, and DFlash2 drafter training. At a context length of 256K, the first two workloads achieve throughput improvements of 1.18–1.45× and 1.27–1.33×, respectively. Drafter training achieves a maximum speedup of 7.59× under a specific 1M-context configuration.

Critical strengths
  1. The method closely follows the dependency structure of the training objective and handles gradient dependencies through the shared context. Beyond assigning different target blocks to different GPUs, the method shards the context to avoid duplicated clean prefixes and aggregates gradient contributions from different target blocks to the clean K/V tensors. Section 4, Appendix A, and Appendix I provide supporting evidence for the mechanism, correctness, and memory benefits, respectively. This improves long-context training efficiency while preserving the original training objective, making it a practically valuable contribution.
  2. The efficiency evaluation covers a broad range of settings, and the ablations and profiling results help explain the gains. The experiments cover multiple models and three training workloads. Appendices I, J, and L examine the effects of context sharding, load balancing, and block size, respectively. Appendix M separately reports communication volume, exposed communication time, and end-to-end step time, helping explain the sources of speedup. The paper also reports settings in which CSBP is less competitive. For example, in Table 2, the DFlash2 drafter for Qwen3.8-27B at a context length of 256K achieves 0.84× the throughput of the purely data-parallel baseline. This helps clarify the method's applicability.
Critical weaknesses
  1. The quantitative breakdown of the largest drafter-training speedup remains incomplete. Section 5.3 and Appendix E explain that the 7.59× speedup at 1M comes from reduced redundant computation and the parallel configuration changes this enables. I recognize the value of this overall system gain. However, the existing results still make it difficult to quantify the gain under a fixed DP/CP configuration and the individual contributions of the different optimizations. Additional profiling details or results under corresponding configurations would help assess how well these gains transfer to other training settings.
Questions for authors
  1. In the 1M DFlash2 setting, how do throughput and peak GPU memory usage change when BP is added while keeping the DP and CP degrees fixed? Can the existing profiling results distinguish the contributions of reduced redundant computation, reduced communication, and increased data parallelism?
  2. Appendix M explains how complete-block ownership and aligned sharding improve sliding-window locality. Could the authors further clarify the communication scheme used by the CP baseline and whether they tried exchanging only the K/V rows required by the window under its original token-sharding scheme? This would help distinguish the gains from data placement from those of the specific communication implementation.
Reviewer_nZubOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Block Parallelism (BP) and Context-Sharded Block Parallelism (CSBP) for efficient long-context block diffusion language model training. Its central insight is that corrupted-block computations are independent across target blocks, whereas clean-prefix representations are shared. CSBP exploits this asymmetry by assigning complete corrupted-block computations to individual GPU ranks while distributing reusable clean context across ranks. The authors establish equivalence to the original training objective and gradients, introduce workload-balanced scheduling, and evaluate BDLM fine-tuning, autoregressive-to-diffusion conversion, and speculative-drafter training. Reported throughput improvements reach 1.61× for BDLM fine-tuning and 7.59× for drafter training, alongside improved fixed-time downstream performance.

Critical strengths
  • Principled, objective-aware parallelization. CSBP separates block-local corrupted computations from shared clean-prefix representations, eliminating unnecessary corrupted K/V communication.
  • Convincing systems-level analysis and ablations. Experiments demonstrate throughput improvements across multiple models and workloads.
  • Practical relevance across training objectives. The method supports BDLM fine-tuning, AR-to-BDLM conversion, and DFlash2 drafter training without changing their objectives.
Critical weaknesses
  • Presentation and writing quality: The paper has several issues in organizing and communicating its central contributions and empirical findings:
    • Insufficiently self-contained main text: Essential quantitative evidence supporting the paper's motivation, design choices, and performance improvements is largely relegated to the appendix (Appendices I, J, and M), while several figures in the main text occupy substantial space.
    • Insufficient transparency in baseline selection: Section 5.1 states that the baseline is the "fastest feasible tested non-BP configuration," while Appendix F lists the parallelism dimensions considered. However, the actual configuration search space and tuning results are not reported.
    • Insufficient discussion of unfavorable results: Figure 1 prominently highlights the 2.48× and 7.59× drafter-training speedups but omits the 256K setting where CSBP is slower than DP8 (0.84×, Table 2). Although Section 5.3 briefly explains this result, the conditions under which CSBP becomes beneficial remain insufficiently characterized.
    • Underdeveloped limitations: Section 6 contains only a single sentence distinguishing training from inference. It does not discuss the block-independence condition required by Proposition 1 (Appendix A), untested block sizes below the 128–1024 range (Table 7), or the performance boundary illustrated by the 0.84× result (Table 2).
    • Unclear novelty positioning: The block-separable objective and reusable clean-prefix computation were established by Block Diffusion [1]. Sections 4.2–4.3 should more clearly distinguish these existing properties from CSBP's genuinely new distributed ownership strategy.
  • The method's applicability to diverse BDLM formulations remains insufficiently demonstrated. SDAR uses much smaller block sizes than those evaluated, while LLaDA 2.1 introduces token-editing objectives whose compatibility with CSBP's block-independence assumption remains unverified.
  • Section 4.4 primarily attributes efficiency gains to eliminating corrupted K/V communication. However, Table 5 shows substantial gains from load balancing, while Table 8 shows that DiffusionGemma's 93.5% communication reduction also benefits from sliding-window attention. These contributions are not sufficiently disentangled.
  • Although the authors claim to open-source their training infrastructure, the paper does not provide an identifiable code repository or complete executable implementation, making it difficult to independently verify the the completeness of the system implementation.
Questions for authors
  • Could the authors compare CSBP against a mask-aware CP baseline that exchanges only the K/V shards required by each block, particularly for DiffusionGemma's sliding-window layers (Appendix M)?
  • Does Proposition 1 extend to models with cross-token operations beyond attention, such as capacity-constrained MoE routing or globally coupled auxiliary losses? What additional assumptions are required for exact gradient equivalence (Appendix A)?
  • At 1M context, could the authors quantitatively separate the 7.59× DFlash2 throughput gain into reduced duplicated computation, communication savings, and additional data-parallel replicas (Table 2; Appendix E)?
Reviewer_uBjHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Block diffusion models are trained on two copies of each text, which are the real version and a partly masked one. That doubles the length, so long inputs get split across GPUs and the GPUs share everything with each other. The authors observe that these masked chunks are only used by themselves, so sharing them is wasted effort. This paper keeps each masked chunk on one GPU and shares only the real text. As a result the training gets about 1.2 to 1.6 times faster with the same results and drafter training gets about 2.5 times faster.

Critical strengths

By keeping each masked chunk on one GPU and sharing only the real text, the method cuts wasted communication between GPUs. Training gets about 1.2 to 1.6 times faster on long inputs.

The authors prove CSBP gives the same loss and gradients as normal training, apart from rounding. They report matching loss values for six models. Though the empirical check is light, the math is consistent, so people can adopt it without changing their training objective.

The authors did a broad evaluation. They evaluated Seven models from 3B to 30B, two GPU types and contexts from 64K to 1M tokens.

Critical weaknesses

CP already places corrupted tokens on the same rank as their queries. The waste comes from CP sending all K/V regardless of the mask. A CP variant that keeps blocks whole on one rank and gathers only clean K/V would look much like CSBP. Without that comparison, it is unclear how much of the gain needs a new method rather than a better CP layout.

BP alone is never used in the main results: it is slower than the baseline or runs out of memory. Every reported result uses CSBP, where the BP degree always equals the CP degree on the same GPUs. In practice BP acts as a placement rule inside CP, not as an independent axis.

The 1.45–1.61times results on DiffusionGemma rest on a 93.5% cut in attention traffic, mostly from window layers and the baseline was not given window-aware exchange. Global-attention models gain only 1.10–1.21times.

The paper shows training gets faster but barely checks the quality of the trained models. Converted models and drafters are never evaluated, and there are no long training curves or same-step comparisons. The one quality test, a 12-hour coding benchmark run, only shows that more data helps. Its gaps come from a single run with no error bars.

Questions for authors

How much of the speedup would a simpler CP fix get, one that keeps corrupted blocks whole on one GPU and shares only clean K/V?

Can you show the faster training doesn't hurt model quality, with long loss curves, conversion and drafter quality, and multi-seed benchmark results?

Reviewer_pzEzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes block parallelism (BP) for distributed long-context training of block diffusion language models (BDLMs). The key observation is that the BDLM objective decomposes over target blocks, allowing each corrupted block to remain on a single rank without communicating its K/V. The paper then extends BP to long contexts with context-sharded block parallelism (CSBP), which additionally shards the shared clean sequence across ranks while preserving exact loss and gradients. At 256K context, the method achieves 1.18–1.45× faster SFT and 1.27–1.33× faster AR-to-BDLM conversion, with larger gains for speculative-drafter training.

Critical strengths
  1. The method is based on a simple and precise observation: corrupted K/V are only used within their own block. It preserves the BDLM objective exactly, with matching losses and gradients up to floating-point reduction order (Proposition 1, Appendix G), making it a low-risk drop-in replacement for existing training pipelines.

  2. The gains are consistent across six models and three workloads, including BDLM fine-tuning, AR-to-BDLM conversion, and speculative-drafter training. At 256K context, CSBP is faster in every setting while matching or reducing peak HBM usage. The public code further supports its practical value for long-context BDLM training.

Critical weaknesses

If I understand correctly, all reported results use CSBP, where BP and CP run on the same ranks with equal degrees (e.g., CP8/BP8). If so, BP does not introduce a new orthogonal parallelism dimension like DP, TP, or EP. Instead, CSBP appears to use a block-aligned layout within the CP group to avoid communicating corrupted K/V that are masked out. However, the baseline is conventional CP, which exchanges all K/V regardless of the attention mask. A mask-aware CP baseline would make it clearer whether the improvement comes from the proposed parallelism design or simply from avoiding unnecessary communication.

Questions for authors
  1. Section 5.1 measures throughput and memory using sequences with exactly the reported context length. How do block assignment, load balancing, and speedup behave for variable-length inputs, such as the SFT data used in Section 5.6?

  2. Appendix L notes that CSBP uses window-aware exchange in DiffusionGemma’s sliding-window layers, while the CP8 baseline exchanges full shards in every layer. Could the authors report the speedup when CP8 also uses window-aware exchange? Similarly, for DFlash2, how much of the gain comes from removing replicated drafter computation and increasing the DP degree, versus keeping corrupted K/V local?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.