- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies distributed training for block diffusion language models and diffusion-based speculative decoding drafters with long contexts. The authors exploit the independence of corrupted-block computations across target blocks and the reuse of shared clean context to introduce Block Parallelism (BP). They further propose Context-Sharded Block Parallelism (CSBP), which shards the shared clean context across the same group of ranks. This keeps corrupted-block K/V tensors and their gradients local while avoiding replicated clean prefixes. The paper provides a proof of computational and gradient equivalence and evaluates efficiency in supervised fine-tuning, autoregressive-to-block-diffusion model conversion, and DFlash2 drafter training. At a context length of 256K, the first two workloads achieve throughput improvements of 1.18–1.45× and 1.27–1.33×, respectively. Drafter training achieves a maximum speedup of 7.59× under a specific 1M-context configuration.
Critical strengths
- The method closely follows the dependency structure of the training objective and handles gradient dependencies through the shared context. Beyond assigning different target blocks to different GPUs, the method shards the context to avoid duplicated clean prefixes and aggregates gradient contributions from different target blocks to the clean K/V tensors. Section 4, Appendix A, and Appendix I provide supporting evidence for the mechanism, correctness, and memory benefits, respectively. This improves long-context training efficiency while preserving the original training objective, making it a practically valuable contribution.
- The efficiency evaluation covers a broad range of settings, and the ablations and profiling results help explain the gains. The experiments cover multiple models and three training workloads. Appendices I, J, and L examine the effects of context sharding, load balancing, and block size, respectively. Appendix M separately reports communication volume, exposed communication time, and end-to-end step time, helping explain the sources of speedup. The paper also reports settings in which CSBP is less competitive. For example, in Table 2, the DFlash2 drafter for Qwen3.8-27B at a context length of 256K achieves 0.84× the throughput of the purely data-parallel baseline. This helps clarify the method's applicability.
Critical weaknesses
- The quantitative breakdown of the largest drafter-training speedup remains incomplete. Section 5.3 and Appendix E explain that the 7.59× speedup at 1M comes from reduced redundant computation and the parallel configuration changes this enables. I recognize the value of this overall system gain. However, the existing results still make it difficult to quantify the gain under a fixed DP/CP configuration and the individual contributions of the different optimizations. Additional profiling details or results under corresponding configurations would help assess how well these gains transfer to other training settings.
Questions for authors
- In the 1M DFlash2 setting, how do throughput and peak GPU memory usage change when BP is added while keeping the DP and CP degrees fixed? Can the existing profiling results distinguish the contributions of reduced redundant computation, reduced communication, and increased data parallelism?
- Appendix M explains how complete-block ownership and aligned sharding improve sliding-window locality. Could the authors further clarify the communication scheme used by the CP baseline and whether they tried exchanging only the K/V rows required by the window under its original token-sharding scheme? This would help distinguish the gains from data placement from those of the specific communication implementation.