- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PulseCol replaces coarse block-level sparsity with a finer-grained column-sparse structure for accelerating diffusion language models. It identifies sparse patterns at the early denoising step and reuses them across subsequent iterations, refreshing them only at a small number of intermediate steps. Experiments show greater practical speedup than prior sparse attention methods while maintaining model quality.
Critical strengths
- The intuition behind column sparsity and periodic refresh is clear.
- The experiments evaluate both generation quality and end-to-end inference speed using a dedicated GPU kernel.
Critical weaknesses
- SparseD focuses on 50% and 70% sparsity, whereas the main accuracy comparisons in this paper use 50% and 80%. It remains unclear how the two methods compare at 70% sparsity when both accuracy and latency are evaluated under the same long-context setting.
- The paper lacks comparisons with closely related sparse attention methods such as BA-Att [1] and Focus-dLLM [2]. It remains unclear whether PulseCol offers a better accuracy–latency trade-off than these methods under matched settings.
[1] Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention (CVPR 2026 Findings)
[2] Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing (ACL 2026)
Questions for authors
Please see the points raised in the Weaknesses section.
4 comments
W2: Lack of accuracy-latency comparisons with BA-Att and Focus-dLLM.
We thank the reviewer for this suggestion. We have extended our experiments to include BA-Att [1] and Focus-dLLM [2]. On LLaDA-1.5, PulseCol achieves higher average accuracy than both methods under their official configurations and at a nominally matched 80% sparsity level, as summarized below.
| Method | sparsity | GSM8K | MATH-500 | HE | RULER-4K | Avg. |
|---|---|---|---|---|---|---|
| Focus-dLLM | 50% | 77.33 | 40.20 | 35.37 | 83.91 | 59.20 |
| BA-Att | 50% | 73.77 | 41.40 | 34.76 | 85.91 | 58.96 |
| PulseCol (Ours) | 50% | 81.05 | 41.80 | 38.41 | 90.36 | 62.91 |
| Focus-dLLM | 80% | 75.36 | 40.00 | 35.98 | 75.48 | 56.70 |
| BA-Att | 80% | 32.75 | 28.20 | 9.76 | 47.03 | 29.43 |
| PulseCol (Ours) | 80% | 76.19 | 38.40 | 38.41 | 91.72 | 61.18 |
Note that Focus-dLLM only sparsifies prompt KV while retaining response KV, resulting in a lower effective sparsity than the nominal setting.
Latency comparison. We compare PulseCol with BA-Att at its official 50% sparsity setting. PulseCol achieves both higher accuracy and lower end-to-end latency, reaching 1.95× speedup over dense attention and 1.37× over BA-Att at 1024 denoising steps.
| Method | 128 steps | 256 steps | 512 steps | 1024 steps |
|---|---|---|---|---|
| Dense (FlashAttention) | 1,462.88 () | 2,932.34 () | 5,908.74 () | 11,930.43 () |
| BA-Att | 1,024.92 () | 2,062.33 () | 4,147.84 () | 8,390.22 () |
| PulseCol (Ours) | 925.63 () | 1,656.04 () | 3,138.28 () | 6,106.12 () |
For Focus-dLLM, a direct comparison of sparse attention efficiency is difficult, as its end-to-end acceleration also involves KV caching and query sparsity, making it difficult to isolate the contribution of attention sparsification.
More broadly, PulseCol is complementary to other inference optimizations. We also evaluate its combination with parallel decoding and Fast-dLLM, achieving further speedups as shown below.
| Method | HumanEval Speedup | GSM8K Speedup |
|---|---|---|
| PulseCol | 1.0x | 1.0x |
| + Threshold decoding | 2.67x | 3.06x |
| + Entropy decoding | 1.98x | 2.45x |
| + Fast-dLLM [3] | 3.95x | 6.44x |
Since BA-Att has not released its code, we reproduced it following the original paper and verified that our results are consistent with the reported performance. We have made both the PulseCol implementation and BA-Att reproduction available through anonymized repositories to facilitate reproducibility.
We have included these results and clarifications in the revised manuscript.
References:
[1] Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention, CVPR 2026 Findings.
[2] Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing, ACL 2026.
[3] Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding, ICLR 2026.
Thank you for the additional experiments. The new comparisons strengthen the evaluation, and I will raise my rating.
Dear Reviewer za7p,
Thank you very much for your thoughtful reconsideration and for raising your score.
We will make sure to incorporate the key clarifications and experiment from this discussion into the revised manuscript.
Best regards,
Authors
W1: Missing aligned accuracy-latency comparison with SparseD at 70% sparsity.
We thank the reviewer for this comment. We would like to clarify that in the original SparseD paper, 70% sparsity is applied only to RULER, whereas all other benchmarks are evaluated at 50% sparsity. In contrast, we evaluate both 50% and 80% sparsity across all tasks to provide a consistent evaluation and explore higher sparsity levels, where PulseCol still maintains accuracy comparable to dense attention.
For a direct comparison, we conducted an additional comparison with LLaDA-1.5 on RULER at 70% sparsity under the exact same long-context setting, evaluating both accuracy and latency.
The average is computed over all 13 RULER subtasks.
Each entry reports end-to-end inference latency (s), with speedup over dense attention in parentheses.
As shown above, at 70% sparsity on RULER, both SparseD and PulseCol already achieve near-dense accuracy, while PulseCol provides lower end-to-end latency. We have included these aligned 70% RULER results in the revised manuscript.