acceptodds
Under review as a conference paper at ICLR 2027

PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Models

11 views

Abstract

Inference in diffusion large language models (dLLMs) is computationally expensive, as full self-attention must be repeatedly executed at each step of the denoising process without KV cache. Recent sparse attention methods for dLLMs mitigate this cost via block-sparse computation, which is applied only in later iterations when model performance is less sensitive to coarse-grained sparse approximation, but yields limited improvements in computational efficiency and acceleration. This motivates a finer-grained sparsification strategy that can be applied from earlier iterations and leverages reusable sparsity patterns, enabling further efficiency gains. In this work, we introduce PulseCol, a periodically refreshed column-sparse attention method for accelerating diffusion language models. PulseCol replaces coarse block-level sparsity with a finer-grained column-sparse structure, allowing important attention interactions to be retained more precisely while exposing greater sparsity. Built on this column-level formulation, PulseCol further identifies sparse patterns at the early denoising step and reuses them across subsequent iterations, refreshing them only at a small number of intermediate steps to track the evolution of sparse attention patterns during denoising. Experiments show that PulseCol achieves higher sparsity and greater practical speedup than prior sparse attention methods for dLLMs, while maintaining model quality. Enabled by optimized GPU kernels for column-sparse attention, PulseCol delivers up to 1.95 end-to-end speedup over FlashAttention across several context lengths.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, sparse attention, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_za7pOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PulseCol replaces coarse block-level sparsity with a finer-grained column-sparse structure for accelerating diffusion language models. It identifies sparse patterns at the early denoising step and reuses them across subsequent iterations, refreshing them only at a small number of intermediate steps. Experiments show greater practical speedup than prior sparse attention methods while maintaining model quality.

Critical strengths
  1. The intuition behind column sparsity and periodic refresh is clear.
  2. The experiments evaluate both generation quality and end-to-end inference speed using a dedicated GPU kernel.
Critical weaknesses
  1. SparseD focuses on 50% and 70% sparsity, whereas the main accuracy comparisons in this paper use 50% and 80%. It remains unclear how the two methods compare at 70% sparsity when both accuracy and latency are evaluated under the same long-context setting.
  2. The paper lacks comparisons with closely related sparse attention methods such as BA-Att [1] and Focus-dLLM [2]. It remains unclear whether PulseCol offers a better accuracy–latency trade-off than these methods under matched settings.

[1] Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention (CVPR 2026 Findings)
[2] Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing (ACL 2026)

Questions for authors

Please see the points raised in the Weaknesses section.

4 comments

W1: Missing aligned accuracy-latency comparison with SparseD at 70% sparsity.

We thank the reviewer for this comment. We would like to clarify that in the original SparseD paper, 70% sparsity is applied only to RULER, whereas all other benchmarks are evaluated at 50% sparsity. In contrast, we evaluate both 50% and 80% sparsity across all tasks to provide a consistent evaluation and explore higher sparsity levels, where PulseCol still maintains accuracy comparable to dense attention.

For a direct comparison, we conducted an additional comparison with LLaDA-1.5 on RULER at 70% sparsity under the exact same long-context setting, evaluating both accuracy and latency.

The average is computed over all 13 RULER subtasks.

MethodSingleMulti-keyMQMVVTSQuADHotpotFWECWEAverage
Dense100.00100.0099.9099.4599.9284.3378.2063.6053.5690.69
SparseD100.00100.0099.9598.50100.0090.0076.6064.0751.0490.78
PulseCol100.0099.60100.0099.8099.6488.6078.2060.5348.8690.34

Each entry reports end-to-end inference latency (s), with speedup\textcolor{purple}{speedup} over dense attention in parentheses.

Method128 steps256 steps512 steps1024 steps
Dense (FlashAttention)1,462.88 (1.00×\textcolor{purple}{1.00\times})2,932.34 (1.00×\textcolor{purple}{1.00\times})5,908.74 (1.00×\textcolor{purple}{1.00\times})11,930.43 (1.00×\textcolor{purple}{1.00\times})
SparseD (70%)1,143.66 (1.28×\textcolor{purple}{1.28\times})2,101.23 (1.40×\textcolor{purple}{1.40\times})4,046.84 (1.46×\textcolor{purple}{1.46\times})7,983.49 (1.49×\textcolor{purple}{1.49\times})
PulseCol (70%)975.24 (1.50×\textcolor{purple}{1.50\times})1,787.15 (1.64×\textcolor{purple}{1.64\times})3,387.57 (1.74×\textcolor{purple}{1.74\times})6,757.32 (1.77×\textcolor{purple}{1.77\times})

As shown above, at 70% sparsity on RULER, both SparseD and PulseCol already achieve near-dense accuracy, while PulseCol provides lower end-to-end latency. We have included these aligned 70% RULER results in the revised manuscript.

W2: Lack of accuracy-latency comparisons with BA-Att and Focus-dLLM.

We thank the reviewer for this suggestion. We have extended our experiments to include BA-Att [1] and Focus-dLLM [2]. On LLaDA-1.5, PulseCol achieves higher average accuracy than both methods under their official configurations and at a nominally matched 80% sparsity level, as summarized below.

MethodsparsityGSM8KMATH-500HERULER-4KAvg.
Focus-dLLM50%77.3340.2035.3783.9159.20
BA-Att50%73.7741.4034.7685.9158.96
PulseCol (Ours)50%81.0541.8038.4190.3662.91
Focus-dLLM80%75.3640.0035.9875.4856.70
BA-Att80%32.7528.209.7647.0329.43
PulseCol (Ours)80%76.1938.4038.4191.7261.18

Note that Focus-dLLM only sparsifies prompt KV while retaining response KV, resulting in a lower effective sparsity than the nominal setting.

Latency comparison. We compare PulseCol with BA-Att at its official 50% sparsity setting. PulseCol achieves both higher accuracy and lower end-to-end latency, reaching 1.95× speedup over dense attention and 1.37× over BA-Att at 1024 denoising steps.

Method128 steps256 steps512 steps1024 steps
Dense (FlashAttention)1,462.88 (1.00×\textcolor{purple}{1.00\times})2,932.34 (1.00×\textcolor{purple}{1.00\times})5,908.74 (1.00×\textcolor{purple}{1.00\times})11,930.43 (1.00×\textcolor{purple}{1.00\times})
BA-Att1,024.92 (1.43×\textcolor{purple}{1.43\times})2,062.33 (1.42×\textcolor{purple}{1.42\times})4,147.84 (1.42×\textcolor{purple}{1.42\times})8,390.22 (1.42×\textcolor{purple}{1.42\times})
PulseCol (Ours)925.63 (1.58×\textcolor{purple}{1.58\times})1,656.04 (1.77×\textcolor{purple}{1.77\times})3,138.28 (1.88×\textcolor{purple}{1.88\times})6,106.12 (1.95×\textcolor{purple}{1.95\times})

For Focus-dLLM, a direct comparison of sparse attention efficiency is difficult, as its end-to-end acceleration also involves KV caching and query sparsity, making it difficult to isolate the contribution of attention sparsification.

More broadly, PulseCol is complementary to other inference optimizations. We also evaluate its combination with parallel decoding and Fast-dLLM, achieving further speedups as shown below.

MethodHumanEval SpeedupGSM8K Speedup
PulseCol1.0x1.0x
+ Threshold decoding2.67x3.06x
+ Entropy decoding1.98x2.45x
+ Fast-dLLM [3]3.95x6.44x

Since BA-Att has not released its code, we reproduced it following the original paper and verified that our results are consistent with the reported performance. We have made both the PulseCol implementation and BA-Att reproduction available through anonymized repositories to facilitate reproducibility.

We have included these results and clarifications in the revised manuscript.

References:

[1] Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention, CVPR 2026 Findings.

[2] Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing, ACL 2026.

[3] Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding, ICLR 2026.

Thank you for the additional experiments. The new comparisons strengthen the evaluation, and I will raise my rating.

Dear Reviewer za7p,

Thank you very much for your thoughtful reconsideration and for raising your score. 

We will make sure to incorporate the key clarifications and experiment from this discussion into the revised manuscript.

Best regards,

Authors

Reviewer_hfcBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PulseCol, a periodically refreshed column-sparse attention method for efficient diffusion language model (dLLM) inference. The key observation is that early denoising attention in dLLMs is not truly dense but exhibits a column-wise sparse pattern: high-scoring attention mass concentrates on a small subset of key columns. Existing block-sparse methods such as SparseD avoid sparsifying early denoising steps because block-level approximation is too coarse and can discard important key-wise connections. PulseCol instead partitions query positions into groups, selects top-k k key columns for each group based on aggregated attention scores, periodically refreshes the column indices during the early denoising window, and persistently reuses the last indices for the remaining steps. The authors also implement a column-sparse FlashAttention-style kernel that gathers only indexed KV columns and maintains online softmax statistics in SRAM. Experiments on LLaDA-1.5 and Dream across GSM8K, MATH-500, HumanEval, and RULER show competitive accuracy at high sparsity and practical speedups over FlashAttention and SparseD.

Critical strengths
  • The observation that attention in early denoising steps of dLLMs exhibits column-wise rather than block-wise sparsity is interesting and provides a plausible explanation for why prior block-sparse methods are fragile in early denoising steps.

  • The combination of initial group-wise column selection, low-frequency periodic refresh, and persistent reuse is straightforward to implement and directly addresses the cost of repeated pattern construction.

Critical weaknesses
  • Figure 6(a) shows accuracy peaks at group size 32 and is worst at 128 on both models, while Figure 5 fixes BM=128B_M=128 and calls it the “query group size.” Figure 10 shows BM=32B_M=32 and BM=64B_M=64 roughly halves the kernel speedup.

  • The column-sparsity evidence is largely empirical. The main evidence is representative heatmaps. Appendix A help, but there is no systematic quantification of how prevalent and stable the column pattern is across denoising steps, tasks, and models, nor a theoretical explanation for why it emerges.

  • Only LLaDA-1.5 and Dream are evaluated. Newer open dLLMs such as LLaDA-2 are untested.

Questions for authors
  • Can you give a deeper explanation of the column/vertical attention pattern?

  • Why does the refresh schedule behave differently across models in Table 7? Is a more general adaptive refresh policy possible?

  • Can you provide a latency breakdown? How much time goes to score collection, TopK/index construction, and the sparse attention kernel?

  • Using the same Top-K size for all layers and heads is basic. Have you tried top-P or adaptive budgets?

  • What is the end-to-end speedup at the accuracy-optimal group size of 32?

3 comments

W1&Q5: Missing end-to-end speedup at group size 32.

We thank the reviewer for highlighting this important accuracy-efficiency trade-off. A larger query group size BMB_M improves GPU utilization through more efficient tiled computation and reduced sparse-indexing overhead, but also reduces the granularity of sparse attention selection, potentially affecting accuracy. We use BM=128B_M=128 in the experiments primarily to match SparseD's default configuration for a fair comparison.

We have conducted additional end-to-end latency experiments on LLaDA-1.5 with a 64K context length, setting BM=32B_M=32 for both PulseCol and SparseD at 80% sparsity. The results are summarized below:

StepsDense (s)SparseD (s)PulseCol (s)Speedup
1281,421.711,348.431,245.611.14x
2562,849.062,481.952,149.621.33x
5125,754.224,784.183,978.451.45x
102411,722.909,399.877,666.981.53x

These results show that PulseCol still provides clear end-to-end speedups at the accuracy-optimal group size of 32. We have included these additional results in the revised manuscript to better illustrate the accuracy-efficiency trade-off.


W2&Q1: More quantitative analysis and explanation on column-wise sparsity.

Quantitative analysis: We thank the reviewer for this suggestion. We first compare the Top-K recall of column and block sparsity at the beginning of denoising across two models and three tasks, as shown below.

ModelSparsity PatternHumanEvalGSM8KRULERAverage
LLaDA 1.5Column90.51%91.27%90.67%90.82%
LLaDA 1.5Block59.91%64.22%60.73%61.62%
DreamColumn90.16%90.39%91.81%90.79%
DreamBlock75.30%60.41%85.50%73.74%

Column sparsity consistently achieves approximately 90% Top-K recall across both models and all three tasks, substantially higher than block sparsity. To further examine whether this pattern is limited to a few attention heads, we report the percentage of heads with Top-K recall exceeding 80%.

ModelHumanEvalGSM8KRULERAverage
LLaDA 1.592.58%94.04%86.23%90.95%
Dream87.76%91.45%88.78%89.33%

These results indicate that column sparsity is prevalent across attention heads, tasks, and models, rather than being limited to a few representative cases.

We further measure the Top-K recall at different stages of the denoising process, as shown below.

ModelTask0%≈25%≈50%≈75%100%
LLaDA 1.5HumanEval90.51%90.11%88.51%87.78%88.41%
LLaDA 1.5GSM8K91.27%89.14%89.18%88.95%89.78%
LLaDA 1.5RULER90.67%90.60%90.57%90.61%90.76%
DreamHumanEval90.16%91.21%91.57%91.90%92.27%
DreamGSM8K90.39%91.19%91.39%91.46%91.41%
DreamRULER91.81%91.96%91.92%91.99%91.89%

The results show that column sparsity remains consistently high throughout denoising across all evaluated models, tasks, and stages.

Mechanism analysis. We attribute column sparsity to the high uncertainty and limited semantic differentiation of masked tokens. During early denoising, most positions are identical [MASK] tokens, providing little token-specific semantic information. Consequently, queries at these positions may exhibit similar attention preferences, concentrating on a small set of contextual or structural anchors, such as informative prompt tokens and attention sinks [1]. Since these anchors attract attention from many different queries, the attention matrix naturally exhibits column-wise concentration, rather than uniformly distributed or coherent block-wise patterns.

As denoising progresses, recovered tokens establish richer semantic dependencies, potentially giving rise to more regular local or block-like attention patterns. Importantly, fine-grained column-wise sparsity can also capture compatible coarse-grained block-wise patterns, allowing PulseCol to accommodate evolving attention structures throughout denoising.

W3: Lack of evaluation on LLaDA-2.

We thank the reviewer for this suggestion. We have extended our experiments on RULER to include a more recent dLLM backbone, LaDA 2.0, along with two recent sparse attention baselines, BA-Att [2] and Focus-dLLM [3].

On LLaDA 2.0, PulseCol achieves an average accuracy of 91.14 at 80% sparsity, closely matching 91.33 with full attention, as shown below. The average is computed over all 13 RULER subtasks.

MethodSingleMulti-keyMQMVVTSQuADHotpotFWECWEAverage
LLaDA 2.099.1399.6083.4099.7592.1684.2063.0071.6096.9891.33
PulseCol99.2099.5383.3099.8591.6083.4063.0071.6095.9291.14

On LLaDA-1.5, PulseCol also achieves higher average accuracy than both BA-Att and Focus-dLLM at 50% and nominal 80% sparsity, as summarized below.

MethodsparsityGSM8KMATH-500HERULER-4KAvg.
Focus-dLLM50%77.3340.235.3783.9159.2
BA-Att50%73.7741.434.7685.9158.96
PulseCol (Ours)50%81.0541.838.4190.3662.91
Focus-dLLM80%75.364035.9875.4856.7
BA-Att80%32.7528.29.7647.0329.43
PulseCol (Ours)80%76.1938.438.4191.7261.18

Note that Focus-dLLM only sparsifies prompt KV while retaining response KV, so its effective sparsity is not directly comparable to that of PulseCol.

Since BA-Att has not released its code, we reproduced it following the original paper and verified that our results are consistent with the reported performance. We have made both the PulseCol implementation and BA-Att reproduction available through anonymized repositories to facilitate reproducibility.

We have included these additional experimental results and references in the revised manuscript.


Q2: Unexplained cross-model refresh behavior and adaptive policy feasibility.

We thank the reviewer for this insightful question. The different behaviors may arise from differences in how attention patterns evolve during denoising across models. For example, concentrating refreshes in earlier steps slightly improves LLaDA-1.5 but degrades performance on Dream, suggesting that Dream may benefit from more evenly distributed updates. Overall, the uniform schedule provides a simple and stable choice across both models.

A more general adaptive policy is a promising direction. One possible approach is to monitor changes in attention patterns and trigger refreshes when the sparse indices become less representative. However, such a policy would require an efficient way to detect these changes without introducing substantial overhead. We will clarify this limitation and discuss adaptive refresh as a direction for future work.


Q3: Missing fine-grained latency breakdown across pipeline components.

Thank you for this helpful suggestion. We have added a latency breakdown on RULER at 32K sequence length, 80% sparsity, and 128 denoising steps. The main values report total latency, while values in parentheses denote per-step time ×\times executed steps. Although index construction introduces some overhead, it is outweighed by the latency savings from sparse attention. We will include a clearer visualization of this breakdown in the revised manuscript.

MetricLLaDAPulseCol
End-to-End Latency469.23 s346.43 s
Full-Attention Computation Time220.64 s (1.72 s x 128)N/A
Indexing OverheadN/A53.72 s (3.35 s x 16)
Column-Sparse Attention TimeN/A40.62 s (0.36 s x 112)
Total Attention Time220.64 s94.34 s
Attention Speedup1.00x2.34x

Q4: Lack of adaptive or top-P budget allocation.

We thank the reviewer for this thoughtful question. We have conducted additional experiments comparing PulseCol's fixed Top-K selection with Ada-KV [4], which uses adaptive budget allocation, and Top-p, which selects tokens based on cumulative attention scores. The results on GSM8K, MATH500, and HumanEval are summarized below, with numbers in parentheses for Top-k indicating the actual achieved sparsity.

MethodBudgetGSM8KMATH500HumanEvalAvg.
PulseCol (ours)top-k=50%81.0541.8038.4153.75
Ada-KVadaptive 50%79.1543.2040.2454.20
PulseCol (ours)top-k=80%76.1938.4038.4151.00
Ada-KVadaptive 80%74.6038.4037.8050.27
Top-pp = 0.506.37 (95.53)1.40 (95.45)1.83 (94.95)3.20
Top-pp = 0.7024.11 (89.20)23.40 (88.88)24.39 (87.77)23.97
Top-pp = 0.8033.74 (82.54)32.00 (82.09)29.88 (80.63)31.87
Top-pp = 0.9053.90 (69.21)36.40 (68.48)35.37 (66.52)41.89
Top-pp = 0.9569.52 (55.52)41.00 (54.33)39.02 (52.06)49.85

PulseCol matches Ada-KV at 50% sparsity and slightly outperforms it at 80% sparsity, while Top-p still lags behind even at p=0.95p = 0.95 with around 54% sparsity. These results indicate that a uniform Top-K budget works well for PulseCol while keeping the design simple and predictable. We have added these comparisons to the revised manuscript.

References:

[1] Attention Sinks in Diffusion Language Models, ACL 2026 Findings.

[2] Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention, CVPR 2026 Findings.

[3] Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing, ACL 2026.

[4] Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference. NeurIPS 2025.

Reviewer_phhYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PulseCol, a sparse attention method for accelerating diffusion language models. The method is motivated by the empirical observation that attention exhibits a column-sparse pattern in the early stages of denoising. Based on this observation, PulseCol groups query tokens and selects important key positions for each group using aggregated attention scores. The selected column indices are reused across neighboring denoising steps, periodically refreshed during the early stage, and held fixed thereafter. This enables early sparse computation while limiting the overhead of sparse pattern construction. The authors also implement a dedicated column-sparse kernel to achieve practical acceleration. Experiments on LLaDA-1.5 and Dream cover mathematical reasoning, code generation, and long-context tasks. The results show that PulseCol better preserves model performance than SparseD at high sparsity levels and achieves up to 1.95x end-to-end speedup over a standard attention.

Critical strengths
  1. This paper introduces an practical method for accelerating DLM with long sequence lengths. This will be more and more important with the development of DLMs.
  2. The method is training-free, and easy to follow. The author addressed the shift of attention in early stages and developed dynamic adjustment of the attention pattern.
  3. The performance under high sparsity are quite good.
Critical weaknesses
  1. At 80% sparsity, SparseD exhibits a substantial performance drop on GSM8K, whereas PulseCol largely preserves the full-attention baseline’s accuracy. However, PulseCol’s advantage over SparseD is much smaller on MATH-500 and is not consistent across models. Could the authors elaborate on this discrepancy? One possible explanation is a difference in sequence lengths, which may affect the effectiveness of the selected sparse patterns. It would be helpful if the authors can elaborate more on this.
  2. Evaluating more recent diffusion language models and additional relevant baselines would strengthen the empirical results.
  3. The paper would benefit from a deeper explanation of why column-wise attention sparsity emerges during early denoising, beyond the existing empirical observations.
Questions for authors

Please see weaknesses.

2 comments

W1: Discrepancy in PulseCol’s gains over SparseD between GSM8K and MATH-500 at 80% sparsity.

We thank the reviewer for this insightful observation. Shorter valid output before EOS relative to the maximum sequence length is indeed the primary cause, as it inflates SparseD’s effective full-attention ratio far beyond its nominal 20% on MATH-500. This behavior is mainly due to the characteristics of the task rather than the sparsity method itself.

Specifically, SparseD schedules full attention for the first 20% of the maximum generation budget (~53 steps out of 256 on GSM8K vs. ~206 steps out of 1024 on MATH-500), which accounts for a disproportionately large share of the valid output once truncated at EOS:

DatasetModelMethodAvg. Actual Denoising StepsAvg. Dense Attention StepsDense Attention Step Ratio
GSM8KLLaDA-1.5SparseD245.1553.0021.62%
GSM8KLLaDA-1.5PulseCol235.3315.976.79%
GSM8KDreamSparseD142.6652.8537.05%
GSM8KDreamPulseCol133.727.795.83%
MATH-500LLaDA-1.5SparseD667.53205.2630.75%
MATH-500LLaDA-1.5PulseCol645.0715.532.41%
MATH-500DreamSparseD407.63201.2749.38%
MATH-500DreamPulseCol417.686.901.65%
  1. Smaller gap on MATH-500: SparseD’s accuracy is supported by a larger share of dense computation, reaching 30.75%–49.38% of valid output length compared to 21.62%-37.05% on GSM8K, which helps preserve its performance and narrows the gap with PulseCol.
  2. Cross-model inconsistency: Because Dream generates more concise responses on MATH-500, SparseD’s full-attention steps cover 49.38% of Dream’s valid output length versus 30.75% on LLaDA-1.5. In contrast, PulseCol uses just 1.65% and 2.41% full steps on both models, meaning SparseD benefits from a substantially larger share of dense steps on Dream than on LLaDA-1.5.

We have included this analysis and clarified this point in the revision.


W2: Lack of evaluation on recent dLLMs and baselines.

We thank the reviewer for this constructive suggestion. We have expanded our experiments on the RULER benchmark to include a more recent dLLM backbone, LLaDA 2.0, as well as two recent sparse baselines, BA-Att [1] and Focus-dLLM [2].

On LLaDA 2.0, PulseCol closely preserves the accuracy of the full-attention baseline at 80% sparsity, confirming its effectiveness on newer dLLM architectures. On LLaDA-1.5, PulseCol achieves clear accuracy advantages over both BA-Att [1] and Focus-dLLM [2] under their official configurations as well as a matched 80% sparsity setting, as summarized below. Since BA-Att has not released its code, we reproduced it following the original paper and verified that our results are consistent with the reported performance. To facilitate reproducibility, we have made both the PulseCol implementation and BA-Att reproduction available through anonymized repositories.

The average is computed over all 13 RULER subtasks.

MethodSingleMulti-keyMQMVVTSQuADHotpotFWECWEAverage
LLaDA 2.099.1399.6083.4099.7592.1684.2063.0071.6096.9891.33
PulseCol99.2099.5383.3099.8591.6083.4063.0071.6095.9291.14
MethodsparsityGSM8KMATH-500HERULER-4KAvg.
Focus-dLLM50%77.3340.2035.3783.9159.20
BA-Att50%73.7741.4034.7685.9158.96
PulseCol (Ours)50%81.0541.8038.4190.3662.91
Focus-dLLM80%75.3640.0035.9875.4856.70
BA-Att80%32.7528.209.7647.0329.43
PulseCol (Ours)80%76.1938.4038.4191.7261.18

Note that Focus-dLLM sparsifies only prompt KV while retaining response KV, resulting in a lower effective sparsity than PulseCol under the nominal 80% setting.

We have incorporated these new results and references into the revised manuscript.

W3: Mechanism behind early column sparsity.

We thank the reviewer for this insightful comment. We provide an information-theoretic interpretation of early column sparsity and the evolution of attention sparsity during denoising.

Early-stage uncertainty and column sparsity. At early denoising steps, most positions contain identical [MASK] tokens and lack differentiated semantic information. Consequently, many queries tend to rely on a limited set of shared contextual or structural anchors, such as informative prompt tokens and attention sinks [3]. When queries consistently attend to these few key positions, attention concentrates along specific columns, giving rise to fine-grained column sparsity rather than coherent block-wise patterns.

Entropy reduction and the emergence of structured sparsity. As denoising progresses, masked positions are progressively resolved, reducing the residual uncertainty about the target sequence as reliable token information accumulates [4]. The sequence gradually evolves from predominantly undifferentiated [MASK] positions into semantically organized units, such as phrases, mathematical expressions, or code segments, reflecting progressive semantic refinement [5]. As these structures emerge, dependencies among related tokens may become more pronounced, potentially giving rise to increasingly coherent local attention patterns, including diagonal or block-like structures. This provides a plausible explanation for why coarse-grained block sparsity becomes more effective in later denoising stages, whereas SparseD retains dense attention early to avoid losing important fine-grained dependencies.

Importantly, fine-grained column-wise sparsity can also capture compatible coarse-grained block-wise patterns, enabling PulseCol to exploit sparsity throughout denoising rather than primarily in later stages.

References:

[1] Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention, CVPR 2026 Findings.

[2] Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing, ACL 2026.

[3] Attention Sinks in Diffusion Language Models, ACL 2026 Findings.

[4] Information-Theoretic Discrete Diffusion, NeurIPS 2025.

[5] Next Semantic Scale Prediction via Hierarchical Diffusion Language Models, NeurIPS 2025.

Comments on the paper

1 comment
Area_Chair_r2DpOpenReview ↗

Please take a look at authors' rebuttal

Dear reviewers,

The authors have now submitted their responses. Please review them and share your thoughts on whether your concerns have been addressed. I encourage you to engage with one another’s assessments, discuss any remaining disagreements, and update your reviews and scores as appropriate.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.