acceptodds
Under review as a conference paper at ICLR 2027

HiAda: History-Aware Adaptive Decoding for Diffusion Large Language Models

Abstract

Diffusion large language models (dLLMs) enable parallel token generation through iterative denoising, but their practical inference remains costly due to dynamically changing bidirectional representations, the limited applicability of conventional KV caching, and rigid block-wise decoding schedules. These limitations call for inference strategies that can reuse computation while adapting to the evolving decoding state. We propose HiAda, a training-free, history-aware, and adaptive inference framework that exploits predictions and attention signals accumulated throughout the denoising trajectory. HiAda constructs a token-level active set to jointly coordinate KV-cache updates and token commitment, while dynamically adapting the decoding scope to the model's evolving confidence. This design reduces redundant computation, mitigates stale cache reuse, and improves the effective parallelism of dLLM decoding. Experiments on LLaDA-8B-Instruct and Dream-7B-Base across GSM8K, MATH, HumanEval, and MBPP show that HiAda achieves up to 51.1× speedup over vanilla diffusion decoding and up to 2.01× speedup over Fast-dLLM, while offering a controllable quality–efficiency trade-off. These results demonstrate that combining decoding history with adaptive computation can substantially improve the efficiency of diffusion language model inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion large language models, efficient inference, adaptive decoding, KV caching, parallel decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1tLaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HiAda, a training-free, history-aware adaptive decoding framework for accelerating diffusion large language models (dLLMs). HiAda introduces an attention-guided active set that separates token prediction from KV-cache updates, an adaptive mask window that removes fixed-block decoding constraints, and a history-aware confidence mechanism that promotes early commitment of repeatedly predicted tokens. Periodic full KV-cache refreshes are employed to mitigate representation staleness. Experiments on LLaDA-8B-Instruct and Dream-7B-Base across four reasoning and code-generation benchmarks demonstrate up to 51.1× speedup over vanilla diffusion decoding and 2.01× over Fast-dLLM, with task-dependent accuracy–efficiency trade-offs.

Critical strengths
  1. Significant inference acceleration. HiAda demonstrates substantial throughput improvements over vanilla diffusion decoding and Fast-dLLM across multiple benchmarks and generation lengths. These results suggest that the proposed framework can effectively reduce redundant computation in dLLM inference.
  2. A unified and training-free inference framework. The proposed method jointly coordinates token selection, KV-cache updates, and parallel decoding without additional model training. The integration of historical attention and prediction signals provides a practically motivated approach to adaptive computation.
  3. Comprehensive component-level analysis. The paper provides incremental ablation studies, hyperparameter sensitivity analyses, and computational-complexity derivations, offering useful insights into the contributions and trade-offs of the proposed components.
Critical weaknesses
  1. Limited methodological novelty. The core components of HiAda closely resemble existing methods. In particular, d²Cache [1] already performs fine-grained token-level KV-cache updates, making the claimed distinction in Lines 114–117 regarding token-level granularity insufficiently justified. Similarly, the history-aware confidence mechanism resembles CreditDecoding [2], which accumulates historical prediction evidence to accelerate token commitment. The novelty appears to lie primarily in combining existing techniques.

  2. Insufficient comparisons with competitive baselines. The main experiments primarily compare HiAda against vanilla decoding and Fast-dLLM, omitting competitive recent methods such as d²Cache [1] and CreditDecoding [2]. Although Dynamic-dLLM and ES-dLLM are additionally evaluated in the appendix, their task-performance scores are not reported, making it difficult to assess the accuracy–efficiency trade-off.

  3. Limited applicability to emerging dLLM architectures. Recent dLLMs, including LLaDA2.0 [3] and Fast-dLLM v2 [4], adopt block-diffusion architectures. HiAda's block-free decoding strategy is not directly compatible with their block-causal constraints, potentially limiting its practical applicability to newer dLLMs.

  4. Potential hardware inefficiency from non-contiguous token selection. HiAda selects scattered token positions, whereas Fast-dLLM processes contiguous blocks. Prior work such as Native Sparse Attention [5] highlights the importance of hardware-aligned computation and efficient memory access. HiAda's irregular indexing and gather/scatter operations may introduce additional overhead, potentially reducing its throughput advantage on GPUs such as the NVIDIA A100.

Minor Writing and Presentation Issues

  1. Section 4.4 states that the full-refresh interval is determined as a function of k, but does not specify the actual function in the main text, requiring readers to consult the appendix.

  2. Several sentences contain missing spaces after punctuation, including Lines 36 ("process.Unlike"), 75 ("quality.As"), 119 ("confidence.Fast-dLLM"), 330 ("updates.Accordingly"), and 507 ("Harness.An").

  3. Line 919 contains stray quotation marks or unintended characters (```).

Reference

[1] Jiang et al. d²Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching. ICLR 2026. Paper

[2] Wang et al. CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Credit. ACL 2026. Paper

[3] Bie et al. LLaDA2.0: Scaling Up Diffusion Language Models to 100B. arXiv 2025. Paper

[4] Wu et al. Fast-dLLM v2: Efficient Block-Diffusion LLM. ICLR 2026. Paper

[5] Yuan et al. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. ACL 2025. Paper

Questions for authors
  1. How does HiAda scale to longer sequences? Under the proposed refresh schedule R=⌊N/k⌋R=\lfloor N/k\rfloor, full KV-cache refreshes become less frequent as the generation length increases, potentially aggravating cache staleness. Could the authors evaluate longer generation lengths (e.g., 1K tokens) and tasks involving substantially longer prompts, comparing both throughput and task performance with recent acceleration methods?

  2. How does HiAda perform on other GPU architectures? As discussed in W4, non-contiguous token selection may introduce additional hardware overhead. Could the authors evaluate HiAda on GPUs such as the NVIDIA A100 and compare its throughput against Fast-dLLM under identical settings? A runtime breakdown of token indexing, sparse computation, and cache updates would help clarify the impact of irregular memory access.

Reviewer_aXQUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HiAda is a training-free decoding framework for masked diffusion language models. During sparse steps, it combines attention-selected masked positions with a FIFO set of recently committed positions for KV recomputation, while reusing cached states elsewhere. A moving window of remaining masked positions and a history-dependent confidence score promote parallel commitment. An attention window supports selection after the window advances, and an argmax fallback ensures progress when no candidate reaches the threshold. Periodic full-sequence refreshes limit cache staleness; It refreshes more frequently as fewer masks remain. Experiments evaluate two models on four reasoning and code benchmarks, with reported speedups up to 51.1× over vanilla decoding and 2.01x over Fast-dLLM, subject to quality trade-offs.

Critical strengths
  1. A concrete token-level inference design. HiAda coordinates prediction at selected masked positions with KV updates at recently committed positions. This reduces token-wise Transformer computation, including FFN work, while retaining access to the full cached context. The extended window and refresh procedure make the scheduling mechanism more complete than a simple proposal to skip tokens.
  2. Meaningful measured speed improvements. Table 1 includes configurations that improve both throughput and task score over Fast-dLLM: Dream GSM8K-512 improves from 41.54 TPS / 76.57 to 61.80 TPS / 76.65, and LLaDA HumanEval-512 from 65.57 TPS / 39.0 to 93.3 TPS / 42.7. Results across two models and four tasks support practical value, although the largest speedup is not universally lossless.
Critical weaknesses
  1. The incremental contribution and quality-matched advantage remain unclear. Elastic-Cache[1] and CreditDecoding[2] already introduce closely related window-based and history-based mechanisms, but neither is experimentally compared. Figure 7 also reports newer baselines’ throughput without task scores, while disabling ES-dLLM’s documented parallel-decoding capability[3]. These omissions make it difficult to distinguish better scheduling from different quality–speed trade-offs.
  2. The evidence for the core selection mechanism needs clarification. Hit ratio is defined as output-token agreement in Section 3.2 but as selected-position precision in Section 5.2/Appendix A.3, leaving Figures 3–4 ambiguous. Furthermore, k changes both selection budget and refresh frequency, so its effects are confounded. Fixed-budget controls are needed to establish the value of attention selection and FIFO updates, especially because below-threshold fallback commitments cannot subsequently be revised.

[1] Quan Nguyen-Tri, Mukul Ranjan, and Zhiqiang Shen. Attention Is All You Need for KV Cache in Diffusion LLMs. arXiv:2510.14973, 2025.

[2] Kangyu Wang et al. CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Credit. ACL, 2026.

[3] Zijian Zhu, Fei Ren, Zhanhong Tan, and Kaisheng Ma. ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping. ICLR, 2026.

Questions for authors
  1. Attention-layer configuration: Which exact layer provides the selection scores in each main experiment? How was it chosen, and is the same relative layer used across models and tasks?
  2. Metric definition: What are the numerator, denominator, reference decoding trajectory, and parallel-decoding setting for the hit ratios in Figures 3–4? Is the metric output-token agreement, recall of the next decoded position, or selected-set precision?
  3. Mechanism validation. With the window size, recomputation budget, and refresh schedule held fixed, how much does attention selection improve over a simple position selector, and what evidence supports FIFO-based cache updates? How frequently does the below-threshold argmax fallback activate, and does restricting it to the selected subset introduce additional quality loss?
  4. Closest comparisons. Does the advantage remain against Elastic-Cache and CreditDecoding, and against the documented parallel version of ES-dLLM, at comparable task quality? Please provide the task scores accompanying Figure 7.
Reviewer_tkC4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HiAda, a training-free, history-aware adaptive decoding framework for diffusion large language models. HiAda introduces an attention-guided active set to jointly coordinate masked-token prediction and KV-cache recomputation at the token level. It further incorporates an adaptive mask window to overcome fixed block boundaries and history-aware confidence accumulation to improve parallel decoding. Experiments on LLaDA and Dream demonstrate the effectiveness of the proposed framework.

Critical strengths

The paper presents a practical and training-free framework that integrates adaptive token selection, KV-cache reuse, and parallel decoding to improve inference efficiency. The experiments on LLaDA and Dream demonstrate substantial speedups across multiple benchmarks, highlighting the practical value of the proposed approach.

Critical weaknesses
  1. Limited comparisons with existing caching methods. Additional comparisons with D²-Cache [1] and dLLM-Cache [2] would help clarify the advantages of HiAda over existing adaptive caching methods.
  2. Lack of qualitative analysis. The paper would benefit from a case study illustrating how the attention-guided selection and history-aware confidence mechanisms affect the decoding process. Such an analysis would provide a more intuitive understanding of the proposed method.

[1] D²-Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Caching.

[2] dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching.

Questions for authors

It would be valuable to provide additional comparisons with D²-Cache and dLLM-Cache, as well as a qualitative case study illustrating the decoding process. These results would help clarify the advantages of HiAda and provide a more intuitive understanding of the proposed method.

It would also be interesting to explore the reverse attention direction for the attention-guided selection strategy. Since newly decoded tokens change from [MASK] to actual tokens, attention from masked positions to newly decoded tokens may better reflect the extent to which these changes affect the masked positions. A comparison of the two attention directions or their combination would help validate the proposed strategy.

If these additional results further demonstrate the effectiveness and advantages of HiAda, I would be willing to increase my score.

1 comment

Thank you for these constructive suggestions. We have conducted additional experiments and investigated the proposed reverse attention direction.

1. Additional Comparisons and Case Study

We have extended Figure 7 with comparisons against D²-Cache and Elastic-Cache under matched hardware and benchmark settings. HiAda achieves 93.3, 226.5, 130.0, and 67.7 TPS on HumanEval, MATH, MBPP, and GSM8K, respectively, compared with approximately 7.5, 6.0, 2.4, and 18.0 TPS for D²-Cache, corresponding to 3.8–54× higher throughput. Compared with Elastic-Cache (approximately 17, 55, 47, and 50 TPS), HiAda achieves 1.4–5.5× higher throughput. Figure 7 also reports the corresponding task accuracy, illustrating the efficiency–quality trade-off.

These improvements arise from several methodological differences:

  • D²-Cache does not explicitly decouple masked-token prediction from KV recomputation, while its cross-layer attention rollout introduces additional overhead. HiAda uses independent active sets for these two tasks, leveraging historical attention from a single layer to predict promising decoding positions with lower selection overhead.

  • dLLM-Cache requires additional Attention/FFN output caches (approximately doubling KV-cache storage under MHA). Its V-verify mechanism may also miss context-induced changes at positions whose token identities remain unchanged, requiring periodic full refreshes. HiAda avoids dense Value verification and additional intermediate-feature caches, directly selecting active positions before sparse computation.

Furthermore, HiAda's block-free adaptive window maintains decoding parallelism without waiting for fixed blocks to complete. We will also provide a qualitative decoding trace illustrating the evolution of active sets and the adaptive window.

2. Reverse Attention Direction

We appreciate this insightful suggestion. The proposed reverse attention direction is well-motivated, as newly decoded tokens transition from [MASK] to actual tokens, and attention from remaining masked positions to these newly decoded tokens may intuitively reflect their influence on subsequent predictions. This provides a meaningful alternative perspective for identifying promising decoding positions.

Motivated by this observation, we evaluated the reverse direction under identical settings.

Hit Ratio (generation length = 512, block size = 64, last layer):

Top-kOursReverse
483.6%77.20%
893.9%87.42%
1297.2%91.76%
1698.2%94.29%
2099.0%95.97%

The reverse direction consistently reduces the hit ratio by 3.0–6.5 percentage points, suggesting that our original direction provides more reliable predictions of future decoding positions.

Furthermore, reverse attention is less compatible with our token-level sparse computation. HiAda computes attention queries only for the active mask and cache sets, while cached keys cover the entire sequence. Our direction therefore directly provides the required attention scores, whereas reverse attention requires additional queries from unselected masked positions, introducing extra computation and undermining sparsity.

These results support our original attention direction in terms of both selection accuracy and computational efficiency.

Comments on the paper

1 comment

We sincerely thank the reviewer for the thoughtful and constructive comments. We appreciate the opportunity to clarify our methodology and strengthen the experimental evaluation. We address the concerns as follows.

1. Attention-layer Configuration

We consistently use the third-to-last attention layer to compute selection scores across all models and tasks, without any model-specific or task-specific tuning. This choice is based on a one-time offline calibration, where we evaluated the hit ratio of each layer independently of downstream task accuracy.

We observed that the hit ratio generally increases with layer depth and saturates around the third-to-last layer in both models. Despite their different depths, the same relative layer position works well for both, suggesting potential generalizability. We hypothesize that this reflects the progressive convergence of deeper-layer representations toward final predictions, consistent with logit-lens observations in autoregressive LLMs. Furthermore, our sensitivity analysis shows only marginal performance variations across the last five layers, indicating robustness to this choice.

2. Metric Definition

The hit-ratio evaluation serves as an initial validation of our position-selection mechanism. We conduct this evaluation on the original dLLM without decoding optimizations to examine whether the positions selected for decoding fall within the candidate set identified by our attention-based method.

We evaluate 500 distinct prompts covering code generation, mathematical reasoning, everyday conversations, and general knowledge question answering. As shown in Figure 3, the hit ratio exceeds 90%, indicating that our method effectively captures the positions selected during the original decoding process. Meanwhile, the impact on downstream task accuracy remains relatively small, supporting the effectiveness of our selection strategy.

3. Mechanism Validation

(1) Attention-based position selection. Existing confidence-based selection methods generally require computing final-layer representations for all candidate tokens, limiting computational sparsity. In contrast, our attention-based strategy enables position selection without relying on final-layer confidence scores, allowing unnecessary computations to be skipped in subsequent layers.

(2) FIFO-based cache updates. Our design is motivated by previous observations that recently decoded tokens tend to exhibit larger changes in their key-value (KV) representations during subsequent decoding steps. Accordingly, we adopt a FIFO-based strategy for efficient cache management.

(3) Sliding-window effectiveness. Experiments on MBPP using LLaDA show that our sliding-window strategy reduces the rollback ratio from 45% to 22% compared with a fixed-window strategy, demonstrating its effectiveness in adapting to changes during iterative decoding.

(4) Fallback strategy. The candidate set used by our method is larger than that of the original block-wise decoding approach, providing a broader pool of potential decoding positions. We believe this helps mitigate possible accuracy degradation when restricting the fallback operation to the selected subset.

4. Closest Comparisons

CreditDecoding has not publicly released its source code, making a direct and reproducible comparison challenging. To address this concern, we have conducted additional experiments with the other relevant baselines and revised the manuscript accordingly. The updated results, including the corresponding downstream task scores, are presented in Figure 7 of the revised manuscript, providing a more comprehensive comparison of decoding efficiency and task performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.