acceptodds
Under review as a conference paper at ICLR 2027

Ask the Masked Block: Predicting Denoised-State Relevance for KV Cache Eviction in Diffusion Language Models

Abstract

Diffusion large language models (dLLMs) incur substantial inference costs by repeatedly processing the full sequence with bidirectional attention. Temporal caching reduces redundant computation but introduces additional memory overhead by retaining full-sequence KV states. Spatial eviction reduces this cache footprint, yet current-state attention may not identify the entries needed after denoising. We introduce Ask-dLLM, a KV cache eviction framework that predicts denoised-state relevance conditioned on the current masked block. A lightweight scorer learns from relevance measured after the same frozen dLLM denoises the block and ranks external cache candidates once per block using masked-state representations. Across 23 reasoning, code, and long-context benchmarks, Ask-dLLM improves the overall mean score over the prior eviction baseline by 7.75 points on LLaDA and 2.39 points on Dream while retaining only 10% of external KV entries. Its overall mean score remains within 1 point of the unmodified Origin model on both backbones. On four LongBench datasets at an 8K context length, Ask-dLLM achieves approximately 9.7× and 11.8× the throughput of Origin on LLaDA and Dream, respectively, while matching or exceeding its average task score. Code is available at https://anonymous.4open.science/r/36675/README.md.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
diffusion language models, KV cache eviction, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_727HOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the KV-cache eviction problem in full dLLMs like LLaDA and Dream. It learns per-head cache rankings from frozen denoised-state in backbone, and predicts eviction once per block without lookahead. Experiments show that it achieves notable gain over Sparse-dLLM across multiple benchmarks, while delivers great throughput speedup.

Critical strengths
  • The method follows directly from well discovered observation in Figure-1
  • The experiments are extensive on 23 diverse benchmarks.
Critical weaknesses
  • teacher targets and scorers must be re-extracted/retrained per backbone, there are gap between training and inference. What the impact of this gap is not studied.
  • In Table-2, the proposed method still get notable worse results than the original Dream, which shows the method is not that practical.
  • The training and testing protocol is not very described, especially what is the difference between training set and benchmark set.
  • What is the setting of the speedup comparison, batch-size, inference-engine, etc.
Questions for authors
  • To my knowledge, LLaDA and Dreams are models more than 20 months older. They do not reflect the applicable trends in dLLM, so the problem itself is far from Amazing to me. What if the method applies to newer models like LLaDA 2.x, SDAR and Diffusion-Gemma.
5 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1)

We thank Reviewer 727H for the careful review and for recognizing our well-motivated approach, extensive evaluation across 23 benchmarks, and notable accuracy and throughput gains. We address each concern point by point below, providing additional analyses and clarifications.

W1 Teacher targets must be re-extracted and the scorer retrained for each new backbone. The reviewer asks how the gap between full-cache training and inference with KV eviction affects final performance.

Thank you for highlighting the difference between the conditions the scorer encounters during training and inference. As described in Figure 2 (p. 3) and Sections 3.2–3.4 (pp. 4–5), the scorer uses masked-state hidden representations available at KV selection time in both settings. We obtain training labels by fully denoising each block with the full KV cache and measuring how much attention the completed block assigns to each KV entry. At inference, each block is denoised using the retained KV entries. The outputs of earlier blocks can therefore differ from those generated by the full-cache teacher, affecting KV selection in subsequent blocks.

To examine how this gap relates to generation quality, we varied the KV keep ratio and measured agreement with the full-cache teacher alongside performance relative to Origin. Table R1 presents these results, which are plotted in Figure 7 (Appendix C.8) of the revised manuscript.

Table R1. Teacher agreement and performance relative to Origin across KV keep ratios

KV keep ratioOurs agreement (%)Sparse-dLLM agreement (%)Ours / Origin (%)Sparse-dLLM / Origin (%)
0.993.692.0100.599.3
0.785.079.6100.998.4
0.578.369.499.496.6
0.372.559.698.080.7
0.166.943.197.748.8

In Table R1, both methods showed lower agreement with the teacher as more KV entries were removed, and the performance gap between them widened. When the keep ratio decreased from 0.5 to 0.1, Sparse-dLLM’s agreement fell from 69.4% to 43.1%, and its performance relative to Origin fell from 96.6% to 48.8%. Over the same range, Ask-dLLM’s agreement declined from 78.3% to 66.9%, while its performance remained close to Origin, moving from 99.4% to 97.7%.

W2 Ask-dLLM scores below the original Dream model in Table 2. The reviewer asks whether the practical benefits justify the accuracy loss.

Thank you for raising this trade-off. KV eviction reduces the number of cached entries available during decoding. Our goal is to preserve as much of the original model’s task performance as possible under a smaller KV budget.

On Dream, Table 2 (p. 7, lines 347–355) shows that even caching methods that retain all external KV entries score below the uncached Origin model (62.22). Fast-dLLM and dLLM-Cache score 60.21 and 60.91, respectively. Ask-dLLM retains only 10% of external KV entries and scores 58.50, a 3.72-point drop from Origin and 1.71 and 2.41 points below the two caching baselines, respectively. At the same 10% KV budget, Ask-dLLM outperforms Sparse-dLLM by 4.15 points on average, including gains of 12.80 points on HumanEval and 7.58 points on GSM8K.

Figure 3 (Section 4.3, p. 8, lines 401–421) shows the efficiency gains on four LongBench datasets. At 2K context, Ask-dLLM increases throughput from 3.00 to 21.19 tokens/s (7.1×) while using comparable peak GPU memory to Origin (15.65 vs. 15.85 GiB) and about 35% less than Fast-dLLM. At 8K, Ask-dLLM achieves a higher mean task score than Origin and approximately 11.8× its throughput. Fast-dLLM runs out of memory at that context length.

On Dream, Ask-dLLM retains 10% of external KV entries and achieves accuracy close to that of caching methods that retain all entries. It also substantially increases throughput at Origin-level peak memory.

AuthorsOpenReview ↗

General Response to All Reviewers (2)

W3 The reviewer asks for more detail on the data used to train the scorer and how it differs from the evaluation benchmarks.

Thank you for pointing out that the relationship between training and evaluation data could be clearer. The main text refers readers to Appendix A.2 (p. 14, lines 749–755) and Table 6 (p. 15) for the composition of the training pool. The pool contains 1,621 prompts from five sources: MATH (500), MBPP (371), GovReport (150), MultiNews (100), and MuSiQue (500). We used the same composition for both backbones. The prompts are disjoint from the evaluation splits listed in Table 5 (p. 14).

To assess how performance depends on the training sources, we recomputed the ρ = 0.1 averages from Tables 1–2 for two groups: five seen benchmarks that share a dataset source with the training prompts and 18 unseen benchmarks from sources absent from training. Here, seen refers to the dataset source, with evaluation prompts held out.

Table R2. Average scores on seen and unseen benchmarks at a 0.1 KV keep ratio.

BackboneMethodSeen (5 benchmarks)Unseen (18 benchmarks)
LLaDAOrigin27.5640.41
LLaDASparse-dLLM16.6234.54
LLaDAOurs25.7941.89
DreamOrigin31.0045.11
DreamSparse-dLLM27.1241.95
DreamOurs29.9344.23

On unseen benchmarks (Table R2), Ask-dLLM scores 1.48 points above Origin on LLaDA and 0.88 points below Origin on Dream. The corresponding differences on seen benchmarks are −1.76 and −1.07 points, respectively. Ask-dLLM thus remains close to Origin on benchmarks from sources absent from training. On LLaDA, its largest gains over Sparse-dLLM occur on GSM8K (+43.22 points) and HumanEval (+26.22 points), both of which are unseen.

In the revised manuscript, Tables 1–2 label each benchmark as seen or unseen, and Table 16 (Appendix C.9) reports the group means shown in Table R2.

W4 The reviewer asks for the experimental settings behind the speedup comparison, including the batch size, inference engine, and timing scope.

Thank you for asking for the details needed to reproduce the speed comparison. We released our implementation in the anonymous repository linked on p. 1, lines 28–29. Section 4.1 (p. 6, lines 311–315) describes the hardware and decoding settings.

We measured throughput and peak GPU memory in Figures 3, 5, and 6 on a single NVIDIA RTX A6000 (48 GB) with a batch size of 1. We used the official implementations of Sparse-dLLM, dLLM-Cache, and Fast-dLLM. All methods, including Origin, ran in the same PyTorch 2.5.1 environment with bf16 precision and without a separate serving engine such as vLLM or SGLang. Within each task, we used the same prompts, seeds, decoding settings, and evaluation harness across methods.

We compute throughput as the number of generated tokens divided by the total wall-clock generation time. This includes cache-refresh forward passes, Ask-dLLM’s scorer calls, and all denoising steps. The default configuration described in Section 4.1 uses a block size of 32 and sets the number of denoising steps equal to the generation length. The accuracy tables compare the two eviction methods at ρ = 0.5 and 0.1, and the speed and memory results in Figures 3, 5, and 6 use ρ = 0.1. The revised manuscript details these measurement conditions and the throughput definition in Appendix A.1 (p. 14, lines 744–747).

AuthorsOpenReview ↗

General Response to All Reviewers (3)

W4 The reviewer asks for the experimental settings behind the speedup comparison, including the batch size, inference engine, and timing scope.

Additional evaluation of practical deployment. To further examine the relationship between task performance and inference costs raised in your comment, we evaluated both LLaDA-8B-Instruct and Dream-v0-Instruct-7B on an NVIDIA Jetson Orin Nano [3], where the CPU and GPU share 8 GB of memory. All methods used TensorRT 10.4 with INT4 weight-only engines and a batch size of 1. Table R3 (Table 4 in Section 4.6 of the revised manuscript) reports performance across six benchmarks, together with decoding latency and KV cache capacity for generating 256 tokens from a 1,792-token GovReport prompt.

Table R3. INT4 benchmark performance, decoding latency, and KV cache size on Jetson Orin Nano. Boldface compares the two eviction methods.

BackboneMethodKV keep ratioHotpotQAQasperTRECSAMSumHumanEvalMATH500Avg.Decoding latency (s/request) ↓KV cache size (GiB) ↓
LLaDA-8B-InstructOrigin1.011.9716.1234.5630.6240.8536.2028.391150.9—
LLaDA-8B-InstructSparse-dLLM0.113.8416.7236.2531.5714.0220.0022.07136.5≈0.098
LLaDA-8B-InstructOurs0.113.2119.7647.9233.6734.7631.8030.19137.2≈0.098
Dream-v0-Instruct-7BOrigin1.032.8027.8330.0826.9939.0237.8032.421033.1—
Dream-v0-Instruct-7BSparse-dLLM0.128.4720.7526.0033.2021.3430.4026.69116.5≈0.011
Dream-v0-Instruct-7BOurs0.131.1320.8128.1731.7430.4931.4028.96117.0≈0.011

On LLaDA, Ask-dLLM reduced decoding latency from 1150.9 s to 137.2 s, achieving an 8.4× decoding speedup over the INT4 Origin model, while scoring 1.80 points above Origin on average. Compared with Sparse-dLLM, it improved the average score by 8.12 points, with a decoding-time difference of only 0.7 s at the same KV cache capacity.

These results provide concrete evidence of practical benefits under a tight memory budget: substantially faster decoding than Origin and higher average quality than Sparse-dLLM at the same KV cache capacity and comparable on-device latency.

Q1. The original evaluation focuses on LLaDA and Dream. Does Ask-dLLM remain effective on more recent diffusion language models such as LLaDA2.1, SDAR, and DiffusionGemma?

Thank you for suggesting an evaluation on more recent backbones. We chose LLaDA and Dream to match the experimental setup of Sparse-dLLM (AAAI 2026), a recent KV eviction baseline. This allowed us to compare KV selection strategies under the same backbones, cache budgets, and decoding settings. The same selection algorithm can be applied to other block-diffusion backbones that cache and reuse the KV states of finalized tokens. To test this, we evaluated Ask-dLLM on SDAR-8B-Chat [1] and LLaDA2.1-Mini [2]. Tables R4–R5 correspond to Tables 12–13 (Appendix C.6) in the revised manuscript. Given DiffusionGemma’s model size and the compute available during the rebuttal period, we will evaluate it and include the results in the final version.

Both models generate blocks sequentially and jointly denoise multiple tokens within the active block. SDAR and the single-block editing configuration of LLaDA2.1-Mini cache the KV states of finalized blocks and reuse them during subsequent decoding. These states remain fixed once a block is finalized, so we permanently evict the entries that Ask-dLLM does not select. Subsequent blocks use the retained KV states together with newly generated KV states.

Reviewer_5G9aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduce Ask-dLLM, a KV cache eviction framework that predicts denoised-state relevance conditioned on the current masked block. The authors argue that existing methods like Sparse-dLLM, uses attention scores from masked states to determine importance of KV entries, but these scores may not accurately reflect their importance after denoising. Their proposed alternative learns a scorer that predicts denoised-state attention relevance from masked-state representations and uses the predictions to select KV entries for each generation block. The author evaluates their method across 23 reasoning, code, and long-context benchmarks, showing improved accuracy compared to Sparse-dLLM at 10% KV retention ratio.

Critical strengths

The proposed method significantly improves task accuracy over Sparse-dLLM at a 10% KV retention ratio, showing the effectiveness of the learned scorer at high sparsity.

The authors compare accuracy, throughput, and memory usage against several existing methods, including Sparse-dLLM, Fast-dLLM, and dLLM-Cache. Ask-dLLM achieves better accuracy than Sparse-dLLM with comparable efficiency, while using substantially less memory than methods without KV eviction. The results also show better scalability to longer contexts.

Critical weaknesses

Limited practical benefit outside aggressive eviction settings: The improvements over Sparse-dLLM are mainly observed at very low KV retention ratios. At a 50% retention ratio, the gains are limited, while the method introduces an additional learned scorer. Although the paper reports memory usage and throughput, it does not establish how much these improvements benefit realistic workloads that have very tight memory budgets.

Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank Reviewer 5G9a for recognizing the accuracy gains under aggressive KV eviction and the favorable accuracy–efficiency trade-off. We appreciate the question about practical benefits under tight memory budgets. Below, we discuss how the gains vary with the KV keep ratio and present results from an 8 GB Jetson Orin Nano.

W1 The gains over Sparse-dLLM are most pronounced at very low KV keep ratios. At a 50% keep ratio, the gains are limited given the cost of an additional learned scorer. The reviewer also asks how the reported memory and throughput improvements translate into practical benefits for workloads with tight memory budgets.

Thank you for recognizing the accuracy gains under aggressive KV eviction and the favorable accuracy–efficiency trade-off. We also appreciate your question about how these gains translate to memory-constrained deployment.

As a KV eviction method for diffusion language models, Ask-dLLM aims to match the original model’s task performance while retaining fewer KV entries. At ρ = 0.5, both methods already perform close to Origin, leaving limited room for improvement. Across the 23 benchmarks in Tables 1–2 (pp. 6–7), Origin, Sparse-dLLM, and Ask-dLLM score 37.62, 37.73, and 37.63 on LLaDA and 42.04, 40.84, and 41.34 on Dream, respectively. Both methods are within 1.2 points of Origin. Ask-dLLM’s advantage grows as the KV budget decreases. At ρ = 0.1, Ask-dLLM outperforms Sparse-dLLM by more than 24 points on LLaDA, averaged across GSM8K, HumanEval, and MATH, as shown in Figure 1(d) (p. 2, lines 80–82).

Evaluation on an 8 GB Jetson. To assess performance under a practical memory constraint, we ran LLaDA-8B-Instruct and Dream-v0-Instruct-7B on an NVIDIA Jetson Orin Nano [1], where the CPU and GPU share 8 GB of memory. We used TensorRT 10.4 with INT4 weight-only engines and a batch size of 1, generating 256 tokens from a 1,792-token GovReport prompt. Table R1 corresponds to Table 4 in Section 4.6 of the revised manuscript.

Table R1. Benchmark performance of the INT4 deployment models and inference costs on Jetson Orin Nano

BackboneMethodKV keep ratioHotpotQAQasperTRECSAMSumHumanEvalMATH500Avg.Decode latency (s)KV cache (GiB)
LLaDA-8B-InstructOrigin1.011.9716.1234.5630.6240.8536.2028.391150.9—
LLaDA-8B-InstructSparse-dLLM0.113.8416.7236.2531.5714.0220.0022.07136.5≈0.098
LLaDA-8B-InstructOurs0.113.2119.7647.9233.6734.7631.8030.19137.2≈0.098
Dream-v0-Instruct-7BOrigin1.032.8027.8330.0826.9939.0237.8032.421033.1—
Dream-v0-Instruct-7BSparse-dLLM0.128.4720.7526.0033.2021.3430.4026.69116.5≈0.011
Dream-v0-Instruct-7BOurs0.131.1320.8128.1731.7430.4931.4028.96117.0≈0.011

Retaining only 10% of external KV entries, Ask-dLLM scored 1.80 points above Origin on LLaDA and 3.46 points below Origin on Dream on average. At the same 10% KV budget, it outperformed Sparse-dLLM by 8.12 points on LLaDA and 2.26 points on Dream. On the Jetson, the two methods used the same KV cache capacity, and their decoding times for 256 tokens differed by only 0.7 s on LLaDA and 0.5 s on Dream. These results show that Ask-dLLM can deliver higher average quality under aggressive eviction with the same KV cache capacity and comparable on-device latency as Sparse-dLLM.

Compared with the INT4 Origin models, Ask-dLLM reduced total decoding latency from 1150.9 s to 137.2 s on LLaDA and from 1033.1 s to 117.0 s on Dream, corresponding to 8.4× and 8.8× decoding speedups, respectively.

Reference

[1] NVIDIA Corporation. “NVIDIA Jetson Orin Nano Developer Kit.”, 2023.

Dear Reviewer 5G9a,

We sincerely appreciate your thoughtful review and your question about practical benefits under tight memory budgets.

Following your comments, we summarized how the gains of Ask-dLLM over Sparse-dLLM vary with the KV keep ratio. To examine its practical benefit in a memory-constrained setting, we also conducted an on-device evaluation on an 8 GB Jetson Orin Nano, reporting benchmark scores, decoding latency, and KV cache size for LLaDA and Dream.

We would be grateful to hear whether these results resolve the concern you raised. If anything remains unclear, we would be glad to provide further details and continue the discussion.

Thank you once more for your time and helpful feedback.

Thank you for your response. I will keep my score.

Reviewer_BWgUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. Motivation: Existing KV cache eviction methods for diffusion LLMs (e.g., Sparse-dLLM) choose which entries to keep from the current block's attention in its masked state. That attention often differs from what the block actually attends to once it is fully denoised. As a result, under tight cache budgets these methods tend to evict critical information and produce wrong answers.
  2. Method: Ask-dLLM treats cache eviction as conditional prediction. From the current block's masked-state hidden states, it predicts how much attention the denoised block will give each external KV entry, which the paper calls denoised-state relevance. This supervision signal is measured offline, with the same frozen dLLM acting as the teacher.
  3. Implementation: Each layer gets a lightweight MLP scorer, trained with a listwise KL loss plus a pairwise ranking loss to match the teacher's ranking. At inference, the scorer runs once per block and keeps the top-K external entries for each layer and KV head. It does not need attention maps, so it stays compatible with FlashAttention. The backbone stays frozen, and adapting to a new backbone takes under 8 GPU-hours.
  4. Results: The method was tested on 23 reasoning, code and long-context benchmarks with LLaDA and Dream. Keeping only 10% of external KV entries, Ask-dLLM scores 7.75 and 2.39 points higher than Sparse-dLLM in overall mean, and stays within 1 point of the original model. At 8K context length, its throughput is 9.7× and 11.8× that of the original model.
Critical strengths
  1. Clear motivation and principled formulation: The paper identifies a concrete mismatch. Existing eviction methods rely on the block's attention in its masked state, but that differs from the attention the block actually needs once it is denoised. The authors support this with the Denoised-State Reference analysis, a block-by-block GSM8K decoding trace, and a gap to Sparse-dLLM that grows to over 24 points at ρ = 0.1. Recasting eviction as predicting denoised-state relevance is simple and principled. The supervision comes from the frozen backbone itself, so no external labels are needed.
  2. Practical and lightweight design: The scorer has about 80M parameters across all 32 layers of LLaDA and takes roughly 40 ms per call. It runs once per block, on hidden states that the cache-refresh forward already computes. It needs no attention maps, so it stays compatible with FlashAttention. The backbone stays frozen, and adapting to a new backbone takes under 8 GPU-hours on a single 48 GB GPU. One trained scorer works across tasks, keep ratios, and block sizes not seen in training (16 and 64) without retraining.
  3. Strong and broad empirical results: Across 23 benchmarks on two backbones, Ask-dLLM keeps only 10% of external KV entries and still scores within 1 point of the original model in overall mean. Under aggressive eviction it beats Sparse-dLLM by a wide margin, e.g., on LLaDA it scores 55.88 vs. 12.66 on GSM8K and 39.02 vs. 12.80 on HumanEval. At 8K context its throughput is 9.7× and 11.8× that of the original model. It also uses less memory than caches that do not evict, such as Fast-dLLM, which runs out of memory on Dream at 8K.
Critical weaknesses
  1. Models are weak. The proposed method is evaluated on LLaDA and Dream. It would be better to provide results on state-of-the-art models such as LLaDA2.1, DiffusionGemma, etc.
  2. Method is limited to traditional bidirectional diffusion language model. Frontier block diffusion and dLLM draft models (such as DFlash, DSpark) are not supported.
Questions for authors
  1. Could the authors provide a per-block latency breakdown at different context lengths?
  2. Ask-dLLM currently selects the retained entries once per block. How does the timing of selection affect the final results?
7 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1)

We thank Reviewer BWgU for the constructive feedback and positive assessment of our clear motivation and principled formulation, practical and lightweight design, and strong empirical results across diverse benchmarks. We address each concern and question point by point below.

w1 The original evaluation focuses on LLaDA and Dream. The reviewer asks whether Ask-dLLM remains effective on more recent diffusion LLMs, such as LLaDA2.1 and DiffusionGemma.

Thank you for suggesting an evaluation on more recent backbones. We chose LLaDA and Dream to match Sparse-dLLM's experimental setting (Song et al., 2026), enabling a controlled comparison of KV selection strategies under matched backbones, cache budgets, and decoding settings.

To address this concern, we extended our evaluation to SDAR-8B-Chat [1] and LLaDA2.1-Mini [2] (Revision: Appendix C.6, Tables 12–13). Given DiffusionGemma’s model size and the compute available to us during the rebuttal period, we will evaluate it and include the results in the final version.

Both models generate blocks sequentially and denoise multiple tokens jointly within the active block. SDAR and the single-block editing configuration of LLaDA2.1-Mini cache the KV states of finalized blocks and reuse them during subsequent decoding. Ask-dLLM applies directly to this cache structure by predicting which of the cached entries will be useful for later decoding. Since the KV states of finalized blocks remain fixed in this setting, we permanently evict the unselected entries. Subsequent blocks use the retained KV states and newly generated KV states.

Table R1. SDAR-8B-Chat and LLaDA2.1-Mini performance under permanent KV eviction

BackboneMethodKV keep ratioHotpotQAQasperTRECSAMSumLCCGovReportHumanEvalMATH500Avg.
SDAR-8B-ChatOrigin1.043.7436.1533.9846.9461.7626.2876.2251.4047.06
SDAR-8B-ChatSparse-dLLM0.543.4935.7935.0846.1261.6124.8061.5950.8044.91
SDAR-8B-ChatOurs0.543.1136.6435.8846.7361.7126.0373.1750.0046.66
SDAR-8B-ChatSparse-dLLM0.343.7432.2935.3145.9459.2522.4148.7849.2042.12
SDAR-8B-ChatOurs0.343.7035.6437.4644.9558.0325.7574.3948.0045.99
LLaDA2.1-MiniOrigin1.045.2943.6838.7736.5255.8629.3578.0543.4046.37
LLaDA2.1-MiniSparse-dLLM0.545.4642.2741.9637.0554.4328.2267.0744.0045.06
LLaDA2.1-MiniOurs0.544.7742.4045.7938.5554.4729.0379.8844.6047.44
LLaDA2.1-MiniSparse-dLLM0.346.4541.4641.1437.1555.0927.8148.1743.0042.53
LLaDA2.1-MiniOurs0.345.4241.7352.0439.3154.8928.8579.2743.2048.09

For the benchmark evaluation (Table R1), we used a maximum sequence length of 4,096 tokens for both backbones, with block sizes of 4 for SDAR-8B-Chat and 32 for LLaDA2.1-Mini. On SDAR, Ask-dLLM updates the KV cache and evicts unselected entries every 32 tokens.

At 50% and 30% KV keep ratios, Ask-dLLM stays within 1.07 points of Origin on SDAR and exceeds Origin on LLaDA2.1-Mini, where it outperforms Sparse-dLLM by 2.38 and 5.56 points, respectively. At the 30% keep ratio, Ask-dLLM improves HumanEval over Sparse-dLLM by 25.61 points on SDAR and 31.10 points on LLaDA2.1-Mini.

Table R2. Decoding latency and KV cache size for SDAR-8B-Chat and LLaDA2.1-Mini on LongBench QMSum at 26K context

BackboneMethodKV keep ratioDecoding latency (s/request) ↓KV cache size (GiB) ↓
SDAR-8B-ChatOrigin1.044.473.656
SDAR-8B-ChatSparse-dLLM0.338.721.100
SDAR-8B-ChatOurs0.340.151.100
LLaDA2.1-MiniOrigin1.013.101.014
LLaDA2.1-MiniSparse-dLLM0.311.960.304
LLaDA2.1-MiniOurs0.312.580.304

We measured decoding latency on 10 LongBench QMSum examples using a single RTX A6000. After one warm-up run, we generated 512 tokens per example using a 26,112-token input context. Decoding latency excludes prefill, and KV cache size is measured at the end of generation.

As shown in Table R2, Ask-dLLM permanently evicts 70% of KV entries, leaving final KV caches of 1.100 GiB on SDAR and 0.304 GiB on LLaDA2.1-Mini. Relative to Origin, decoding latency decreases by 9.7% on SDAR and 4.0% on LLaDA2.1-Mini. At the same keep ratio, Ask-dLLM and Sparse-dLLM have identical final KV cache sizes, and Ask-dLLM’s decoding latency is 3.7% higher on SDAR and 5.2% higher on LLaDA2.1-Mini.

AuthorsOpenReview ↗

General Response to All Reviewers (2)

W2. Frontier block diffusion and dLLM draft models (such as DFlash, DSpark) are not supported.

Ask-dLLM is not inherently limited to traditional bidirectional dLLMs. Its principle of predicting completed-block KV relevance from currently available representations also extends to block-diffusion models and diffusion-based drafters. Adaptation involves matching the cache candidates and scorer inputs to each framework and training on completed-block relevance collected with eviction disabled. This supervision is used only offline; inference requires no lookahead.

Our original experiments followed Sparse-dLLM’s setting for a controlled comparison. We have now additionally evaluated LLaDA2.1-Mini and SDAR (Tables R1–R2) and Qwen3-4B [4] with the DSpark [5] and DFlash [6] drafters in DeepSpec (Tables R3–R4) to directly assess applicability beyond this initial setting. Tables R3–R4 correspond to Tables 14–15 in Appendix C.7 of the revised manuscript.

In our speculative decoding experiments, the drafter constructs a seven-position block, and Qwen3-4B verifies the proposal in a single forward pass. The target model can reuse the KV states of committed tokens in subsequent verification steps. At the first proposal and approximately every 32 generated tokens thereafter, we select which entries to retain for each layer and KV head before target verification and permanently evict the remaining entries. Selection uses the existing cache and drafter representations, so it requires no additional full-sequence forward pass.

The scorer uses the context feature at each candidate position and the hidden states of the masked draft block, both already computed by the drafter. To account for permanent eviction, we extend the supervision horizon to the rest of the answer. Each KV entry’s training label is the maximum target-model attention it receives from any remaining answer token. We applied the same KV budget and permanent eviction policy to Sparse-dLLM, which selects entries using scores from the first target verification in each window. Table R3 reports generation quality, and Table R4 reports decoding latency and target KV cache size.

Table R3. Qwen3-4B generation quality with DSpark and DFlash

DrafterMethodKV keep ratioMATH500HumanEvalQasperHotpotQAGovReportAvg.
BothOrigin1.082.4088.4138.9743.6828.7456.44
DSparkSparse-dLLM0.519.8054.8838.6743.8527.8637.01
DSparkOurs0.576.4081.7138.5643.5928.7053.79
DFlashSparse-dLLM0.516.8054.2738.9043.6927.9436.32
DFlashOurs0.574.8082.9338.4943.6928.8053.74

With 50% of the target KV retained, Ask-dLLM outperformed Sparse-dLLM by 16.78 points with DSpark and 17.42 points with DFlash on average. With either drafter, Ask-dLLM scored within 2.70 points of the full-KV average of 56.44. The gains ranged from 56.60 to 58.00 points on MATH500 and 26.83 to 28.66 points on HumanEval.

Decoding latency and KV cache size. We measured latency on 10 LongBench QMSum examples using a single RTX A6000. For each example, we used the full 26,112-token input context and generated 512 tokens with greedy speculative decoding. Decoding latency excludes prefill. KV cache size is measured for the target model at the end of generation.

Table R4. Decoding latency and target KV cache size with DSpark and DFlash on LongBench QMSum at 26K context

DrafterMethodKV keep ratioDecoding latency (s/request) ↓KV cache size (GiB) ↓
DSparkOrigin1.024.413.657
DSparkSparse-dLLM0.513.551.830
DSparkOurs0.515.771.830
DFlashOrigin1.023.613.656
DFlashSparse-dLLM0.516.261.829
DFlashOurs0.516.491.829

Permanently evicting half of the target KV entries required no additional full-sequence forward pass and approximately halved the target KV cache size at the end of generation. Compared with full-KV decoding, latency decreased by 35.4% with DSpark and 30.2% with DFlash.

AuthorsOpenReview ↗

General Response to All Reviewers (3)

Q1 . How long does it take to decode one block at different context lengths, and how much does each operation contribute to the total latency?

Thank you for asking for a per-block latency breakdown. Table R5 (Table 10 in Appendix C.4 of the revised manuscript) reports separate profiling measurements for LLaDA-8B-Instruct on a subset of LongBench GovReport at 2K and 8K context lengths using a single RTX A6000. Figure 3 in Section 4.3 (p. 8) reports aggregate throughput and peak GPU memory across four datasets. The GovReport panel in Figure 5 (Appendix C.3) uses the full GovReport evaluation set.

Table R5 breaks down per-block latency into full-sequence forward passes, KV selection, denoising, and sampling. Each profiled request generated 512 tokens in 16 blocks of 32 tokens, and both eviction methods used a KV keep ratio of 0.1.

Table R5. Per-block decoding latency breakdown and per-request decoding latency at 2K and 8K context

Context lengthMethodFull-sequence forwards (count × ms/pass)KV selection latency (ms/block)Denoising latency (ms/block)Sampling latency (ms/block)Total latency (ms/block)Decoding latency (s/request) ↓
2KOurs2 × 38829.0937.215.11757.228.11
2KSparse-dLLM2 × 3929.2931.315.11740.127.84
2KFast-dLLM1 × 387–1973.867.12427.738.84
2KdLLM-Cache0.375 × 371–3599.361.63800.060.80
8KOurs2 × 1727102.7983.937.24577.373.24
8KSparse-dLLM2 × 172630.1985.037.14503.672.06
8KFast-dLLM1 × 1721–4236.0166.56123.597.98
8KdLLM-Cache0.375 × 1717–10046.980.610771.3172.34

Forward-pass counts are averages per block. Per-pass times are rounded. Per-request decoding latency covers all 16 blocks and is approximately 16 times the average per-block latency. KV selection accounted for 1.7% of Ask-dLLM’s per-block latency at 2K and 2.2% at 8K. At the same KV budget, Ask-dLLM’s per-block latency was 1.0% higher than Sparse-dLLM’s at 2K and 1.6% higher at 8K. Full-sequence forward passes accounted for the largest increase in Ask-dLLM’s latency as the context grew from 2K to 8K. At 8K, Ask-dLLM’s per-request decoding latency was 73.24 s, 25.3% lower than Fast-dLLM’s and 57.5% lower than dLLM-Cache’s.

Additional on-device latency evaluation. We evaluated LLaDA-8B-Instruct and Dream-v0-Instruct-7B on an NVIDIA Jetson Orin Nano [3] with 8 GB of shared CPU/GPU memory. We used TensorRT 10.4 with INT4 weight-only engines and a batch size of 1, generating 256 tokens from a 1,792-token GovReport prompt. Table R6 (Table 4 in Section 4.6 of the revised manuscript) reports decoding latency and KV cache size alongside the six-benchmark scores of the INT4 models. Boldface compares the two eviction methods.

Table R6. INT4 benchmark performance, decoding latency, and KV cache size on Jetson Orin Nano

BackboneMethodKV keep ratioHotpotQAQasperTRECSAMSumHumanEvalMATH500Avg.Decoding latency (s/request) ↓KV cache size (GiB) ↓
LLaDA-8B-InstructOrigin1.011.9716.1234.5630.6240.8536.2028.391150.9—
LLaDA-8B-InstructSparse-dLLM0.113.8416.7236.2531.5714.0220.0022.07136.5≈0.098
LLaDA-8B-InstructOurs0.113.2119.7647.9233.6734.7631.8030.19137.2≈0.098
Dream-v0-Instruct-7BOrigin1.032.8027.8330.0826.9939.0237.8032.421033.1—
Dream-v0-Instruct-7BSparse-dLLM0.128.4720.7526.0033.2021.3430.4026.69116.5≈0.011
Dream-v0-Instruct-7BOurs0.131.1320.8128.1731.7430.4931.4028.96117.0≈0.011

Compared with the INT4 Origin models, Ask-dLLM reduced decoding latency from 1150.9 s to 137.2 s on LLaDA and from 1033.1 s to 117.0 s on Dream, corresponding to 8.4× and 8.8× decoding speedups, respectively. Ask-dLLM scored 1.80 points above Origin on LLaDA and 3.46 points below Origin on Dream on average.

At the same 10% external-KV keep ratio, Ask-dLLM and Sparse-dLLM used the same KV cache capacity. Their decoding times differed by 0.7 s on LLaDA and 0.5 s on Dream, while Ask-dLLM improved the average benchmark scores by 8.12 points on LLaDA and 2.26 points on Dream. These measurements demonstrate substantial acceleration over Origin on an 8 GB device, while delivering higher average quality than Sparse-dLLM at the same KV cache capacity and comparable decoding latency.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Responses to All Reviewers

We thank Reviewers BWgU, 5G9a, and 727H for their constructive feedback.

We appreciate the reviewers’ recognition of the motivation and lightweight design of Ask-dLLM (BWgU), the breadth of the evaluation and the gains in accuracy and throughput (727H), and the accuracy–efficiency trade-off under aggressive KV eviction (5G9a).

Below we summarize the main revisions. Detailed point-by-point responses appear in the individual review threads.


Summary of Key Revisions

  • New backbone evaluations on SDAR-8B-Chat and LLaDA2.1-Mini under permanent KV eviction (Appendix C.6).
  • Speculative decoding experiments with Qwen3-4B using DSpark and DFlash drafters (Appendix C.7).
  • On-device evaluation on an 8 GB Jetson Orin Nano, reporting benchmark scores, decoding latency, and KV cache size for INT4 LLaDA and Dream (Section 4.6).
  • Latency profiling and ablations covering per-block decoding costs and the timing and frequency of KV selection (Appendices C.4–C.5).
  • Full-cache teacher agreement analysis across KV keep ratios, relating selection agreement to task performance (Appendix C.8).
  • Scorer training and generalization analysis, clarifying training sources and reporting separate averages for seen and unseen benchmarks (Appendices A.2 and C.9).
  • Throughput measurement details, including batch size, execution environment, and the operations included in generation time (Appendix A.1).

Dear Reviewers and Area Chair,

Thank you for taking the time to review our submission. Have our responses resolved your concerns? Only about a day is left in the discussion period, and it would mean a lot to hear your thoughts before it closes.

Following your comments, the revision now includes evaluations on the recent backbones SDAR-8B-Chat and LLaDA2.1-Mini, an extension to speculative decoding using Qwen3-4B with the DSpark and DFlash drafters, and an on-device evaluation on an 8 GB Jetson Orin Nano. It also adds a latency analysis, clarifies how the training and evaluation data relate, and details the speed measurement settings.

We fully understand that you are handling multiple submissions and have a very busy schedule. Still, if anything remains unresolved, please let us know and we will respond right away, before the discussion closes.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.