- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the KV-cache eviction problem in full dLLMs like LLaDA and Dream. It learns per-head cache rankings from frozen denoised-state in backbone, and predicts eviction once per block without lookahead. Experiments show that it achieves notable gain over Sparse-dLLM across multiple benchmarks, while delivers great throughput speedup.
Critical strengths
- The method follows directly from well discovered observation in Figure-1
- The experiments are extensive on 23 diverse benchmarks.
Critical weaknesses
- teacher targets and scorers must be re-extracted/retrained per backbone, there are gap between training and inference. What the impact of this gap is not studied.
- In Table-2, the proposed method still get notable worse results than the original Dream, which shows the method is not that practical.
- The training and testing protocol is not very described, especially what is the difference between training set and benchmark set.
- What is the setting of the speedup comparison, batch-size, inference-engine, etc.
Questions for authors
- To my knowledge, LLaDA and Dreams are models more than 20 months older. They do not reflect the applicable trends in dLLM, so the problem itself is far from Amazing to me. What if the method applies to newer models like LLaDA 2.x, SDAR and Diffusion-Gemma.
5 comments
General Response to All Reviewers (2)
W3 The reviewer asks for more detail on the data used to train the scorer and how it differs from the evaluation benchmarks.
Thank you for pointing out that the relationship between training and evaluation data could be clearer. The main text refers readers to Appendix A.2 (p. 14, lines 749–755) and Table 6 (p. 15) for the composition of the training pool. The pool contains 1,621 prompts from five sources: MATH (500), MBPP (371), GovReport (150), MultiNews (100), and MuSiQue (500). We used the same composition for both backbones. The prompts are disjoint from the evaluation splits listed in Table 5 (p. 14).
To assess how performance depends on the training sources, we recomputed the ρ = 0.1 averages from Tables 1–2 for two groups: five seen benchmarks that share a dataset source with the training prompts and 18 unseen benchmarks from sources absent from training. Here, seen refers to the dataset source, with evaluation prompts held out.
Table R2. Average scores on seen and unseen benchmarks at a 0.1 KV keep ratio.
| Backbone | Method | Seen (5 benchmarks) | Unseen (18 benchmarks) |
|---|---|---|---|
| LLaDA | Origin | 27.56 | 40.41 |
| LLaDA | Sparse-dLLM | 16.62 | 34.54 |
| LLaDA | Ours | 25.79 | 41.89 |
| Dream | Origin | 31.00 | 45.11 |
| Dream | Sparse-dLLM | 27.12 | 41.95 |
| Dream | Ours | 29.93 | 44.23 |
On unseen benchmarks (Table R2), Ask-dLLM scores 1.48 points above Origin on LLaDA and 0.88 points below Origin on Dream. The corresponding differences on seen benchmarks are −1.76 and −1.07 points, respectively. Ask-dLLM thus remains close to Origin on benchmarks from sources absent from training. On LLaDA, its largest gains over Sparse-dLLM occur on GSM8K (+43.22 points) and HumanEval (+26.22 points), both of which are unseen.
In the revised manuscript, Tables 1–2 label each benchmark as seen or unseen, and Table 16 (Appendix C.9) reports the group means shown in Table R2.
W4 The reviewer asks for the experimental settings behind the speedup comparison, including the batch size, inference engine, and timing scope.
Thank you for asking for the details needed to reproduce the speed comparison. We released our implementation in the anonymous repository linked on p. 1, lines 28–29. Section 4.1 (p. 6, lines 311–315) describes the hardware and decoding settings.
We measured throughput and peak GPU memory in Figures 3, 5, and 6 on a single NVIDIA RTX A6000 (48 GB) with a batch size of 1. We used the official implementations of Sparse-dLLM, dLLM-Cache, and Fast-dLLM. All methods, including Origin, ran in the same PyTorch 2.5.1 environment with bf16 precision and without a separate serving engine such as vLLM or SGLang. Within each task, we used the same prompts, seeds, decoding settings, and evaluation harness across methods.
We compute throughput as the number of generated tokens divided by the total wall-clock generation time. This includes cache-refresh forward passes, Ask-dLLM’s scorer calls, and all denoising steps. The default configuration described in Section 4.1 uses a block size of 32 and sets the number of denoising steps equal to the generation length. The accuracy tables compare the two eviction methods at ρ = 0.5 and 0.1, and the speed and memory results in Figures 3, 5, and 6 use ρ = 0.1. The revised manuscript details these measurement conditions and the throughput definition in Appendix A.1 (p. 14, lines 744–747).
General Response to All Reviewers (3)
W4 The reviewer asks for the experimental settings behind the speedup comparison, including the batch size, inference engine, and timing scope.
Additional evaluation of practical deployment. To further examine the relationship between task performance and inference costs raised in your comment, we evaluated both LLaDA-8B-Instruct and Dream-v0-Instruct-7B on an NVIDIA Jetson Orin Nano [3], where the CPU and GPU share 8 GB of memory. All methods used TensorRT 10.4 with INT4 weight-only engines and a batch size of 1. Table R3 (Table 4 in Section 4.6 of the revised manuscript) reports performance across six benchmarks, together with decoding latency and KV cache capacity for generating 256 tokens from a 1,792-token GovReport prompt.
Table R3. INT4 benchmark performance, decoding latency, and KV cache size on Jetson Orin Nano. Boldface compares the two eviction methods.
| Backbone | Method | KV keep ratio | HotpotQA | Qasper | TREC | SAMSum | HumanEval | MATH500 | Avg. | Decoding latency (s/request) ↓ | KV cache size (GiB) ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| LLaDA-8B-Instruct | Origin | 1.0 | 11.97 | 16.12 | 34.56 | 30.62 | 40.85 | 36.20 | 28.39 | 1150.9 | — |
| LLaDA-8B-Instruct | Sparse-dLLM | 0.1 | 13.84 | 16.72 | 36.25 | 31.57 | 14.02 | 20.00 | 22.07 | 136.5 | ≈0.098 |
| LLaDA-8B-Instruct | Ours | 0.1 | 13.21 | 19.76 | 47.92 | 33.67 | 34.76 | 31.80 | 30.19 | 137.2 | ≈0.098 |
| Dream-v0-Instruct-7B | Origin | 1.0 | 32.80 | 27.83 | 30.08 | 26.99 | 39.02 | 37.80 | 32.42 | 1033.1 | — |
| Dream-v0-Instruct-7B | Sparse-dLLM | 0.1 | 28.47 | 20.75 | 26.00 | 33.20 | 21.34 | 30.40 | 26.69 | 116.5 | ≈0.011 |
| Dream-v0-Instruct-7B | Ours | 0.1 | 31.13 | 20.81 | 28.17 | 31.74 | 30.49 | 31.40 | 28.96 | 117.0 | ≈0.011 |
On LLaDA, Ask-dLLM reduced decoding latency from 1150.9 s to 137.2 s, achieving an 8.4× decoding speedup over the INT4 Origin model, while scoring 1.80 points above Origin on average. Compared with Sparse-dLLM, it improved the average score by 8.12 points, with a decoding-time difference of only 0.7 s at the same KV cache capacity.
These results provide concrete evidence of practical benefits under a tight memory budget: substantially faster decoding than Origin and higher average quality than Sparse-dLLM at the same KV cache capacity and comparable on-device latency.
Q1. The original evaluation focuses on LLaDA and Dream. Does Ask-dLLM remain effective on more recent diffusion language models such as LLaDA2.1, SDAR, and DiffusionGemma?
Thank you for suggesting an evaluation on more recent backbones. We chose LLaDA and Dream to match the experimental setup of Sparse-dLLM (AAAI 2026), a recent KV eviction baseline. This allowed us to compare KV selection strategies under the same backbones, cache budgets, and decoding settings. The same selection algorithm can be applied to other block-diffusion backbones that cache and reuse the KV states of finalized tokens. To test this, we evaluated Ask-dLLM on SDAR-8B-Chat [1] and LLaDA2.1-Mini [2]. Tables R4–R5 correspond to Tables 12–13 (Appendix C.6) in the revised manuscript. Given DiffusionGemma’s model size and the compute available during the rebuttal period, we will evaluate it and include the results in the final version.
Both models generate blocks sequentially and jointly denoise multiple tokens within the active block. SDAR and the single-block editing configuration of LLaDA2.1-Mini cache the KV states of finalized blocks and reuse them during subsequent decoding. These states remain fixed once a block is finalized, so we permanently evict the entries that Ask-dLLM does not select. Subsequent blocks use the retained KV states together with newly generated KV states.
General Response to All Reviewers (1)
We thank Reviewer
727Hfor the careful review and for recognizing our well-motivated approach, extensive evaluation across 23 benchmarks, and notable accuracy and throughput gains. We address each concern point by point below, providing additional analyses and clarifications.Thank you for highlighting the difference between the conditions the scorer encounters during training and inference. As described in Figure 2 (p. 3) and Sections 3.2–3.4 (pp. 4–5), the scorer uses masked-state hidden representations available at KV selection time in both settings. We obtain training labels by fully denoising each block with the full KV cache and measuring how much attention the completed block assigns to each KV entry. At inference, each block is denoised using the retained KV entries. The outputs of earlier blocks can therefore differ from those generated by the full-cache teacher, affecting KV selection in subsequent blocks.
To examine how this gap relates to generation quality, we varied the KV keep ratio and measured agreement with the full-cache teacher alongside performance relative to Origin. Table R1 presents these results, which are plotted in Figure 7 (Appendix C.8) of the revised manuscript.
Table R1. Teacher agreement and performance relative to Origin across KV keep ratios
In Table R1, both methods showed lower agreement with the teacher as more KV entries were removed, and the performance gap between them widened. When the keep ratio decreased from 0.5 to 0.1, Sparse-dLLM’s agreement fell from 69.4% to 43.1%, and its performance relative to Origin fell from 96.6% to 48.8%. Over the same range, Ask-dLLM’s agreement declined from 78.3% to 66.9%, while its performance remained close to Origin, moving from 99.4% to 97.7%.
Thank you for raising this trade-off. KV eviction reduces the number of cached entries available during decoding. Our goal is to preserve as much of the original model’s task performance as possible under a smaller KV budget.
On Dream, Table 2 (p. 7, lines 347–355) shows that even caching methods that retain all external KV entries score below the uncached Origin model (62.22). Fast-dLLM and dLLM-Cache score 60.21 and 60.91, respectively. Ask-dLLM retains only 10% of external KV entries and scores 58.50, a 3.72-point drop from Origin and 1.71 and 2.41 points below the two caching baselines, respectively. At the same 10% KV budget, Ask-dLLM outperforms Sparse-dLLM by 4.15 points on average, including gains of 12.80 points on HumanEval and 7.58 points on GSM8K.
Figure 3 (Section 4.3, p. 8, lines 401–421) shows the efficiency gains on four LongBench datasets. At 2K context, Ask-dLLM increases throughput from 3.00 to 21.19 tokens/s (7.1×) while using comparable peak GPU memory to Origin (15.65 vs. 15.85 GiB) and about 35% less than Fast-dLLM. At 8K, Ask-dLLM achieves a higher mean task score than Origin and approximately 11.8× its throughput. Fast-dLLM runs out of memory at that context length.
On Dream, Ask-dLLM retains 10% of external KV entries and achieves accuracy close to that of caching methods that retain all entries. It also substantially increases throughput at Origin-level peak memory.