acceptodds
Under review as a conference paper at ICLR 2027

Step-Wise Endorsement: Evaluating Approximate KV Caching in Diffusion LLMs One Decision at a Time

Abstract

Most diffusion large language models (dLLMs) use bidirectional attention, so an exact KV cache is impossible: every newly decoded token changes the keys and values (KV) of all other tokens. Approximate KV caching methods such as Fast-dLLM reuse KV from an earlier step and recompute (refresh) them only occasionally. Such a method is good to the extent that the cached model approximates cache-free exact decoding, a property no metric measures directly. The refresh interval, however, orders these caches in advance: refreshing more often leaves less of the decoding on stale KV. Varying only this interval gives a refresh-dose scale of caches whose order is known, which any sound evaluation must follow. Accuracy, the evaluation used most, does not: in two of four configurations of Fast-dLLM, and on HumanEval and MBPP, it calls the cache that never refreshes the better one. Comparing the cached and the exact output token by token barely separates the settings. We propose step-wise endorsement: before each step of the cached model, the exact model receives the same state and decides there itself, and four metrics record how often and how far the two disagree on which [MASK] to decode and which token to write. All four follow the scale in every configuration on math, at both ends of it on code, and along the refresh knobs of dLLM-Cache, dKV-Cache, and Elastic-Cache; with a tenth of the questions, they order the settings far more reliably than accuracy does with all of them. They also show why Fast-dLLM works: stale KV change decisions, but rarely, and mostly where the exact model itself is uncertain.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion LLMs, Approximate KV Caching, Step-wise Endorsement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_WS1UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article studies the evaluation problem of approximate KV cache in diffusion language models. The author points out that due to the bidirectional attention mechanism commonly used in dLLM, reusing KV cache from previous time steps can introduce approximation errors, and existing work mainly relies on the accuracy of the final task to measure the quality of caching methods, making it difficult to directly reflect the impact of caching on model inference decisions. Overall, the problem studied in this article has practical significance, and the experiment also demonstrates the sensitivity of the proposed indicators to cache refresh strategies.

Critical strengths

The research question has certain value. The evaluation method has a certain interpretability.

Critical weaknesses
  1. Lack of strict criteria for core evaluation premise: Does more frequent KV refresh necessarily mean better caching? This article takes the refresh sequence scale composed of different KV cache refresh frequencies as the reference order for evaluation indicators, and believes that the more frequent the refresh, the closer the cache is to accurate decoding without cache. Therefore, a reliable evaluation indicator should follow this order. However, I believe that this inference is not strictly valid. Although more frequent refreshes can reduce the time for outdated KV to be reused, this does not necessarily mean that the differences between the model output distribution, local decoding decisions, or fully generated trajectories and the accurate model will monotonically decrease.
  2. The position consistency rate and token recognition rate proposed in this article are essentially comparing whether the discrete decisions of two models are consistent under the same input state. The positional deviation and token deviation measure the probability difference between the decision selected by the caching model and the optimal decision of the exact model. These indicators have good intuitive explanations, but their basic ideas are closer to existing teacher student prediction agreements, top-k prediction consistency, and decision evaluations based on probability differences.
  3. The core operation of Step Wise Endorsement is to make an independent prediction of the exact model based on the actual state reached at each step of the caching model, and determine whether the current caching decision is recognized.This design can avoid the error accumulation problem caused by directly comparing two different generated trajectories, but it also brings an important limitation:This indicator measures the local decision consistency on the current trajectory of the cached model, rather than necessarily the global consistency between the cached model and the original precise decoding trajectory.
  4. This article found through bootstrap resampling that the preset refresh order can be restored more stably with fewer issues, which is a valuable result. However, the current experiment mainly resamples on a fixed 300 questions, which cannot fully demonstrate that the indicators have the same sample efficiency for different problem distributions.
Questions for authors
  1. Why can we assume in advance that a cache with a higher refresh rate is necessarily closer to accurate decoding? Can the author provide strict theoretical conditions or directly analyze whether the actual prediction error has monotonicity under different refresh settings? What are the unique advantages of the four proposed indicators compared to KL divergence, Jensen Shannon divergence, total variation distance, or top-k consistency in the same state? Can these baselines also stably restore the refresh sequence scale?
  2. If the caching model has deviated from the precise decoding trajectory in earlier steps, can higher endorsement rates still accurately characterize the fidelity of the entire generation process? Can you provide further travel level analysis?
  3. How should the proposed metrics be defined when the cache model and the exact model select different numbers of MASK positions at a certain step, or when using random sampling? Can these indicators be extended to the original dynamic submission strategy of Fast dLLM?
  4. If the number of evaluation steps is fixed under different cache settings or only the common decoding state is compared, instead of separately counting the first EOS, does the main experimental conclusion still hold?
Reviewer_XH2bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that approximate KV caches for diffusion LLMs should be judged by how closely they reproduce exact, cache-free decoding, not by task accuracy. To test any evaluation metric, it builds a “refresh-dose scale”, a settings of a single refresh knob whose order of quality is known in advance. On this scale, accuracy and token-by-token output agreement fail to rank the settings correctly. It proposes step-wise endorsement as a fix -- at every step, the exact model is run on the cached model’s current state, and four metrics record how often and by how much it disagrees with the cached model’s choice of position and token. All four metrics follow the scale for Fast-dLLM and for dLLM-Cache, dKV-Cache and Elastic-Cache, using only a few dozen questions.

Critical strengths

S1. The refresh-dose scale seems like a nice way to check whether an evaluation metric works, because the correct order is fixed by the refresh operation before any metric is read. The papers seems well motivated as well. The scoring-rule reversal shows properly how fragile accuracy-based comparisons of caches are.

S2. The proposed method is simple and does not depend on how a cache works.

S3. It also explains clearly why Fast-dLLM degrades so little under NR. It also shows that KV similarity measured between adjacent steps understates long-range drift.

Critical weaknesses

W1. The validation is close to tautological. A refresh makes the cached KV equal to the exact KV, so any metric that compares against exact decoding at the same state should improve with more refreshes almost by construction. Beating accuracy, which measures a different goal, is therefore a weak test.

W2. Missing fidelity baselines. The fair competitors are other fidelity measures, such as per-step KL or TV against the exact model [1], teacher-forced NLL of the cached output, or set-based and semantic output agreement [2], and none of these is compared on the scale. The positional exact-match baseline used in the paper is a fragile version of output agreement.

W3. Limited practical use. Following the scale is shown only within a single knob, where the order is already known and no metric is needed. The more useful case, comparing different caching methods, is out of scope, and it is unclear how the metrics translate into loss of accuracy or usefulness.

W4. Limited scope. Fast-dLLM is evaluated with a fixed number of tokens per step instead of its released confidence threshold, and only with greedy decoding. The models are 7–8B with short responses, and the code tasks cover only the two ends of the scale.


References

  1. Dutta, Abhinav, et al. “Accuracy Is Not All You Need.” NeurIPS (2024). arXiv:2407.09141.

  2. Dou, Yaoxuan, and Yang Shu. “Faster but Different: Diagnosing and Controlling Content Drift in Accelerated Multimodal Diffusion Language Models.” arXiv:2607.29079 (2026).

Questions for authors

Please see the weaknesses. In particular:

  1. Do simpler same-state measures such as per-step KL or TV, or the teacher-forced NLL, follow the scale with equally few questions? If so, what do the four metrics add?

  2. Does set-based or semantic output agreement (e.g., Jaccard or BERTScore) follow the scale in your configurations?

  3. Do the results hold under Fast-dLLM’s confidence-threshold decoding and under sampling?

  4. Can the metrics be used to rank different caching methods at matched compute?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.