- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article studies the evaluation problem of approximate KV cache in diffusion language models. The author points out that due to the bidirectional attention mechanism commonly used in dLLM, reusing KV cache from previous time steps can introduce approximation errors, and existing work mainly relies on the accuracy of the final task to measure the quality of caching methods, making it difficult to directly reflect the impact of caching on model inference decisions. Overall, the problem studied in this article has practical significance, and the experiment also demonstrates the sensitivity of the proposed indicators to cache refresh strategies.
Critical strengths
The research question has certain value. The evaluation method has a certain interpretability.
Critical weaknesses
- Lack of strict criteria for core evaluation premise: Does more frequent KV refresh necessarily mean better caching? This article takes the refresh sequence scale composed of different KV cache refresh frequencies as the reference order for evaluation indicators, and believes that the more frequent the refresh, the closer the cache is to accurate decoding without cache. Therefore, a reliable evaluation indicator should follow this order. However, I believe that this inference is not strictly valid. Although more frequent refreshes can reduce the time for outdated KV to be reused, this does not necessarily mean that the differences between the model output distribution, local decoding decisions, or fully generated trajectories and the accurate model will monotonically decrease.
- The position consistency rate and token recognition rate proposed in this article are essentially comparing whether the discrete decisions of two models are consistent under the same input state. The positional deviation and token deviation measure the probability difference between the decision selected by the caching model and the optimal decision of the exact model. These indicators have good intuitive explanations, but their basic ideas are closer to existing teacher student prediction agreements, top-k prediction consistency, and decision evaluations based on probability differences.
- The core operation of Step Wise Endorsement is to make an independent prediction of the exact model based on the actual state reached at each step of the caching model, and determine whether the current caching decision is recognized.This design can avoid the error accumulation problem caused by directly comparing two different generated trajectories, but it also brings an important limitation:This indicator measures the local decision consistency on the current trajectory of the cached model, rather than necessarily the global consistency between the cached model and the original precise decoding trajectory.
- This article found through bootstrap resampling that the preset refresh order can be restored more stably with fewer issues, which is a valuable result. However, the current experiment mainly resamples on a fixed 300 questions, which cannot fully demonstrate that the indicators have the same sample efficiency for different problem distributions.
Questions for authors
- Why can we assume in advance that a cache with a higher refresh rate is necessarily closer to accurate decoding? Can the author provide strict theoretical conditions or directly analyze whether the actual prediction error has monotonicity under different refresh settings? What are the unique advantages of the four proposed indicators compared to KL divergence, Jensen Shannon divergence, total variation distance, or top-k consistency in the same state? Can these baselines also stably restore the refresh sequence scale?
- If the caching model has deviated from the precise decoding trajectory in earlier steps, can higher endorsement rates still accurately characterize the fidelity of the entire generation process? Can you provide further travel level analysis?
- How should the proposed metrics be defined when the cache model and the exact model select different numbers of MASK positions at a certain step, or when using random sampling? Can these indicators be extended to the original dynamic submission strategy of Fast dLLM?
- If the number of evaluation steps is fixed under different cache settings or only the common decoding state is compared, instead of separately counting the first EOS, does the main experimental conclusion still hold?