- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when KV cache compression changes the dense model’s output under greedy decoding. The authors propose the Output Crossing Predictor (OCP), which uses a first-order approximation to predict whether a candidate compression will change the next token, and apply it to estimate the first divergence time and to guide the compression level of existing methods.
Critical strengths
- The paper clearly characterizes the condition for output preservation under KV cache compression as the ratio of compression pressure to the decision margin.
- The authors demonstrate that the proposed framework is applicable across multiple models and compression methods.
Critical weaknesses
- I have concerns about the core premise of this paper. Should a good KV cache compression method necessarily reproduce exactly the same output tokens as the uncompressed model? A compressed model could reach the same correct answer through a different, or even more efficient, reasoning trajectory.
- The authors control the dense outputs to be deterministic by using a batch size of 1 on a single GPU. However, most practical serving systems use batching, and even under greedy decoding, floating-point results can vary with batch composition and kernel reduction order. In this case, divergence between dense outputs can occur at steps with small margins even without any compression. How can divergence caused by compression be distinguished from divergence caused by such numerical nondeterminism?
- The paper defines and analyzes output crossing under the assumption of greedy decoding. However, recent models such as Qwen3 discourage greedy decoding in their model cards due to repetition and endless loops, and instead recommend sampling (e.g., T=0.6). Under sampling, one should examine changes in the output distribution rather than whether a single output token matches. Can the proposed framework be extended to this setting?
- The models used in this paper (Qwen2.5, Llama-3.1, and Mistral) are not reasoning models. An evaluation on long reasoning scenarios, where the demand for KV cache compression is greatest, is needed.
Questions for authors
Please see the weaknesses above.