- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Paper Summary
In one sentence: a common way to compress a language model's memory is to give extra bits to the parts that hurt most when compressed one at a time; this paper tests whether that works and finds it only holds in some settings.
Problem:
A language model keeps a memory of past tokens (the KV cache), and for long texts this memory is large. Storing it with fewer bits saves space but adds error. Some parts of the memory are more sensitive than others, so a common approach gives those parts more bits. To find them, it compresses one part at a time and checks how much worse the model gets.
Goal:
Check whether testing one part at a time still picks the right parts when the whole memory is compressed together.
Method:
The paper does not propose a new method; it runs tests on three models (Qwen3-0.6B, Qwen3-1.7B, Mistral-7B) using long books, always at the same total memory. It asks three questions: Is the damage from compressing everything at once equal to the sum of the damages from compressing each part alone? Does it matter more to give extra bits to keys or to values, and does the answer change with how the compression range is set? Does testing each part beat a simple rule like "give extra bits to the first layers"?
Results:
- With 4 bits, the damages roughly add up (shown on Qwen3-1.7B). With 2 bits, compressing everything at once does about 3 times more damage than the sum of the parts.
- Whether keys or values need extra bits is not fixed; the answer changes with how the compression range is set.
- Testing each part sometimes beats the simple rule and sometimes does not, depending on the model and the memory budget.
Difference between this paper and the closest baseline:
The closest baseline is KVTuner, from "KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference" (Li et al., 2025, arXiv:2502.04420). KVTuner already gives extra bits to the parts that need them most. This paper adds no new method; it adds measurements of when testing parts one at a time can be trusted.
Critical strengths
Critical Strengths
S1: The paper shows clearly that testing parts one at a time can be misleading at low bits. With 2 bits, compressing everything at once does about 3 times more damage than the sum of the parts, on both Qwen3-1.7B and Mistral-7B.
Why it matters: It tells practitioners to test the whole compressed memory, not just add up per-part scores.
Critical weaknesses
Critical Weaknesses
W1: The "scale timing" claim in the title is not supported. The paper says that whether keys or values need more bits depends on when the compression range is set: from a whole block of tokens (including later ones), or only from tokens already written. But the paper's own setup closest to real use, which also uses only written tokens, does not show this flip. The paper itself admits that "future reading alone does not explain it" and that it has not separated timing from other changes in how the range is set.
Why it matters: The title names scale timing as a main finding, but the evidence does not show that timing is the cause.
W2: KVTuner already does this, and goes further. KVTuner already gives keys and values in each layer their own bits based on measured sensitivity, picks the setting by testing the whole compressed model, and notes that the compression setting changes whether keys or values matter more. The paper says this itself in its related work. KVTuner also tests on math reasoning tasks and reports real speedups (up to 21% higher throughput than 8-bit KIVI), while this paper only simulates compression and never runs KVTuner itself; it compares only to its own "KVTuner-inspired" search.
Why it matters: The paper's only addition is measurements, and without a direct comparison it is unclear whether they would change what KVTuner picks.
W3: No standard KV-cache benchmarks. All tests use loss on PG-19 books and a retrieval test built from the same books. The paper does not report results on benchmarks commonly used for KV-cache compression, such as LongBench, RULER, or Needle-in-a-Haystack, nor on math reasoning tasks like those KVTuner uses.
Why it matters: Without these, it is unknown whether the loss differences matter for real tasks.
Questions for authors
Questions For Authors
See weaknesses.