- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This is a measurement study of how a key-cache compression budget should be spent when three actions are available: quantize every token, evict tokens, or mix zero-rate (eviction) and finite-precision tiers across tokens. The paper does not propose a better compressor. It builds SIEVE, a per-KV-head, per-token allocator that is optimal under a local first-order attention-output-distortion surrogate, with eviction as the zero-rate tier and values kept exact, and uses it as an instrument to ask three questions: when does the mixed interior beat both endpoints under the distortion objective, does the answer transfer to end-task accuracy, and does the proxy behind such allocators predict what users observe.
Critical strengths
- The truncation analysis (F6) explains why token-separable allocation fails on multi-token answers, since a kept token protects only itself and a multi-token needle value is then destroyed, shows that the eviction methods avoiding this all smooth scores over neighboring positions, and demonstrates that pooling the allocator's scores recovers most of the gap (interior mean 0.458 to 0.756 on Llama, past OBCache-K + Ada-KV). F5 adds the counterexample that makes the proxy failure vivid, TurboQuant's high-error/high-accuracy profile. Between them these give the field a diagnosis and two corrected rules of thumb, allocate spans rather than tokens, and use distortion for budget sizing but not for method choice.
- The study quantifies how much each protocol choice moves a verdict (Table 3): the eviction corner (5.1-8.6 band points), the information contract (the STOP crossing moves from 46.9% to 57.4%), question visibility at compression time (SnapKV goes from 0.11-0.65 to 1.00), and the choice of dense baseline (0.87 to 0.99 on Qwen), against a prompt-sampling standard deviation of 0.4-3.0 points that changed no verdict. F7 is directly actionable for anyone benchmarking cache compression, and the paired-rerun design with a measured noise threshold is something I would like to see other papers in this area copy. The disclosure apparatus (pre-registration, a detailed AI-use statement, spread estimation) is well above community norms.
Critical weaknesses
-
The generality of the headline recommendations exceeds the measured scope. Implication 1, report dense low-bit quantization next to any eviction or mixed method, rests on four retrieval-style tasks, three models, 2-3 key code bits, values uncompressed, and the workload family where dense quantization is known in advance to be strongest. Eviction methods are generally expected to do relatively better on generative or aggregative workloads, summarization, long-document QA beyond needle lookup, reasoning traces, and none of these is measured. The attempt to widen the task base, LongBench-v2, was abandoned when it could not separate policies at these budgets. The paper scopes its claims honestly (Table 1, Section 8), but the conclusion most readers will carry away is broader than the evidence. The recommendation would be more defensible if stated as conditional on retrieval-style evaluation, or supported by one non-retrieval workload at budgets where policies differ.
-
Accuracy-only measurement of an accuracy-motivated problem, with an instrument of unknown deployability. The paper motivates the problem by batch size and decode throughput, but reports no throughput, latency, or memory measurements, and its own appendix notes the eager attention path means accuracy, not cost, is measured. SIEVE needs per-token weights a_i^2 * ||v_i - o||^2, per-KV-head routing, re-budgeting every few decode steps (F3), and cascade re-scoring. Table 5 prices these as regret multiples, not in compute or memory, so a reader cannot judge whether the router's partial accuracy advantages survive any realistic serving constraint. Since dense quantization already wins on all three end-task models, this gap does not change the paper's central negative verdict. It does leave Implication 5, re-budget often, unpriced.
Questions for authors
-
Has the pre-registered decisive test (positional pooling at Llama-3.1-8B, 128k, B=2) now been run? If so, what is the result? If not, why does the paper still call it "the decisive test" while reporting only the 8k-32k evidence?
-
Do the F4/F5 verdicts hold on any non-retrieval workload at budgets where policies differ, for example a larger LongBench-v2 sample or a summarization task? If measurement cost makes this infeasible, please state in Implication 1 that the dense-beats-all recommendation is conditional on retrieval-style evaluation.
-
How do the conclusions change when values are also compressed, say 2-bit KIVI-style value quantization, which is the practical regime? Does the mixed interior become more competitive when both K and V are lossy, or does its advantage remain confined to the key-only setting?
-
Given the 0.15 tie threshold and single full-grid runs, how stable are SIEVE's wins over OBCache-K + Ada-KV (+0.163) and over the best per-cell eviction method (+0.127)? Would 3-5 reruns of these specific pairs, or 100 prompts on a subset of cells, move these means across the threshold in either direction?
-
What is the measured serving overhead of SIEVE (weight computation, routing, re-budgeting, cascade scoring) relative to TurboQuant at matched accuracy? Even one end-to-end throughput measurement on a single cell would calibrate whether the router is deployable and whether Implication 5 is affordable.
-
H2O is implemented with raw accumulated attention, no pooling, and no recent-token budget beyond the 32-token window shared by all methods (Table 7), and scores 0.35, well below commonly reported behavior. Does giving H2O its standard sink plus recent-window configuration change any F4 verdict?