- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies runtime certificates that decide whether a quantized attention output satisfies , falling back to the original FP16 cache otherwise. Arguing that TV-based bounds are conservative because they ignore value structure, the authors derive an exact scalar output envelope (Theorem 2) under joint range, mean, and variance constraints, attained by two-point laws. They also give attainable TV envelopes under partial probability information, with saturation and stopping criteria. Validation is an offline replay of 1,008 case/budget pairs from GPT-2, where Joint certifies 357 pairs versus 249 for centered TV and 352 for coordinatewise Mix.
Critical strengths
The joint envelope in Theorem 2 is the exact optimum within the stated information class: every regime is attained by a two-point law, so no tighter bound is possible from the same range/mean/variance summary.
The certificate is numerically sound, using directed rounding and exact rational comparisons, and every certified pair passes the independent enclosure of actual original-FP16 output error.
Critical weaknesses
The practical need for the proposed certificate is not demonstrated. The key contribution, joint constraints, certifies only five more pairs than coordinatewise Mix out of 1,008, all at , i.e., tolerances of and , whereas the output error is already trivially bounded by roughly since both outputs are convex combinations of values. None of the evaluated methods certifies any INT4-key case at , and the paper does not relate to output quality, so it is unclear what practical benefit the tighter bound delivers.
Second, comparisons and evaluation scope are insufficient. There is no direct comparison against prior certificates for the same problem (Calver, 2026; Wei et al., 2026), and the evaluation is limited to an offline replay on GPT-2 with a 512-token prefix. Since Calver (2026) already evaluates on LLaMA 3.1-8B with contexts up to 128K, evaluation on modern models and KV quantization methods is needed.
Questions for authors
-
Could you show that modern KV cache quantization methods (e.g., [1], [2]) actually produce large per-head errors, and that such errors change the final model output?
-
How should be chosen in deployment? Please justify the grid and explain how relates to output quality such as perplexity.
-
How does Joint compare against the bounds used by Calver (2026) and Wei et al. (2026) under identical settings?
-
Why is the evaluation limited to GPT-2 with a 512-token prefix? Does the gain of Joint persist on modern models and longer contexts?
-
Since the original FP16 cache and a packed mirror are both kept on the CPU, total memory exceeds plain FP16. What is the advantage over plain KV offloading, and what is the fallback rate during actual generation?
References:
[1] Liu, Zirui, et al. "Kivi: A tuning-free asymmetric 2bit quantization for kv cache." arXiv preprint arXiv:2402.02750 (2024).
[2] Hooper, Coleman, et al. "Kvquant: Towards 10 million context length llm inference with kv cache quantization." Advances in Neural Information Processing Systems 37 (2024): 1270-1303.
3 comments
Thank you for the response. My main concerns (practical need, meaning of , and evaluation scope) remain unaddressed, so I maintain my score.
New experiments on modern models: calibrated tolerances and KV-group admissions
The new experiments show that the stronger bound changes complete KV-group decisions on modern models at a fixed, separately calibrated tolerance. On Qwen3.5-2B with K8/V8 and rho=0.05, Joint admits 25/768 additional complete KV groups at a 4,096-token prefix and 38/768 at an 8,160-token prefix over coordinatewise Mix: gains of 3.26 and 4.95 percentage points. Both bounds are evaluated on exactly the same states, summaries, and error allowances along the Joint trajectory. Because every query head sharing a KV group must pass before the group can avoid retrieving its original KV, these are changes in the operational group-level decision.
We have also broadened the evaluation to SmolLM2-360M, Qwen3-0.6B, and Qwen3.5 at 0.8B and 2B. The local comparison covers 48 model/context/precision settings: four prefix lengths (512, 2,048, 4,096, and 8,160), three formats (K4/V4, K8/V4, and K8/V8), and ten tolerance ratios, using six common held-out documents and the first, middle, and final full-attention layers. We followed the 2B/K8V8 operating point identified by this matrix with closed-loop model evaluation on four held-out documents and 16 teacher-forced tokens per prefix. At 4K/approximately 8K, Joint gives mean delta NLL of -0.000147/+0.000254 nat/token and mean KL of 2.74e-5/2.71e-5 relative to matched dense attention. The additional group admissions above are measured in these closed-loop runs.
To make epsilon operational, we use epsilon=rho R0: an absolute infinity-norm error budget for one head output relative to the same query and its original FP16 KV, where R0 bounds the original value-coordinate magnitudes. We select rho against model-quality criteria on calibration data, then test it on held-out text. In the Qwen3.5-0.8B study, four calibration documents selected rho=0.05 using mean delta NLL<=0.01 nat/token, mean KL<=0.01, and delta NLL<=0.05 for every document/context. On eight held-out documents with 32-token continuations, Joint+fallback gives mean delta NLL +0.00332/-0.00227 at 512/4K, compared with +0.01518/+0.01567 for unconditional K8/V4 quantization. We transferred the selected rho to the expanded study without retuning it on the new models. Thus, epsilon has a precise local meaning and an explicit calibration procedure for a downstream quality target; it is not itself a perplexity percentage.
The practical role is to decide when the compressed attention output is usable and when the original KV is needed. Joint and coordinatewise Mix already share the same moments and correction terms, so replacing Mix with Joint requires the stronger scalar-envelope evaluation, without another original-KV transfer or moment-construction pass. The new group-level results quantify what this analytical increment changes. We will include the calibration protocol, full precision/context matrix, and closed-loop measurements in the main-text evaluation and discussion.
The model runs use nominal FP64 attention with FP16 inputs/outputs and dense prefill; Qwen3.5 interventions cover its six softmax-attention layers. For the 2B/K8V8 layout, the original host KV plus packed mirror occupies about 1.54 times the original KV storage, excluding temporary workspace. We report these costs explicitly and treat the results as model-quality and gate-decision evidence, with physical transfer and serving latency requiring separate measurement. The new evidence directly addresses the meaning of epsilon, extends the evaluation beyond the original GPT-2 setting, and demonstrates additional usable group decisions. We would appreciate reconsideration of the rating in light of these results.
Reviewer Zuo7
The five additional admissions over coordinatewise Mix are a small gain in this replay. The theoretical result addresses what can be certified from the stated information: Theorem 2 establishes the optimal attainable scalar output bound for the given range, mean, variance and likelihood-ratio information. The gain comes from enforcing these constraints at the same event mass. For (a,b,v,κ)=(1,25,16,9), the joint bound is 128/25=5.12 and passes budget 26/5=5.2, while variance alone gives 16/3 and range/mean alone gives 100/17. A two-point law attains the joint value. Figure 2(a) therefore establishes a strict change in the budget decision, together with optimality within the information class.
The replay measures how often that additional constraint changes decisions in the captured data. Across 144 head/format cases and seven budgets, Joint certifies 357 pairs, centered TV 249, and coordinatewise Mix 352. The five additional admissions over Mix are all INT4 cases, two at ρ=.5 and three at ρ=1. Table 2 gives their margins, including remaining budget 2.777830, Joint 2.522542 and Mix 2.964915. At ρ≤.25, no INT4 case passes any evaluated method.
For the comparison with Calver and Wei, fixing the logit-error convention is essential. Under ∣δi∣≤Δ, Equation (5) gives the sharp uniform TV bound tanh(Δ/2), attained by two tokens. Calver v1 Equation (9) gives tanh(Δ) under the same convention. Wei's deterministic bound uses per-token errors and reconstructed probabilities, which can tighten the result when large error bounds occur on low-probability tokens. Theorem 2 additionally exploits value range and variance jointly. Calver · Wei et al.
We completed an offline formula comparison on the saved cases using a common per-token L2 residual witness computed from the original FP16 trace. Reconstructed probabilities, value-coordinate widths and value/arithmetic allowances are identical across formulas. Ordinary FP64 evaluation gives 293 nominal passes for sharp uniform, 215 for Calver's uniform formula and 216 for Wei's deterministic formula over the same 1,008 budgets. These are nominal FP64 decisions using the additional residual witness. The original Joint count of 357 uses the paper's existing ratio bounds and additional value moments.
The grid ε=ρR0 normalizes tolerances by an upper bound on the original value magnitude and provides a common scale for comparing certificate decisions. A deployment can select ρ against its model-quality criterion, but we have not measured its relationship to model quality. Our fixed 512-token GPT-2 capture supplies common inputs for exact comparisons and independent numerical validation; it does not establish whether the gain persists on modern models or longer contexts.
Keeping the original FP16 cache and packed mirror increases CPU storage. Each successful certificate allows the quantized result to be used directly and avoids transferring the original head to the GPU. The benefit therefore depends on certificate cost, transfer cost and fallback frequency, which we have not measured during actual generation.