- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents an adaptation of WUSH's data-adaptive transforms to low-bit KV-cache quantization. It constructs separate key and value transforms from calibration-time second-order statistics, folds the value transform into the model weights for inference efficiency, and provides a near-optimality analysis for the WUSH transform. Experiments show reduced reconstruction error and perplexity, which extends to competitive or better 2-bit performance than OSCAR (sota) across the evaluated Qwen3 models and downstream tasks.
Critical strengths
- WUSH-KV substantially reduces reconstruction error, and these gains translate into strong 2-bit perplexity and downstream accuracy across multiple model sizes and tasks.
- The method is well motivated theoretically: it uses second-order statistics from both sides of the attention products, and the near-optimality result provides a principled justification of the transform.
- The evaluation is well structured, progressing from controlled layer-wise reconstruction studies to perplexity and end-to-end downstream benchmarks.
Critical weaknesses
-
The main methodological contribution is the adaptation of WUSH to the attention setting, particularly through attention-specific sensitivity matrices (i.e., query statistics for keys and output-projection statistics for values) and transform placement around RoPE and normalization. However, the underlying transform construction remains essentially unchanged from WUSH. Thus, WUSH-KV can be viewed as a relatively direct extension of WUSH from weight-activation quantization to the KV cache, somewhat limiting the degree of methodological novelty.
-
The evaluation would be more convincing if it included different model families, since accuracy was evaluated only on different sizes of Qwen3 models. i.e. Does this method extend to different model families?
-
A latency/throughput evaluation is missing. While KV-cache reduction should provide theoretical bandwidth savings in long-context, memory-bound settings, it is unclear how much overhead (e.g. applying K rotation on-the-fly) or even benefits (e.g. folding V rotations into weights) the methodology introduces. Especially, the latency/throughput comparison against BF16 and other 2-bit methods are needed.
Questions for authors
No specific questions other than what i described in the weakness section.
2 comments
Please respond to the rebuttal
Dear Reviewer D9xd, the authors have posted their response. Please add a brief follow-up indicating whether your concerns have been addressed and noting any remaining questions, if you haven't done so. Thank you!
Thank you for your time and dedication to reviewing this submission. We address your concerns below.
W1: Novelty.
Our paper introduces a new mathematical formulation, theoretical guarantees, architectural analysis, a KV-cache-specific calibration procedure, and practical contributions. The novelty lies in extending the WUSH weight-activation quantization framework to a fundamentally different KV cache quantization problem and solving the resulting mathematical and architectural challenges.
1. New mathematical formulation. WUSH was originally developed for two-sided weight-activation quantization, whereas KV-cache quantization is one-sided. To bridge this gap, we develop a new Gram/Hessian formulation (Definition 1), which requires identifying the appropriate sensitivity matrices for keys and values under grouped-query attention. We derive these matrices from the query vectors and output-projection weights, respectively (Eq. 4). We further introduce a normalization-invariant formulation (Eq. 1) that preserves the geometry of these statistics regardless of their scaling.
2. New theoretical guarantee. The original WUSH analysis only covers AbsMax-scaled non-clipped quantizers (optimal for FP, asymptotically optimal for INT). We extend its optimality analysis to an RMS-scaled, clipped, integer quantizer (QuEST). This is the first optimality argument for such transforms over a clipped quantizer.
3. Novel transform placement analysis. Appendix A shows that the only key transforms that can be folded into fixed projection and normalization weights are paired sign flips, which do not redistribute coordinate magnitudes. This is why the full key transforms are applied online after RoPE, while the value transforms are folded into the weights.
4. KV-cache calibration procedure. We design a forward-only calibration pipeline (Algorithm 1) to efficiently gather the required second-order statistics under grouped-query attention. We also derive exact attention-aware Hessians that require only forward-pass quantities (Appendix C).
5. Practical contribution. To the best of our knowledge, our accuracy results become the new state-of-the-art for 2-bit KV cache quantization. We also integrate our method into SGLang for efficient decoding.
W2: Other model families.
To show the downstream benefits on other model families, we add experiments on Granite-4.2-8B, which is a newer model from IBM that includes native step-by-step reasoning capabilities. Older models like Llama-3 perform substantially worse in our benchmarks even in full precision, making it difficult to distinguish the effects of different transforms at 2-bit precision.
On Granite-4.2-8B, WUSH-KV outperforms OSCAR under the same setups. Below are the tables for the scores and average output tokens per request. We have also added these numbers to Table 2 in the updated manuscript (marked in blue font).
Scores:
Output Tokens:
We are also extending our SGLang integration to support more model families.
W3: Throughput.
We measure Qwen3-8B decoding throughput (tokens/s) on an NVIDIA B300 GPU. The results are shown in the table below. We have also added this table to the updated manuscript as Table 3 (marked in blue font). At matched context lengths and batch sizes, WUSH-KV achieves approximately 1.3-2.2× higher decode throughput than BF16. WUSH-KV preserves the inference efficiency of OSCAR while improving accuracy.
WUSH-KV also generates fewer tokens than OSCAR per request, improving per-request latency. Table 2 shows the mean token count for each task on each model. On Qwen3-8B, WUSH-KV uses 11-27% fewer tokens.