acceptodds
Under review as a conference paper at ICLR 2027

WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms

Abstract

KV cache memory and bandwidth costs grow with context length and batch size, which limits efficient long-context inference. To address this bottleneck, we introduce WUSH-KV for low-bit KV-cache quantization. It adapts WUSH, which constructs a data-aware transform from second-order statistics of both factors in a matrix product to reduce quantization error. WUSH-KV uses calibration data to construct separate key and value transforms, with the value transform folded into the model weights and the key transform applied after RoPE. The transforms can be paired with clipped quantizers. For one such quantizer, QuEST INT, we show that, under mild assumptions, the WUSH transform is near-optimal. With this quantizer, WUSH-KV reduces layerwise reconstruction error and achieves the lowest end-to-end perplexity among other tested transforms. For end-to-end evaluation, we integrate WUSH-KV into SGLang using OSCAR-style percentile-clipped affine quantization. At 2-bit, it outperforms the OSCAR transform on Qwen3-8B over downstream benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache, Quantization, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_D9xdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents an adaptation of WUSH's data-adaptive transforms to low-bit KV-cache quantization. It constructs separate key and value transforms from calibration-time second-order statistics, folds the value transform into the model weights for inference efficiency, and provides a near-optimality analysis for the WUSH transform. Experiments show reduced reconstruction error and perplexity, which extends to competitive or better 2-bit performance than OSCAR (sota) across the evaluated Qwen3 models and downstream tasks.

Critical strengths
  1. WUSH-KV substantially reduces reconstruction error, and these gains translate into strong 2-bit perplexity and downstream accuracy across multiple model sizes and tasks.
  2. The method is well motivated theoretically: it uses second-order statistics from both sides of the attention products, and the near-optimality result provides a principled justification of the transform.
  3. The evaluation is well structured, progressing from controlled layer-wise reconstruction studies to perplexity and end-to-end downstream benchmarks.
Critical weaknesses
  1. The main methodological contribution is the adaptation of WUSH to the attention setting, particularly through attention-specific sensitivity matrices (i.e., query statistics for keys and output-projection statistics for values) and transform placement around RoPE and normalization. However, the underlying transform construction remains essentially unchanged from WUSH. Thus, WUSH-KV can be viewed as a relatively direct extension of WUSH from weight-activation quantization to the KV cache, somewhat limiting the degree of methodological novelty.

  2. The evaluation would be more convincing if it included different model families, since accuracy was evaluated only on different sizes of Qwen3 models. i.e. Does this method extend to different model families?

  3. A latency/throughput evaluation is missing. While KV-cache reduction should provide theoretical bandwidth savings in long-context, memory-bound settings, it is unclear how much overhead (e.g. applying K rotation on-the-fly) or even benefits (e.g. folding V rotations into weights) the methodology introduces. Especially, the latency/throughput comparison against BF16 and other 2-bit methods are needed.

Questions for authors

No specific questions other than what i described in the weakness section.

2 comments

Thank you for your time and dedication to reviewing this submission. We address your concerns below.

W1: Novelty.

Our paper introduces a new mathematical formulation, theoretical guarantees, architectural analysis, a KV-cache-specific calibration procedure, and practical contributions. The novelty lies in extending the WUSH weight-activation quantization framework to a fundamentally different KV cache quantization problem and solving the resulting mathematical and architectural challenges.

1. New mathematical formulation. WUSH was originally developed for two-sided weight-activation quantization, whereas KV-cache quantization is one-sided. To bridge this gap, we develop a new Gram/Hessian formulation (Definition 1), which requires identifying the appropriate sensitivity matrices for keys and values under grouped-query attention. We derive these matrices from the query vectors and output-projection weights, respectively (Eq. 4). We further introduce a normalization-invariant formulation (Eq. 1) that preserves the geometry of these statistics regardless of their scaling.

2. New theoretical guarantee. The original WUSH analysis only covers AbsMax-scaled non-clipped quantizers (optimal for FP, asymptotically optimal for INT). We extend its optimality analysis to an RMS-scaled, clipped, integer quantizer (QuEST). This is the first optimality argument for such transforms over a clipped quantizer.

3. Novel transform placement analysis. Appendix A shows that the only key transforms that can be folded into fixed projection and normalization weights are paired sign flips, which do not redistribute coordinate magnitudes. This is why the full key transforms are applied online after RoPE, while the value transforms are folded into the weights.

4. KV-cache calibration procedure. We design a forward-only calibration pipeline (Algorithm 1) to efficiently gather the required second-order statistics under grouped-query attention. We also derive exact attention-aware Hessians that require only forward-pass quantities (Appendix C).

5. Practical contribution. To the best of our knowledge, our accuracy results become the new state-of-the-art for 2-bit KV cache quantization. We also integrate our method into SGLang for efficient decoding.

W2: Other model families.

To show the downstream benefits on other model families, we add experiments on Granite-4.2-8B, which is a newer model from IBM that includes native step-by-step reasoning capabilities. Older models like Llama-3 perform substantially worse in our benchmarks even in full precision, making it difficult to distinguish the effects of different transforms at 2-bit precision.

On Granite-4.2-8B, WUSH-KV outperforms OSCAR under the same setups. Below are the tables for the scores and average output tokens per request. We have also added these numbers to Table 2 in the updated manuscript (marked in blue font).

Scores:

MethodAIME 2025MATH-500GPQA DiamondLiveCodeBench v6
BF1688.9 ± 1.992.2 ± 0.965.7 ± 1.371.8 ± 2.9
OSCAR75.6 ± 3.991.7 ± 0.362.3 ± 1.564.0 ± 0.6
WUSH-KV82.2 ± 1.991.7 ± 1.063.8 ± 1.169.0 ± 0.7

Output Tokens:

MethodAIME 2025MATH-500GPQA DiamondLiveCodeBench v6
BF1628.5k5.2k21.0k22.1k
OSCAR28.1k6.3k17.8k23.5k
WUSH-KV27.0k5.5k18.6k22.1k

We are also extending our SGLang integration to support more model families.

W3: Throughput.

We measure Qwen3-8B decoding throughput (tokens/s) on an NVIDIA B300 GPU. The results are shown in the table below. We have also added this table to the updated manuscript as Table 3 (marked in blue font). At matched context lengths and batch sizes, WUSH-KV achieves approximately 1.3-2.2× higher decode throughput than BF16. WUSH-KV preserves the inference efficiency of OSCAR while improving accuracy.

Context LengthBatch SizeBF16OSCAR 2-BitWUSH-KV 2-Bit
32K160.6113.3112.7
32K8379.2477.8477.8
64K133.874.474.0
64K8192.1249.7249.7

WUSH-KV also generates fewer tokens than OSCAR per request, improving per-request latency. Table 2 shows the mean token count for each task on each model. On Qwen3-8B, WUSH-KV uses 11-27% fewer tokens.

Area_Chair_XwWAOpenReview ↗

Please respond to the rebuttal

Dear Reviewer D9xd, the authors have posted their response. Please add a brief follow-up indicating whether your concerns have been addressed and noting any remaining questions, if you haven't done so. Thank you!

Reviewer_ymTqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

WUSH-KV is a post-training KV cache quantization technique for large language models. WUSH-KV adapts WUSH, a weight/activation quantization approach to work on KV cache quantization. WUSH-KV constructs data aware transform for keys and valuess, with the value transform folded into the model weights and the key transform applied after RoPE. The transforms can be paired with clipped quantizers. At 2-bit KV Cache, WUSH-KV outperforms OSCAR and TurboQuant baselines.

Critical strengths

Strengths :

  1. Substantial improvements in selected settings. At 2 bits, Qwen3-8B WikiText-2 perplexity improves from OSCAR’s 13.74 to 10.51, against BF16’s 9.72. The downstream LiveCodeBench improvements are also substantial.
  2. Useful controlled experiments in section 5.1 and 5.2. The reconstruction study holds the quantizer fixed, separately evaluates key and value quantization, and removes full-precision windows to isolate transform quality.
  3. Paper is written well and easy to follow.
Critical weaknesses

Weaknesses :

  1. Novelty concerns. WUSH-KV adapts WUSH to work on KV cache quantization. The adaptation seems straightforward and the folding of value transform has already been done in several prior works (Quarot, SpinQuant, KVLinC, OSCAR).
  2. Theoretical analysis and experiment discrepancy. Theoretical analysis demonstrates that WUSH-KV is near optimal for Quest-INT kind of quantizers but the downstream evaluation experiments are done on OSCAR-style quantization.
  3. Paper does not demonstrate any practical inference benefit in terms of inference latency reduction or memory reduction.
  4. Downstream tasks show meaningful but uneven improvements. On Qwen3-8B, WUSH-KV outperforms OSCAR on all benchmarks but trails behind OSCAR on three out of four benchmarks on Qwen-3-32B.
  5. Results are only shown on Qwen3 backbone and generalizability across different backbones is not clearly demonstrated.
  6. Evaluation only considers a single competitive baseline (OSCAR). It would be valuable if the authors can also compare against more transform based KV cache quantization baselines like FlatQuant, Quarot, RotateKV).
  7. Appendix D.2 demonstrates that WUSH-KV calibration is sensitive to calibration dataset choice since the authors report that GPQA-calibrated WUSH-KV obtain suboptimal results on AIME.
Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you for your time and detailed feedback. Below, we address your concerns.

W1: Novelty.

Our paper introduces a new mathematical formulation, theoretical guarantees, architectural analysis, a KV-cache-specific calibration procedure, and practical contributions. The novelty lies in extending the WUSH weight-activation quantization framework to a fundamentally different KV cache quantization problem and solving the resulting mathematical and architectural challenges.

1. New mathematical formulation. WUSH was originally developed for two-sided weight-activation quantization, whereas KV-cache quantization is one-sided. To bridge this gap, we develop a new Gram/Hessian formulation (Definition 1), which requires identifying the appropriate sensitivity matrices for keys and values under grouped-query attention. We derive these matrices from the query vectors and output-projection weights, respectively (Eq. 4). We further introduce a normalization-invariant formulation (Eq. 1) that preserves the geometry of these statistics regardless of their scaling.

2. New theoretical guarantee. The original WUSH analysis only covers AbsMax-scaled non-clipped quantizers (optimal for FP, asymptotically optimal for INT). We extend its optimality analysis to an RMS-scaled, clipped, integer quantizer (QuEST). This is the first optimality argument for such transforms over a clipped quantizer.

3. Novel transform placement analysis. Appendix A shows that the only key transforms that can be folded into fixed projection and normalization weights are paired sign flips, which do not redistribute coordinate magnitudes. This is why the full key transforms are applied online after RoPE, while the value transforms are folded into the weights. (We do not claim we are the first paper to fold the value transforms, though.)

4. KV-cache calibration procedure. We design a forward-only calibration pipeline (Algorithm 1) to efficiently gather the required second-order statistics under grouped-query attention. We also derive exact attention-aware Hessians that require only forward-pass quantities (Appendix C).

5. Practical contribution. To the best of our knowledge, our accuracy results become the new state-of-the-art for 2-bit KV cache quantization. We also integrate our method into SGLang for efficient decoding.

W2: Theory and experiment discrepancy.

We use QuEST for theoretical analysis since it is a tractable model of a clipped quantizer. Empirically, WUSH-KV extends beyond the idealized setting of our theoretical analysis and transfers across different practical clipped quantizers. Importantly, WUSH-KV's benefits hold under both the QuEST quantizer and the OSCAR-style quantizer. We show in the paper that WUSH with QuEST achieves the lowest layerwise error (Figure 1) and perplexity (Table 1). The Qwen3-8B AIME 2025 results below demonstrate downstream improvements under both quantizers.

TransformOSCAR-style quantizerQuEST quantizer
OSCAR34.4 ± 8.40.0 ± 0.0
WUSH-KV46.7 ± 3.328.9 ± 6.9

QuEST is a symmetric quantizer, while the OSCAR-style quantizer is an asymmetric affine quantizer, so the OSCAR-style quantizer costs 0.125 more bit per element and empirically achieves better end-to-end accuracy than QuEST under the same transform. Using the OSCAR-style quantizer in our main evaluations improves the accuracy and enables a direct comparison with the OSCAR paper under the same quantizer. As shown in the above table, OSCAR also works poorly with the QuEST quantizer.

W3: Practical benefit.

We measure Qwen3-8B decoding throughput (tokens/s) on an NVIDIA B300 GPU. The results are shown in the table below. We have also added this table to the updated manuscript as Table 3 (marked in blue font). At matched context lengths and batch sizes, WUSH-KV achieves approximately 1.3-2.2× higher decode throughput than BF16. WUSH-KV preserves the inference efficiency of OSCAR while improving accuracy.

Context LengthBatch SizeBF16OSCAR 2-BitWUSH-KV 2-Bit
32K160.6113.3112.7
32K8379.2477.8477.8
64K133.874.474.0
64K8192.1249.7249.7

WUSH-KV also generates fewer tokens than OSCAR per request, improving per-request latency. Table 2 shows the mean token count for each task on each model. On Qwen3-8B, WUSH-KV uses 11-27% fewer tokens.

The per-token KV-cache memory reduction follows directly from our storage analysis (Section 4.5). Each quantized element takes 2 bits plus 0.25 bit for shared scale and zero point. This is 14% of BF16. Since SGLang preallocates its KV-cache pool at startup, the measured GPU memory utilization does not directly reflect the reduced per-token storage. Nevertheless, as a sanity check, we have verified that SGLang indeed supports more cached tokens with the quantized configuration under the same memory budget.

As shown in the token number analysis, WUSH-KV also reduces the output tokens per request compared to OSCAR, further reducing the memory usage.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W4: Downstream improvements.

After the initial submission, we identified a YaRN configuration mismatch caused by a Hugging Face Transformers interface change, which affected some of the Qwen3-4B-Thinking-2507 (should be disabled) and Qwen3-32B (should be enabled) evaluations. We have fixed the issue, rerun the affected evaluations, and corrected the numbers in the updated manuscript (marked in blue font). The corrected Qwen3-32B results are also listed below. WUSH-KV now achieves higher accuracy than OSCAR on all four Qwen3-32B benchmarks.

MethodAIME 2025MATH-500GPQA DiamondLiveCodeBench v6
OSCAR64.4 ± 6.993.1 ± 0.660.4 ± 2.034.9 ± 2.1
WUSH-KV67.8 ± 5.194.9 ± 0.362.0 ± 1.635.4 ± 1.0

Also, despite the fact that we compare with OSCAR, we would like to note that OSCAR is contemporaneous with ours.

W5: Other model families.

To show the downstream benefits on other model families, we add experiments on Granite-4.2-8B, which is a newer model from IBM that includes native step-by-step reasoning capabilities. Older models like Llama-3 perform substantially worse in our benchmarks even in full precision, making it difficult to distinguish the effects of different transforms at 2-bit precision.

On Granite-4.2-8B, WUSH-KV outperforms OSCAR under the same setups. Below are the tables for the scores and average output tokens per request. We have also added these numbers to Table 2 in the updated manuscript (marked in blue font).

Scores:

MethodAIME 2025MATH-500GPQA DiamondLiveCodeBench v6
BF1688.9 ± 1.992.2 ± 0.965.7 ± 1.371.8 ± 2.9
OSCAR75.6 ± 3.991.7 ± 0.362.3 ± 1.564.0 ± 0.6
WUSH-KV82.2 ± 1.991.7 ± 1.063.8 ± 1.169.0 ± 0.7

Output Tokens:

MethodAIME 2025MATH-500GPQA DiamondLiveCodeBench v6
BF1628.5k5.2k21.0k22.1k
OSCAR28.1k6.3k17.8k23.5k
WUSH-KV27.0k5.5k18.6k22.1k

We are also extending our SGLang integration to support more model families.

W6: Baselines.

Our evaluation is not limited to OSCAR. Figure 1 and Table 1 compare WUSH-KV against identity, random rotation, Hadamard, and OSCAR transforms. In particular, the Hadamard baseline represents the KV-cache rotation used in QuaRot and practical TurboQuant-style implementations. Table 2 also includes this Hadamard baseline in downstream evaluations.

OSCAR is used as a main baseline because it is a contemporaneous work and the most closely related state-of-the-art work. OSCAR and WUSH-KV both feature offline second-order statistics, closed-form formulation, and the post-RoPE placement. RotateKV applies the transform before RoPE, which introduces decoding overhead (Appendix A) and is not supported by our efficient SGLang serving integration. FlatQuant learns its transforms via gradient-based calibration, which has a higher calibration cost for the same amount of data.

W7: Calibration dataset.

WUSH-KV obtains suboptimal results on GPQA Diamond calibration data not because of data sensitivity, but because it requires more data to reliably estimate WUSH-KV's second-order statistics. Compared to OSCAR's orthogonal transforms, WUSH-KV's general invertible transforms have more degrees of freedom and require more calibration data for accurate estimation. WUSH-KV follows the original WUSH calibration setup, using 128 task-independent FineWeb-Edu sequences (approximately 4.2M tokens). In contrast, the GPQA Diamond dataset contains only about 8k tokens.

Our main configuration uses the same generic corpus (FineWeb-Edu), with transforms calibrated once per model and reused across all downstream tasks. No task-specific data is used. Our evaluation results demonstrate strong performance across all tasks, indicating that the transforms are not sensitive to the calibration dataset.

To test sensitivity directly, we recalibrated WUSH-KV on the C4 dataset with the same budget as our main setup and re-evaluated Qwen3-8B WikiText-2 perplexity. The results are essentially unchanged at every bitwidth, and they are much better than the OSCAR results in Table 1.

Calibration4-Bit3-Bit2-Bit
FineWeb-Edu9.739.8210.51
C49.749.8210.53

In fact, Appendix D.2 is intended to demonstrate OSCAR's sensitivity to calibration data and explain why we use GPQA-Diamond-calibrated OSCAR transforms for comparison in the main paper. Recalibrating OSCAR on FineWeb-Edu with the same calibration budget as WUSH-KV reduces its accuracy on GPQA Diamond as well as other downstream tasks. We therefore retain OSCAR's recommended GPQA-Diamond-calibrated transforms rather than weakening this baseline.

Reviewer_ymTqOpenReview ↗

Response to authors

I thank the authors for their rebuttal. I will maintain my original rating, as I remain concerned about the limited novelty of this work. Transform-aware placement is not a new finding. While the paper provides new theoretical analysis for Quest-INT-style quantizers, the evaluation uses OSCAR-style quantization, leaving a gap between the analysis and the empirical results. The authors describe OSCAR as concurrent work, but WUSH-KV nevertheless builds on OSCAR-style quantization. Finally, the implementation discrepancies acknowledged in the rebuttal, particularly the YaRN configuration mismatch, raise concerns about the reliability of the reported comparisons.

Thank you for the follow-up! We would like to provide further clarification.

Regarding transform placement, we do not claim novelty for the post-RoPE placement itself. Appendix A gives a mathematical characterization of which key transforms can be folded into the weights, which explains why the post-RoPE placement is used by WUSH-KV. Such an argument is new to the WUSH paper.

Our empirical evaluation includes the QuEST quantizer and is not limited to the OSCAR-style quantizer. Figures 1-3 and Table 1 use QuEST for all compared transforms for layerwise reconstruction losses and end-to-end perplexities, and Table 4 and our rebuttal additionally report downstream results with QuEST. These results show WUSH-KV performs better than the baseline transforms under the same QuEST quantizer, directly supporting our theoretical analysis on the QuEST quantizer. Table 2 uses the OSCAR-style quantizer because it achieves better end-to-end accuracy for both transforms, which also shows that the benefits of WUSH-KV transfer to another practical quantizer.

OSCAR is solely a preprint on arXiv today and is not yet published in peer-reviewed conference proceedings or journals, so it is a contemporaneous work by ICLR's definition. OSCAR uses orthogonal transforms, while WUSH-KV uses general invertible transforms that allow anisotropic rescaling. By Theorem 3, the reconstruction loss of any orthogonal transform is greater than or equal to the optimum attained by WUSH-KV. The WUSH-KV transform, its calibration, and its theory do not depend on OSCAR. Table 2 adopts the OSCAR-style quantizer so that the transform is the only difference between the compared methods, enabling a controlled comparison of transform quality, and the QuEST results above show that this quantizer choice is optional rather than intrinsic to WUSH-KV. We believe that including and outperforming a strong state-of-the-art baseline should be viewed as a strength of our evaluation rather than a reason to penalize our work.

Regarding implementation reliability, the YaRN configuration mismatch arose from differences in Hugging Face Transformers versions across the machines used for our experiments, which caused some runs to use unintended YaRN settings. The issue affected certain Qwen3-4B-Thinking-2507 and Qwen3-32B evaluations. We have standardized the software versions, corrected the configurations, rerun all affected experiments, and updated the manuscript with the correct and consistent results. The other evaluations were unaffected. We also clarify that YaRN configurations are model-specific, depending on the model's default supported context length. For fair comparisons, the relevant requirement is that all KV-cache quantization methods use consistent configurations for the same model, rather than requiring identical YaRN settings across different models.

Reviewer_rPkLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

WUSH-KV adapts WUSH from weight-activation quantization to KV-cache quantization. By constructing separate key and value transforms with clipped quantizers, it provides a conditional near-optimality guarantee for QuEST INT and reduces error. Implemented into SGLang, WUSH-KV reports to perform comparably to or outperform the OSCAR transform across evaluated models and downstream tasks.

Critical strengths
  1. The method is shown to be a substantial improvement over OSCAR, with the geometric mean module-output error dropping from 0.309 to 0.208 and perplexity dropping from 13.74 to 10.51 when quantized down to 2-bits.
  2. Prior work, including the WUSH transform, is elaborated upon in detail and this clearly separates what has already been established versus what has been developed by WUSH-KV.
Critical weaknesses
  1. The net latency benefits of WUSH-KV remain unclear without a proper throughput evaluation. Reduced cache storage does not directly translate to faster throughput compared to other quantization methods.
  2. With only the Qwen model family tested, the generality of these results is unclear.
Questions for authors
  1. How does WUSH-KV perform in another model family such as Llama?
  2. At matched context length and batch sizes, what is the throughput of WUSH-KV in comparison to OSCAR?
2 comments

Thank you for your time and effort in reviewing this submission. We appreciate your detailed and thoughtful feedback and address your concerns and questions below.

W1 & Q2: Throughput.

We measure Qwen3-8B decoding throughput (tokens/s) on an NVIDIA B300 GPU. The results are shown in the table below. We have also added this table to the updated manuscript as Table 3 (marked in blue font). At matched context lengths and batch sizes, WUSH-KV achieves approximately 1.3-2.2× higher decode throughput than BF16. WUSH-KV preserves the inference efficiency of OSCAR while improving accuracy.

Context LengthBatch SizeBF16OSCAR 2-BitWUSH-KV 2-Bit
32K160.6113.3112.7
32K8379.2477.8477.8
64K133.874.474.0
64K8192.1249.7249.7

WUSH-KV also generates fewer tokens than OSCAR per request, improving per-request latency. Table 2 shows the mean token count for each task on each model. On Qwen3-8B, WUSH-KV uses 11-27% fewer tokens.

W2 & Q1: Other model families.

To show the downstream benefits on other model families, we add experiments on Granite-4.2-8B, which is a newer model from IBM that includes native step-by-step reasoning capabilities. Older models like Llama-3 perform substantially worse in our benchmarks even in full precision, making it difficult to distinguish the effects of different transforms at 2-bit precision.

On Granite-4.2-8B, WUSH-KV outperforms OSCAR under the same setups. Below are the tables for the scores and average output tokens per request. We have also added these numbers to Table 2 in the updated manuscript (marked in blue font).

Scores:

MethodAIME 2025MATH-500GPQA DiamondLiveCodeBench v6
BF1688.9 ± 1.992.2 ± 0.965.7 ± 1.371.8 ± 2.9
OSCAR75.6 ± 3.991.7 ± 0.362.3 ± 1.564.0 ± 0.6
WUSH-KV82.2 ± 1.991.7 ± 1.063.8 ± 1.169.0 ± 0.7

Output Tokens:

MethodAIME 2025MATH-500GPQA DiamondLiveCodeBench v6
BF1628.5k5.2k21.0k22.1k
OSCAR28.1k6.3k17.8k23.5k
WUSH-KV27.0k5.5k18.6k22.1k

We are also extending our SGLang integration to support more model families.

Thank you for addressing my questions and concerns. I will maintain my score of acceptance.

Comments on the paper

2 comments

Dear Reviewers,

As the author-reviewer discussion period is going to end in 24 hours, we would greatly appreciate it if you could take a look at our responses, let us know whether they address your concerns, and update your ratings if your concerns have been resolved. We would be happy to clarify any remaining questions.

Thank you for your time and consideration!

AuthorsOpenReview ↗

General Response

We thank the reviewers for finding our paper clear and sound. They also recognize our practical contribution: WUSH-KV substantially improves upon OSCAR, the state-of-the-art 2-bit KV-cache quantization method.

The main shared concerns were inference efficiency and generality. We address both with new results (also marked in blue in the revised PDF):

  • Throughput. WUSH-KV achieves 1.3-2.2× BF16 decode throughput, on par with OSCAR, while generating fewer tokens per request.
  • Generality. WUSH-KV extends to a new model family, IBM Granite-4.2, where it again outperforms OSCAR.

Across all evaluated models and tasks, WUSH-KV consistently outperforms OSCAR, with especially large gains on challenging reasoning and long-context tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.