acceptodds
Under review as a conference paper at ICLR 2027

OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond

Abstract

The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a dominant memory bottleneck for efficient deployment. Extreme low-bit quantization has emerged as a fundamental imperative to reclaim memory efficiency and sustain high-throughput inference. While the established per-channel quantization effectively accommodates intrinsic channel-wise outliers in Key tensors, its efficacy diminishes under extreme compression. In this work, we revisit the inherent limitations of the per-channel quantization paradigm from both empirical and theoretical perspectives. Our analysis identifies Token Norm Imbalance (TNI) as the primary bottleneck to quantization fidelity. We demonstrate that TNI systematically amplifies errors when shared quantization parameters are required to span token groups exhibiting substantial norm disparities. Instead of relying on intricate quantization pipelines (e.g., TurboQuant), we propose OScaR (Omni-Scaled Canalized Rotation), an accurate and lightweight KV cache compression framework for X-LLMs (i.e., text-only, multi-modal, and omni-modal LLMs). Advancing the per-channel paradigm, OScaR employs Canalized Rotation followed by Omni-Token Scaling to mitigate TNI-induced sequence-dimensional variance both effectively and efficiently, further supported by our optimized system design and CUDA kernels. Extensive evaluations across X-LLMs show that OScaR consistently outperforms existing methods and achieves near-lossless performance under INT2 quantization, establishing it as a robust, low-complexity, and universal framework that defines a new Pareto front. Compared with the BF16 FlashDecoding-v2 baseline, our OScaR implementation achieves a notable up to 3.0× speedup in decoding, reduces memory footprint by 5.3×, and increases throughput by 4.1×. Code for result reproduction is provided in the anonymous supplementary materials.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache quantization, low-bit quantization, efficient inference, large language models, multimodal models, token norm imbalance

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oJorOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OScaR studies how to compress the KV cache used by large language models during response generation to reduce GPU memory use. It rotates the vectors in the cache and adjusts the norms of vectors for different tokens to reduce errors caused by low-precision storage. The paper provides implementation code and evaluates task performance, GPU memory use, and other metrics after compression on text and multimodal tasks.

Critical strengths
  1. The method does not require retraining the model.
  2. Across multiple models and tasks, performance after compression remains close to that of the full-precision models, which has practical value.
  3. The experiments cover multiple models and tasks. They also examine the role of each component and measure speed, GPU memory use, and other metrics.
Critical weaknesses
  1. The paper identifies token norm imbalance as the main bottleneck in per-channel quantization and provides theoretical analysis and quantization-error experiments. However, the experimental evidence is not sufficient to show why it is the main bottleneck or how much it affects different models and tasks.

  2. The gains in the main experiments are small, and performance on some tasks is below that of existing methods.

In Table 1, the average scores for Llama-3.1-8B and Qwen3-8B exceed those of OTT by only 1.01 and 0.53 percentage points, respectively. For each model, OScaR also scores below at least one other quantization method on three tasks. For example, Llama-3.1-8B scores below KIVI on HotpotQA, while Qwen3-8B scores below OTT on GovReport and MultiNews. The paper does not further explain why performance is weaker on these tasks.

  1. The ablation settings differ from those of the main experiments.

Table 12 tests Canalized Rotation, Omni-Token Scaling, and their combination. However, the models, tasks, and settings such as group size differ from those of the main experiments in Table 1. Should the paper also report each component's contribution under the main experimental settings?

  1. Some experimental results are described repeatedly in the main text and appendix.

For example, Section 4.3 and Appendix W both describe an approximately 3-fold decoding speedup over BF16. I suggest reducing this repetition.

Questions for authors
  1. Why does OScaR perform worse than baseline methods such as KIVI, RotateKV, and OTT on some tasks?

  2. What is the contribution of each component under the main experimental settings in Table 1?

  3. What experimental evidence shows that token norm imbalance is the main bottleneck in per-channel quantization?

1 comment

We sincerely thank the reviewer for the careful assessment and constructive feedback. Below, we address the shared-range evidence, task-level results, component contributions, and presentation.

Response to W1 and Q3: Evidence for TNI in shared-range per-channel quantization. In grouped per-channel Key quantization, tokens share a range within each channel. Token-magnitude disparities can enlarge this range and coarsen the resolution for other tokens. Appendix H analyzes this mechanism under a uniform-noise approximation. The following submitted results quantify the associated error differences and test the approximation.

Error comparison (Table 2). On LLaVA-v1.5-7B with RTN and G32, Key MSE ×100 is:

PrecisionWith low-norm outlier tokensWith those outliers removed
INT40.230.16
INT31.340.78
INT25.923.52

At INT2, this comparison shows approximately 68% higher MSE with the outlier tokens under the evaluated grouping conditions.

Real-tensor validation (Table 3). On Qwen3-8B layer 18, we evaluate post-RoPE Keys before CR/OTS using 16 Paul Graham sequences of 4,096 tokens, G32, and the paper's asymmetric RTN rule:

PrecisionPredicted MSEMeasured MSEMeasured / predictedPearson (log MSE)
INT40.0091050.0090950.99890.9872 ± 0.0002
INT30.0418100.0417090.99760.9872 ± 0.0002
INT20.2276340.2263620.99440.9869 ± 0.0002

Correlations are computed per sequence (mean ± standard deviation). The MSE agreement and strong correlations support the approximation in this evaluated setting.

These results support TNI as an important structural source of error in extreme per-channel Key quantization and motivate addressing token-norm disparity and channel geometry together. We will replace “the primary bottleneck” with this characterization in the abstract and main text, making the setting and assumptions explicit.

Response to W2 and Q1: Average gains and task-level exceptions. Our interpretation is that these exceptions reflect task-dependent quantization trade-offs: different transforms, grouping schemes, and token-protection policies distribute errors differently, and their impact on attention depends on the queries (Appendices H.1 and J.1).

In Table 1, OScaR achieves the highest eight-task average among the compared quantized methods on both models. Although it does not lead on every task, five of the six gaps to the best competing quantized result are 0.06–0.39 score points, including the highlighted Qwen3-8B GovReport and MultiNews gaps to OTT (0.23 and 0.36 points, respectively). The largest gap occurs on Llama-3.1-8B HotpotQA: OScaR's 13.68 remains close to BF16's 13.76, while KIVI's 14.66 exceeds BF16. On TriviaQA for the same model, OScaR's 90.41 and RotateKV's 90.56 both exceed BF16's 90.15.

The corresponding mean gains over OTT are 1.01 points on Llama-3.1-8B and 0.53 points on Qwen3-8B. Our contribution combines the shared-range analysis, a simple, training-free CR/OTS design, and a fused implementation, with competitive INT2 quality across the evaluated text-only, multimodal, and omni-modal models. We will discuss these task-level trade-offs in the main text.

Response to W3 and Q2: Component contributions at G32/R128. We evaluated four configurations on Qwen3-8B at Table 1's K2V2, G32/R128 settings across all eight LongBench-E tasks (2,318 examples per configuration). Only Key-side CR and OTS vary; Value-side processing and other conditions are fixed. Plain disables both components. All runs share SDPA prefill and a target 32K input cap and are reported separately from the original Table 1.

ConfigurationEight-task averageChange vs. Plain
Plain47.66—
CR48.33+0.67
OTS47.84+0.18
CR + OTS48.45+0.79

The combination achieves the highest observed mean, 0.79 points above Plain. CR provides most of the average gain; adding OTS raises the observed mean by 0.12 points, with task-dependent effects. This complements Table 12's multimodal/G128 ablation with component evidence under the main quantization settings. We will add this comparison to the main text.

Response to W4: Reducing repetition. We will reduce overlapping efficiency descriptions in Section 4.3 and Appendix W, keeping the main findings and essential settings in the main text and the detailed measurement protocols and component overhead breakdown in the appendix. Cross-references will replace repeated summaries.

Thank you again for helping us strengthen the evidence and improve the clarity of the paper.

Reviewer_aGUkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OScaR (Omni-Scaled Canalized Rotation), a training-free framework for extreme KV-cache quantization in X-LLMs (i.e., text-only, multi-modal, and omni-modal LLMs). Through empirical and theoretical analysis, the authors identify Token Norm Imbalance (TNI) as the fundamental bottleneck limiting per-channel quantization in X-LLMs, and introduce a two-step mitigation: Canalized Rotation employs the Hadamard transform to distribute the concentrated energy in the outlier channels across various dimensions, and Omni-Token Scaling reduces the norm disparities between different tokens before quantization through token-wise normalization. The experimental data show that OScaR, when quantized under INT2 KV Cache, can typically maintain task performance close to the 16-bit baseline and outperform the compared quantization baselines. Additionally, for the optimization of the system, OScaR has brought about significant efficiency improvements.

Critical strengths

1.The paper, through empirical and theoretical analysis, demonstrates the impact of Token Norm Imbalance (TNI) on per-channel Key quantization, providing a clear basis for method design. 2.The method is concise and targeted. OScaR combines Canalized Rotation and Omni-Token Scaling in a training-free manner, and the complementarity of the two components is supported by ablation experiments. 3.Experiments show that OScaR maintains model performance close to the 16-bit baseline under INT2 KV cache quantization, and achieves memory savings and decoding efficiency improvements under the tested long context and batch inference configurations.

Critical weaknesses

1.The theoretical explanation for the amplification of errors caused by TNI is mainly developed in Appendix; it was not fully elaborated in the main text. But now it is fine for this submission. 2.The current experiments lack comparisons with some of the more recent related methods, such as Kitty[1] and NSNQuant[2], which makes it unclear what the performance advantages of OScaR are over these related solutions. 3.In the paper, the efficiency comparison mainly focuses on BF16 FlashDecoding-v2; it is suggested to conduct complete measurements for five baselines to better separate the contributions of OScaR.

[1] Xia H, Li J, Wu T, et al. Kitty: Accurate and efficient 2-bit KV cache quantization with dynamic channel-wise precision boost[J]. Proceedings of Machine Learning and Systems, 2026, 8: 511-525. [2] Son D, Choi E, Yoo S. NSNQuant: A double normalization approach for calibration-free low-bit vector quantization of KV cache[J]. Advances in Neural Information Processing Systems, 2026, 38: 43124-43159.

Questions for authors

Q1: The paper specifies a fixed residual length R and group size G, but no sensitivity analysis of these two parameters has been reported. Could the authors change R and G respectively and report the performance, memory usage and decoding delay? Q2: Could the authors provide a more detailed explanation of the rationale behind applying a uniform token-wise normalization to different modalities?

1 comment

We sincerely thank the reviewer for the careful review and constructive suggestions.

Response to W1: Presentation of the analysis. We will briefly highlight the key assumptions and the connection between the analysis and CR/OTS in the main text, retaining the detailed derivations in the appendix.

Response to W2: Kitty and NSNQuant. Using NSNQuant's official implementation and Kitty's official quality simulator, we added two comparisons. The NSNQuant release used here provides Llama/Mistral adapters but no Qwen3 adapter, so we use Meta-Llama-3-8B; Kitty supports Qwen3-8B, a main-text model.

Both comparisons cover the same four LongBench-E tasks: HotpotQA, Qasper, Passage Retrieval, and RepoBench-P, including all 1,124 examples per method. Within each comparison, pretrained weights, input tokens, greedy generation, and scoring are shared. Each method retains its own grouping, protection, metadata settings, and numerical path. OScaR uses K2V2 and G32/R128 in both comparisons.

NSNQuant: Meta-Llama-3-8B, FP16, target input cap of 7,680 tokens.

MethodFour-task average
FP1624.60
OScaR24.07
NSNQuant23.94

NSNQuant uses its official 2-bit codebook, Hadamard transform, and window/residual128. OScaR's observed mean is 0.13 points higher. All inputs plus generation fit the 8,192-token context; this Llama-3 comparison is reported separately from Table 1's Llama-3.1 results.

Kitty: Qwen3-8B, BF16, target 32K input cap.

MethodFour-task average
BF1644.84
OScaR43.54
Kitty43.33

Kitty uses its paper configuration: G128, residual128, sink32, and 12.5% of Key channels at INT4. OScaR's observed mean is 0.21 points higher.

These results support OScaR's quality competitiveness in the evaluated settings. Our contribution links shared-range error analysis to a simple, training-free CR/OTS design for scalar Key quantization, a regular low-bit cache layout, and fused implementation.

Response to W3: Efficiency comparisons with quantized baselines. Comparisons beyond BF16 need to account for the model, KV precision/protection settings, and kernel/runtime. Available releases also differ in their deployment setup: RotateKV's public code provides fake-quantization simulation, and official code for the original TurboQuant paper was not available at the time of our evaluation. We will clarify the comparison conditions of our existing efficiency evaluation. Broader cross-method benchmarking with aligned models and deployment configurations would be a useful future extension.

Response to Q1: Independent R and G sensitivity. We independently varied R and G on Qwen3-8B: R64/128/256 at G32, and G32/128 at R128. The QDQ quality study uses BF16 weights, K2V2, common SDPA prefill, a target 32K input cap, and greedy generation on complete HotpotQA and Qasper datasets (524 examples per configuration). G sets both Key token-group and Value channel-group sizes.

GRHotpotQAQasperAverage
326412.638.9110.77
3212812.719.3411.02
3225612.969.6311.29
12812812.589.3210.95

Within the tested settings, larger R or smaller G yielded higher observed means, while requiring greater BF16 residual capacity or more grouping metadata. This illustrates the trade-off between quality and compression.

Under Appendix M.6's metadata convention, nominal packed storage at G32/G128 is approximately 3.016/2.266 bits/KV value, including scale, zero point, and the shared Key norm, before residual buffers. These are storage-format estimates, not measured whole-model peak memory.

Response to Q2: A common normalization across modalities. After CR redistributes channel energy, OTS normalizes each Key token using its own norm, bringing tokens from different modalities to a comparable norm scale for quantization. The common rule therefore adapts to each token and directly reduces norm disparities among tokens sharing per-channel quantization ranges. Original norms are retained as metadata and restored in attention, preserving each token's magnitude information.

The submitted WorldSense ablation in Table 12 provides supporting evidence at K2V2/G128:

ModelPlain INT2CROTSCR + OTS
Qwen2.5-Omni-3B22.6437.7417.6138.36
Qwen2.5-Omni-7B39.6241.5135.2242.77

CR+OTS achieves the highest observed score on both models, while OTS alone scores below Plain, supporting the combined design in these settings. The shared rule also supports a simple, regular computation path with fused preprocessing and magnitude restoration in attention.

Thank you again for helping us improve the clarity and completeness of the paper.

Reviewer_GnTfOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies token norm imbalance (i.e., large magnitude differences between tokens sharing per-channel quantization parameters) as a key source of error. To make 2-bit Key-cache quantization accurate, OScaR first redistributes large feature channels with a Hadamard rotation, then normalizes each token’s magnitude. This order matters because normalization alone can create new channel outliers, while rotation alone leaves token norms unchanged. The original magnitudes are saved and restored when computing attention.

Critical strengths
  • Teh comparison includes a good number of previous KV cache quantization methods.
  • Sensible study and criticism of per-channel key quantization.
  • Simple, training-free method that preserves good accuracy under 2-bit quantization.
  • Practical CUDA implementation with clear improvements in decoding speed, memory usage, and throughput.
Critical weaknesses
  • Limited theoretical justification for when rotation followed by scaling reduces reconstruction error.
  • Speed comparisons with TurboQuant+ use different runtimes, making the gains harder to attribute to the method.
  • Main text-only evaluation is limited to two 8B models: this is not a weakness in itself, but considering the current models and content length, the experimental setup is not timely.
Questions for authors
  • While the comparison covers a good range of KV-cache quantization methods, why didn’t you test newer text-only models? For example, Llama 3 is quite old now. Do TNI and OScaR’s gains generalize to larger models and different attention architectures?

  • Under what assumptions does rotation followed by scaling reduce reconstruction MSE? Can you support this with a theoretical argument or direct error ablation?

  • Can the authors clarify how the norm characterization in Section 3.1 relates to existing work? E.g., Quantize What Counts: More for Keys, Less for Values (ACL 2026) characterizes spectral and Frobenius norm disparities between Keys and Values and connects them to quantization sensitivity, or A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression (EMNLP 2024) studies token-level Key norms and their relationship with attention allocation. Please discuss these works and distinguish the new empirical observations in OScaR from its analysis of shared-range per-channel quantization error and its rotation–scaling design.

2 comments
AuthorsOpenReview ↗

Response to Reviewer GnTf (Part 1/2)

We sincerely thank the reviewer for the careful assessment and constructive feedback. We address the error analysis, runtime comparisons, model coverage, and prior work below.

Response to W1 and Q2: Shared-range error analysis and the CR/OTS design. We appreciate this question. Section 3.1 and Appendix H (Eqs. 9–11) analyze shared-range bb-bit per-channel Key quantization. For channel range Rj,gR_{j,g} and step size Δj,g=Rj,g/(2b−1)\Delta_{j,g}=R_{j,g}/(2^b-1), the uniform-noise approximation gives E[ϵ2]=Δj,g2/12\mathbb E[\epsilon^2]=\Delta_{j,g}^2/12. The model's group-average expected squared L2L_2 error satisfies

=\frac{\sum_jR_{j,g}^2}{12(2^b-1)^2} \ge\frac{(r_{\max}-r_{\min})^2}{12(2^b-1)^2}.$$ Here $r_{\max},r_{\min}$ are the group's largest and smallest token norms. For the corresponding tokens $m,n$, the bound follows from $R_{j,g}\ge|k_{m,j}-k_{n,j}|$ and the reverse triangle inequality. Appendix I.1/Table 3 calibrates this per-channel approximation on Qwen3-8B layer 18 (post-RoPE Keys before CR/OTS; 16 Paul Graham Essays sequences of 4,096 tokens; G32, asymmetric RTN). At INT2, predicted and measured MSE are **0.227634 and 0.226362** (measured/predicted: **0.9944**), supporting the approximation in this setting. Section 3.2 and Appendix J (Eqs. 14–15) explain the complementary roles of CR and OTS. When low-norm tokens have relatively uniform entries while other tokens concentrate energy in a few channels, scaling alone can make the low-norm tokens relative outliers in the other tokens' weak channels. CR redistributes channel energy while preserving token norms; OTS balances norms in the quantization representation and retains the original norms for magnitude restoration in attention. Figure 5 and Appendix K (Figures 22–24) visualize these channel and norm changes, supporting the simple, training-free joint design. **Response to W2: Scope and attribution of the TurboQuant+ comparison.** Thank you for raising this point. With no official TurboQuant implementation available at evaluation time, we used open-source TurboQuant+ (llama.cpp; Appendix O). Table 14 compares its H20 deployment performance with OScaR (PyTorch), reflecting both method and runtime effects. We will clarify this scope in the caption and main text. Appendix W.1/Table 15 quantifies CR/OTS overhead using an INT2 control on the same BitDecoding backend. This KIVI-style control uses per-channel Keys and per-token Values with CR/OTS disabled; it is not the official KIVI runtime: | Context | BF16 FlashDecoding | INT2 control | OScaR | |---|---:|---:|---:| |4K|28.94|32.25|34.09| |16K|29.80|32.15|33.89| |64K|54.29|34.36|35.38| |128K|91.40|47.89|48.90| Entries are TPOT in ms/token for Qwen3-8B on one A800 80GB GPU, batch 1, with identical inputs and positions and 320 generated tokens. We report medians of five requests after warmup; TPOT averages the subsequent 319 decoding steps, including cache update/flush and attention. At 64K/128K, OScaR adds **3.0%/2.1%** TPOT over the INT2 control and reduces TPOT by **34.8%/46.5%** relative to BF16, showing modest fused CR/OTS overhead and deployment benefits at the evaluated long contexts. Extending comparisons to a common open-source runtime with aligned models and deployment configurations would be useful future work. Responses to W3/Q1 and Q3 continue in Part 2.
AuthorsOpenReview ↗

Response to Reviewer GnTf (Part 2/2)

Continuing our response with model/context coverage and related work:

Response to W3 and Q1: Model and context coverage. Thank you for this suggestion. To extend the text-only study beyond Llama-3.1-8B and Qwen3-8B, we report three LongBench-E tasks on Qwen3-14B, covering all 824 examples per configuration from these tasks:

TaskExamplesBF16OScaR
HotpotQA30012.9812.74
Qasper22411.0110.86
RepoBench-P30060.0860.38
Three-task average—28.0327.99

OScaR uses K2V2, G32/R128; both configurations share weights, raw prompts, input tokens, SDPA prefill, a target 32K input cap, greedy generation, and scoring. The average is the unweighted task mean.

On these three tasks, all scores are within 0.30 points of BF16, adding INT2 quality evidence at 14B within Qwen3 and complementing Section 4's Qwen3-VL and Qwen3-Omni evaluations.

For context coverage, Appendix P/Figure 29 evaluates Qwen3-8B across 15 insertion depths and context lengths up to approximately 41K tokens, with NIAH retrieval accuracy of 96.5% for OScaR and 96.0% for BF16. The 128K results are separate latency measurements.

Further evaluation on newer text-only models and different attention structures, including hybrid/linear attention, would help assess TNI and OScaR's applicability.

Response to Q3: Relation to prior norm-based studies. Thank you for these references. Quantize What Counts relates spectral/Frobenius norms in Key/Value projection weights to quantization sensitivity and precision allocation, with token-wise and channel-wise evaluations. The L2-norm strategy uses token-level Key norm/attention relationships to guide KV retention.

OScaR studies within-group token-norm disparities under grouped per-channel Key quantization, where tokens share each channel's range. Section 3.1 and Appendix G characterize inter-modality norm gaps and large-norm outliers. Appendix H derives a reconstruction-error lower bound in terms of norm disparity under a uniform-noise approximation, complemented by Appendix I's empirical error comparisons.

Our contribution connects this shared-range analysis to a simple, training-free CR/OTS design and efficient fused implementation. CR redistributes channel energy; OTS balances token norms for quantization and retains the original norms for magnitude restoration in attention. We will cite these works and distinguish empirical observations, error analysis, and method design in Section 3.1 and related work.

Thank you again for helping us make the conditions, evidence, and contributions more precise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.