- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OScaR studies how to compress the KV cache used by large language models during response generation to reduce GPU memory use. It rotates the vectors in the cache and adjusts the norms of vectors for different tokens to reduce errors caused by low-precision storage. The paper provides implementation code and evaluates task performance, GPU memory use, and other metrics after compression on text and multimodal tasks.
Critical strengths
- The method does not require retraining the model.
- Across multiple models and tasks, performance after compression remains close to that of the full-precision models, which has practical value.
- The experiments cover multiple models and tasks. They also examine the role of each component and measure speed, GPU memory use, and other metrics.
Critical weaknesses
-
The paper identifies token norm imbalance as the main bottleneck in per-channel quantization and provides theoretical analysis and quantization-error experiments. However, the experimental evidence is not sufficient to show why it is the main bottleneck or how much it affects different models and tasks.
-
The gains in the main experiments are small, and performance on some tasks is below that of existing methods.
In Table 1, the average scores for Llama-3.1-8B and Qwen3-8B exceed those of OTT by only 1.01 and 0.53 percentage points, respectively. For each model, OScaR also scores below at least one other quantization method on three tasks. For example, Llama-3.1-8B scores below KIVI on HotpotQA, while Qwen3-8B scores below OTT on GovReport and MultiNews. The paper does not further explain why performance is weaker on these tasks.
- The ablation settings differ from those of the main experiments.
Table 12 tests Canalized Rotation, Omni-Token Scaling, and their combination. However, the models, tasks, and settings such as group size differ from those of the main experiments in Table 1. Should the paper also report each component's contribution under the main experimental settings?
- Some experimental results are described repeatedly in the main text and appendix.
For example, Section 4.3 and Appendix W both describe an approximately 3-fold decoding speedup over BF16. I suggest reducing this repetition.
Questions for authors
-
Why does OScaR perform worse than baseline methods such as KIVI, RotateKV, and OTT on some tasks?
-
What is the contribution of each component under the main experimental settings in Table 1?
-
What experimental evidence shows that token norm imbalance is the main bottleneck in per-channel quantization?
We sincerely thank the reviewer for the careful assessment and constructive feedback. Below, we address the shared-range evidence, task-level results, component contributions, and presentation.
Response to W1 and Q3: Evidence for TNI in shared-range per-channel quantization. In grouped per-channel Key quantization, tokens share a range within each channel. Token-magnitude disparities can enlarge this range and coarsen the resolution for other tokens. Appendix H analyzes this mechanism under a uniform-noise approximation. The following submitted results quantify the associated error differences and test the approximation.
Error comparison (Table 2). On LLaVA-v1.5-7B with RTN and G32, Key MSE ×100 is:
At INT2, this comparison shows approximately 68% higher MSE with the outlier tokens under the evaluated grouping conditions.
Real-tensor validation (Table 3). On Qwen3-8B layer 18, we evaluate post-RoPE Keys before CR/OTS using 16 Paul Graham sequences of 4,096 tokens, G32, and the paper's asymmetric RTN rule:
Correlations are computed per sequence (mean ± standard deviation). The MSE agreement and strong correlations support the approximation in this evaluated setting.
These results support TNI as an important structural source of error in extreme per-channel Key quantization and motivate addressing token-norm disparity and channel geometry together. We will replace “the primary bottleneck” with this characterization in the abstract and main text, making the setting and assumptions explicit.
Response to W2 and Q1: Average gains and task-level exceptions. Our interpretation is that these exceptions reflect task-dependent quantization trade-offs: different transforms, grouping schemes, and token-protection policies distribute errors differently, and their impact on attention depends on the queries (Appendices H.1 and J.1).
In Table 1, OScaR achieves the highest eight-task average among the compared quantized methods on both models. Although it does not lead on every task, five of the six gaps to the best competing quantized result are 0.06–0.39 score points, including the highlighted Qwen3-8B GovReport and MultiNews gaps to OTT (0.23 and 0.36 points, respectively). The largest gap occurs on Llama-3.1-8B HotpotQA: OScaR's 13.68 remains close to BF16's 13.76, while KIVI's 14.66 exceeds BF16. On TriviaQA for the same model, OScaR's 90.41 and RotateKV's 90.56 both exceed BF16's 90.15.
The corresponding mean gains over OTT are 1.01 points on Llama-3.1-8B and 0.53 points on Qwen3-8B. Our contribution combines the shared-range analysis, a simple, training-free CR/OTS design, and a fused implementation, with competitive INT2 quality across the evaluated text-only, multimodal, and omni-modal models. We will discuss these task-level trade-offs in the main text.
Response to W3 and Q2: Component contributions at G32/R128. We evaluated four configurations on Qwen3-8B at Table 1's K2V2, G32/R128 settings across all eight LongBench-E tasks (2,318 examples per configuration). Only Key-side CR and OTS vary; Value-side processing and other conditions are fixed. Plain disables both components. All runs share SDPA prefill and a target 32K input cap and are reported separately from the original Table 1.
The combination achieves the highest observed mean, 0.79 points above Plain. CR provides most of the average gain; adding OTS raises the observed mean by 0.12 points, with task-dependent effects. This complements Table 12's multimodal/G128 ablation with component evidence under the main quantization settings. We will add this comparison to the main text.
Response to W4: Reducing repetition. We will reduce overlapping efficiency descriptions in Section 4.3 and Appendix W, keeping the main findings and essential settings in the main text and the detailed measurement protocols and component overhead breakdown in the appendix. Cross-references will replace repeated summaries.
Thank you again for helping us strengthen the evidence and improve the clarity of the paper.