- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes ROKI, a RoPE-aware KV cache quantization method. The authors identify a limitation of existing weight-foldable transformations with RoPE: with distinct RoPE frequencies, rotations are restricted to individual pairs and cannot redistribute large activations across pairs. ROKI selectively ties nearby low-frequency RoPE pairs to a shared frequency, enabling compatible group-wise rotations that can be folded into the projection weights. It further allocates 2/4-bit key precision using post-transformation QK contributions and optimizes reciprocal QK scaling. At a nominal average 3-bit KV cache budget, ROKI achieves the best average LongBench-E scores among the evaluated quantized baselines on three models. Its implementation maintains decode latency close to or slightly better than FP16, and reports up to 4.22x speedup over the low-bit baselines at long contexts.
Critical strengths
-
The main idea is technically interesting and well motivated. Instead of accepting the pair-wise restriction imposed by the original RoPE frequencies, the authors modify a selected low-frequency subspace to enable richer transformations without additional online QK transformations. The mathematical construction explains why these rotations commute with the modified RoPE, while the reconstruction objective explicitly accounts for the distortion introduced by frequency tying. The ablations and context-length analysis provide useful evidence that the quantization benefit outweighs this distortion in the evaluated settings.
-
The authors translate the algorithmic idea into a practical implementation. ROKI combines competitive low-bit accuracy with a regular cache layout and an optimized attention kernel. Maintaining roughly FP16-level decode latency while substantially reducing KV cache storage is a meaningful result.
Critical weaknesses
-
The offline preparation cost is insufficiently quantified. Group search evaluates candidate configurations across context lengths up to 128K, followed by 1,024 rotation optimization steps and another 1,024 scaling optimization steps. However, the paper does not report the total calibration time needed for this procedure. These costs matter for judging how easily the method can be applied to newer and larger models, especially compared with tuning-free KV cache quantization methods.
-
The efficiency evaluation has limited coverage. Decode latency is measured only for one model on an RTX 3090 at batch size 1, with no other batch size results, and QServe/FPTQuant are excluded, but they should be key baselines because their transformations are also foldable. A comparison with these methods would still help establish the overall efficiency tradeoff.
Questions for authors
-
What is the total offline preparation time required for each evaluated model, and how does it break down into ICE group search, rotation optimization, and scaling optimization? Please also specify the hardware used.
-
How does ROKI compare with QServe and FPTQuant in decode latency? How do the latency comparisons change at larger batch sizes?
Response to Comments on Offline Preparation Cost and Decode Latency
We thank the reviewer for the valuable feedback on offline preparation cost and decode latency.
Q1. Offline preparation time, stage-wise breakdown, and hardware
We measured ICE group search, rotation optimization, and scaling optimization for each model using eight NVIDIA GeForce RTX 3090 GPUs.
These times exclude generation of FP16 reference outputs, which depends on calibration data and sequence length.
For context, on the same hardware and Llama-3.1-8B-Instruct, the official SpinQuant implementation took 8h 55m for 100 rotation-optimization steps, whereas ROKI's three reported stages totaled 55m 3s. SpinQuant optimizes next-token prediction loss by propagating through the full model, while ROKI minimizes attention-output MSE and optimizes transformations at the attention level without full-model backpropagation. These are not identical optimization workloads; this comparison contextualizes offline preparation cost rather than establishing a direct speed ranking. Since both procedures are performed offline and transformations are reused during inference, even a longer optimization time need not be prohibitive. ROKI's reported preparation stages completed within about 2h 20m for all tested models.
Q2. Decode latency vs. QServe/FPTQuant and larger batch sizes
Figure 4 originally focused on online Q/K transformation overhead. On Llama-3.1-8B-Instruct, ROKI, FPTQuant, and QServe all fold Q/K transformations into projection weights, eliminating a separate online transformation. We added context-length and batch-size latency results to Appendix F.5.
Context-length sweep (batch size 1; ms/token):
The three methods show comparable latency across all measured context lengths.
Batch-size sweep (16K context; ms/decoding step):
At B=8, FP16 runs out of memory, while ROKI takes 41.71 ms/step: 2.17×, 6.05×, and 8.43× faster than QuaRot, KIVI 3-bit, and Block-GTQ, respectively. Its speedup over Block-GTQ increases from 2.65× at B=1 to 8.43× at B=8. ROKI remains comparable to weight-folded FPTQuant and QServe while substantially outperforming the other evaluated low-bit baselines, particularly at larger batches.