acceptodds
Under review as a conference paper at ICLR 2027

ROKI: RoPE-Aware KV Cache Quantization via Isofrequency Grouping

Abstract

Low-bit KV-cache quantization reduces the memory and bandwidth costs of long-context LLM inference, but post-RoPE per-token quantization is sensitive to large activations that dominate the shared quantization range. Cross-channel rotations can reduce these outliers, yet general cross-pair rotations do not commute with Rotary Position Embedding (RoPE) and therefore cannot be folded into Q/K projection weights. We propose ROKI, a RoPE-aware KV-cache quantization framework based on Isofrequency Commutant Expansion (ICE). Rather than restricting transformations to the fixed RoPE operator, ICE selectively ties nearby low-frequency RoPE pairs, creating isofrequency groups that enlarge the exact RoPE-commuting transformation space from pair-wise to cross-pair rotations. ROKI uses this expanded space to redistribute Q/K activations and applies transformation-aware mixed-precision key quantization based on post-transformation QK contributions. All Q/K transformations are optimized offline and folded into the projection weights, requiring no additional online transformation. At an average 3-bit KV-cache budget, ROKI achieves the highest average LongBench-E score among the evaluated quantized baselines on three Llama/Qwen models while remaining close to FP16. Our ROKI implementation also achieves up to 4.22 decode speedup over the evaluated low-bit baselines at long contexts.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache quantization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_D5dxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ROKI, a RoPE-aware KV cache quantization method. The authors identify a limitation of existing weight-foldable transformations with RoPE: with distinct RoPE frequencies, rotations are restricted to individual pairs and cannot redistribute large activations across pairs. ROKI selectively ties nearby low-frequency RoPE pairs to a shared frequency, enabling compatible group-wise rotations that can be folded into the projection weights. It further allocates 2/4-bit key precision using post-transformation QK contributions and optimizes reciprocal QK scaling. At a nominal average 3-bit KV cache budget, ROKI achieves the best average LongBench-E scores among the evaluated quantized baselines on three models. Its implementation maintains decode latency close to or slightly better than FP16, and reports up to 4.22x speedup over the low-bit baselines at long contexts.

Critical strengths
  1. The main idea is technically interesting and well motivated. Instead of accepting the pair-wise restriction imposed by the original RoPE frequencies, the authors modify a selected low-frequency subspace to enable richer transformations without additional online QK transformations. The mathematical construction explains why these rotations commute with the modified RoPE, while the reconstruction objective explicitly accounts for the distortion introduced by frequency tying. The ablations and context-length analysis provide useful evidence that the quantization benefit outweighs this distortion in the evaluated settings.

  2. The authors translate the algorithmic idea into a practical implementation. ROKI combines competitive low-bit accuracy with a regular cache layout and an optimized attention kernel. Maintaining roughly FP16-level decode latency while substantially reducing KV cache storage is a meaningful result.

Critical weaknesses
  1. The offline preparation cost is insufficiently quantified. Group search evaluates candidate configurations across context lengths up to 128K, followed by 1,024 rotation optimization steps and another 1,024 scaling optimization steps. However, the paper does not report the total calibration time needed for this procedure. These costs matter for judging how easily the method can be applied to newer and larger models, especially compared with tuning-free KV cache quantization methods.

  2. The efficiency evaluation has limited coverage. Decode latency is measured only for one model on an RTX 3090 at batch size 1, with no other batch size results, and QServe/FPTQuant are excluded, but they should be key baselines because their transformations are also foldable. A comparison with these methods would still help establish the overall efficiency tradeoff.

Questions for authors
  1. What is the total offline preparation time required for each evaluated model, and how does it break down into ICE group search, rotation optimization, and scaling optimization? Please also specify the hardware used.

  2. How does ROKI compare with QServe and FPTQuant in decode latency? How do the latency comparisons change at larger batch sizes?

1 comment
AuthorsOpenReview ↗

Response to Comments on Offline Preparation Cost and Decode Latency

We thank the reviewer for the valuable feedback on offline preparation cost and decode latency.

Q1. Offline preparation time, stage-wise breakdown, and hardware

We measured ICE group search, rotation optimization, and scaling optimization for each model using eight NVIDIA GeForce RTX 3090 GPUs.

ModelICE searchRotation opt.Scaling opt.Total
Llama-3.1-8B-Instruct12m 32s28m 8s14m 23s55m 3s
Qwen3-8B14m 23s48m 10s23m 2s1h 25m 35s
Qwen3-14B29m 30s1h 18m 23s30m 25s2h 18m 18s

These times exclude generation of FP16 reference outputs, which depends on calibration data and sequence length.

For context, on the same hardware and Llama-3.1-8B-Instruct, the official SpinQuant implementation took 8h 55m for 100 rotation-optimization steps, whereas ROKI's three reported stages totaled 55m 3s. SpinQuant optimizes next-token prediction loss by propagating through the full model, while ROKI minimizes attention-output MSE and optimizes transformations at the attention level without full-model backpropagation. These are not identical optimization workloads; this comparison contextualizes offline preparation cost rather than establishing a direct speed ranking. Since both procedures are performed offline and transformations are reused during inference, even a longer optimization time need not be prohibitive. ROKI's reported preparation stages completed within about 2h 20m for all tested models.

Q2. Decode latency vs. QServe/FPTQuant and larger batch sizes

Figure 4 originally focused on online Q/K transformation overhead. On Llama-3.1-8B-Instruct, ROKI, FPTQuant, and QServe all fold Q/K transformations into projection weights, eliminating a separate online transformation. We added context-length and batch-size latency results to Appendix F.5.

Context-length sweep (batch size 1; ms/token):

ContextROKIFPTQuantQServe
4K19.7619.9619.99
16K21.9922.1522.08
32K24.7324.6324.75
64K29.9329.9229.90
128K40.0140.1640.13

The three methods show comparable latency across all measured context lengths.

Batch-size sweep (16K context; ms/decoding step):

MethodB=1B=2B=4B=8
FP1621.9927.0333.69OOM
ROKI21.9925.7831.2241.71
FPTQuant22.1525.8431.2741.58
QServe22.0825.7731.1241.74
QuaRot28.5337.8655.4790.65
KIVI 3-bit46.0675.01131.85252.52
Block-GTQ58.38100.03180.56351.80

At B=8, FP16 runs out of memory, while ROKI takes 41.71 ms/step: 2.17×, 6.05×, and 8.43× faster than QuaRot, KIVI 3-bit, and Block-GTQ, respectively. Its speedup over Block-GTQ increases from 2.65× at B=1 to 8.43× at B=8. ROKI remains comparable to weight-folded FPTQuant and QServe while substantially outperforming the other evaluated low-bit baselines, particularly at larger batches.

Reviewer_vZu2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ROKI, a method for post-RoPE KV-cache quantization. Its core component, Isofrequency Commutant Expansion (ICE), assigns a shared frequency to each group of selected low-frequency RoPE pairs, enabling cross-pair rotations that commute with the modified RoPE. The method further combines pair permutation, reciprocal Q/K scaling, and mixed-precision quantization to reduce quantization error. The authors claim that rotation can be folded offline into the projection weights and present experiments evaluating quantization quality and decoding speed.

Critical strengths

The core idea is novel and well motivated. Tying RoPE frequencies expands the space of commuting rotations, offering a new way to fold cross-pair rotations into projection weights. The empirical analysis further supports targeting low-frequency pairs, where large Q/K activations tend to occur and frequency gaps are smaller.

Critical weaknesses

(1) Weight folding for Qwen3 needs clarification. Qwen3 applies RMSNorm with learned channel-wise gains between the Q/K projections and RoPE, but Eq. (23) does not explain how these layers are handled. Since offline folding is a key advantage of ROKI, the authors should explain how RMSNorm is handled in their Qwen3 implementation.

(2) Frequency tying needs stronger long-range validation. Table 3 shows increasing FP16 reconstruction error and a narrowing quantized MSE advantage on Qwen3 as context length grows. The LongBench-E and Llama NIAH results provide useful supporting evidence, but the context-length MSE analysis in Appendix C.1 uses fixed Native FP16 layer inputs, and NIAH covers only Llama at 2K–80K. The end-to-end effect of frequency tying on Qwen3 at 64K–128K therefore remains insufficiently established.

Questions for authors

Q1. How does the weight folding in Eq. (23) account for the RMSNorm layers between the Q/K projections and RoPE in Qwen3?

Q2. Could the authors extend the frequency-tying-only evaluation in Appendix C.5 to long-context tasks, particularly Qwen3 at 64K–128K, and report performance changes relative to the original FP16 model at each context length?

Q3. Could the authors add a Pair-wise + P baseline to Table 5? Since adding permutation yields noticeable gains, this comparison would help isolate the additional benefit of ICE when both methods use P.

1 comment
AuthorsOpenReview ↗

Response to Comments on Qwen3 RMSNorm, Long-Context Validation, and Ablation

We thank the reviewer for the constructive questions on Qwen3 RMSNorm, long-context frequency tying, and the Pair-wise + P baseline.

Q1. Weight folding with Q/K RMSNorm in Qwen3 (Eq. 23)

Eq. (23) describes directly folding Q/K transformations into projection weights WQW_Q and WKW_K in Llama. In Qwen3, per-head RMSNorm lies between the projections and RoPE, preventing the same direct folding of the entire transformation into projection weights.

For a head projection y=Wxy=Wx and learned gain matrix Γ=diag⁡(γ)\Gamma=\operatorname{diag}(\gamma), Qwen3 computes

RMSNorm⁡(y)=ΓyRMS⁡(y).\operatorname{RMSNorm}(y)=\Gamma\frac{y}{\operatorname{RMS}(y)}.

Moving the full ROKI transformation before RMSNorm would generally change the input to the RMS denominator. We therefore preserve RMS computation on the original projection output and precompose the gains of q_norm.weight and k_norm.weight with the ROKI transformations (Eq. 22):

MQ=SPRΓQ,MK=S−1PRΓK.M_Q=SPR\Gamma_Q,\qquad M_K=S^{-1}PR\Gamma_K.

The Q/K outputs immediately before RoPE become

q′=MQWQxRMS⁡(WQx),k′=MKWKxRMS⁡(WKx).q'=M_Q\frac{W_Qx}{\operatorname{RMS}(W_Qx)},\qquad k'=M_K\frac{W_Kx}{\operatorname{RMS}(W_Kx)}.

Qwen3 already performs Q/K RMSNorm every token and layer. We fuse these precomposed transformations into the existing RMSNorm kernels, at the stage where channel-wise gains would otherwise be applied; no separate Q/K transformation kernel launch is required.

Qwen3-8B implementationAggregated runtime, 36 layers
Original RMSNorm92.16 μ\mus
ROKI-fused RMSNorm93.46 μ\mus
Additional runtime1.30 μ\mus

The additional 1.30 μ\mus is approximately 0.0067% of the measured 19.31 ms end-to-end decode latency at 4K under the same setting. We clarify architecture-specific transformation placement and inference computations in Appendix F.3.

Q2. Frequency-tying-only FP16 at 64K–128K vs. Native FP16

To extend the PPL and reconstruction-error analysis in Appendix C.5, we evaluated Qwen3-8B on NIAH with only the selected RoPE frequencies tied (no quantization or other transformations) and compared it with Native FP16.

ContextICE FP16 attn.-output MSE ↓Native FP16 NIAH ↑Tying-only FP16 NIAH ↑
64K7.369×10−47.369\times10^{-4}70.0%70.0%
128K5.005×10−35.005\times10^{-3}70.0%70.0%

The ICE FP16 reconstruction MSE increases from 64K to 128K. In the frequency-tying-only NIAH evaluation, Native FP16 and tying-only FP16 achieved the same average accuracy (70.0%) at both lengths. NIAH evaluates the correctness of the final response, so numerical differences in attention outputs need not translate into changes in retrieval accuracy. We added the NIAH results to Appendix E.1 (Figure 11).

Q3. Pair-wise + P baseline in Table 5

On Qwen3-8B, we compared ICE and Pair-wise when both use importance-aware permutation (P). Results are added to main Table 5 and Appendix C.7.

ConfigurationAttn.-output MSE ↓PPL ↓
FP16—7.27
Pair-wise0.0358787.41
ICE0.0293777.36
Pair-wise + P0.0255547.35
ICE + P (ROKI)0.0215227.31

With P in both methods, ICE + P yields 15.78% lower attention-output MSE than Pair-wise + P (0.025554 to 0.021522), while PPL improves from 7.35 to 7.31. ICE therefore provides a measurable benefit beyond the permutation alone.

Reviewer_aNmxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ROKI, a KV cache quantization method that is designed around RoPE. For a chosen set of low-frequency coordinate pairs it ties them to one shared frequency, so that rotations mixing those pairs can be absorbed into the existing projection weights. On top of that it permutes the pairs so that mixed precision quantization works better.

The design is answering a genuine limitation. When every RoPE pair has its own frequency, any rotation that can be absorbed into the weights has to act on each coordinate pair seperately, so it has no way to spread a pair with large norm over the other pairs. Shrinking the keys doesn't help for free either, since the matching queries then have to be scaled up by the same amount. Sec 2.1 lays out both constraints and explains why they hurt quantization. The closest prior work, FPTQuant is already among the baselines.

Table 5 is informative. With the same mixed 2/4-bit key setting on Qwen3-8B, swapping pair-wise rotations for the grouped construction lowers attention-output MSE and perplexity. With the pair rearrangment added, and scaling reoptimized for the new layout, perplexity further improves. So the transformation on it's own is doing real work, separate from the rest of the pipeline.

In the broader evaluation the gains shrink. Table 2 covers Llama-3.1-8B-Instruct, Qwen3-8B and Qwen3-14B. Compared to the best quantized baseline, ROKI raises the LongBench-E average on the three models. The margins are small but they point the same way on all three.

What I'd most like clarified is what tying the frequencies does to the model. We see some numbers in Table 1, but further details would be helpful. Once the frequencies are tied, the rotation leaves attention scores unchanged, but the tying itself is a change to the original model, wich the authors do acknowledge as a positional distortion. Table 3 also shows that the reconstruction-error gain over pair-wise rotation drops from ~21 % at 2K to ~10 % at 128K.

Sec 4.2 gives some retrieval evidence over contexts from 2K to 80K. I'd have liked to see a position-sensitive task at 128K too, because at that length Table 3 only reports reconstruction error. An easier-to-quantize cache only pays off if the modified positional encoding keeps task accuracy.

The timing results could be filled out more (Fig 4). FPTQuant and QServe are in the accuracy tables but missing here.

I lean towards accept as the submission addresses a relevant problem, proposes a principled sulition, and the ablation shows that the change is responsible for the improvement. I'd still want a clearer treatment of the issues raised above.

Further related work:

  • RotateKV, IJCAI 2025.
Critical strengths

See above

Critical weaknesses

See above

Questions for authors

See above

1 comment
AuthorsOpenReview ↗

Response to Comments on Frequency Tying, 128K Retrieval, and Latency

We thank the reviewer for the constructive feedback on frequency tying, 128K retrieval, and decode latency.

1. Effect of frequency tying on the original model

ROKI assigns a shared frequency to selected low-frequency RoPE pairs within each group. For pairs whose frequencies change, the relative rotation angle between tokens can differ from that in the original model, potentially changing attention scores.

To isolate this effect, we compare Native FP16 with frequency-tying-only FP16, disabling quantization, rotation, scaling, and permutation (Appendix C.5, Table 13). Attention-output MSE is measured relative to Native FP16.

ModelConfigurationAttn.-output MSE ↓PPL ↓
Llama-3.1-8B-InstructNative FP1606.23
Llama-3.1-8B-InstructTying-only FP164.60437×10−74.60437\times10^{-7}6.24
Qwen3-8BNative FP1607.27
Qwen3-8BTying-only FP162.58409×10−42.58409\times10^{-4}7.27

Frequency tying changes attention outputs but has little effect on PPL. On Qwen3-8B, tying-only FP16 also matched Native FP16 in average NIAH accuracy at 2K, 32K, 64K, and 128K. NIAH evaluates correctness of the final response, so numerical differences in attention outputs need not change retrieval accuracy. The results for each context length are presented in our response to Question 2 below.

A change in RoPE frequency changes the relative rotation angle in proportion to the distance between two tokens, so frequency tying can introduce greater positional distortion over longer distances. The increase in ICE's FP16 reconstruction MSE with context length in Table 3 is consistent with this interpretation.

The percentage gain also depends on the pair-wise baseline at each context length. From 64K to 128K, ICE's quantized attention-output MSE decreased from 0.030970 to 0.029612, while the pair-wise MSE decreased by a larger amount, from 0.037434 to 0.033120. Thus, ICE's relative MSE reduction narrowed from 17.27% to 10.59%. Because the reference is Native FP16, the reported MSE includes frequency-tying distortion. Even with this effect included, ICE achieved lower MSE than pair-wise rotation at every evaluated context length.

2. Position-sensitive retrieval at 128K

Thank you for suggesting this evaluation. We extended NIAH on Qwen3-8B to 128K (Appendix E.1, Figure 11). At 128K, ROKI achieved 60.0% accuracy, compared with 53.3% for KIVI and 43.3% for Block-GTQ; Native FP16 and tying-only FP16 both achieved 70.0% on average across insertion positions.

Native FP16Tying-only FP16ROKIKIVIBlock-GTQ
70.0%70.0%60.0%53.3%43.3%

Qwen3-8B, 128K NIAH average accuracy across insertion positions.

The following table combines reconstruction-error results (extending main Table 3 with 64K) and NIAH accuracy at four context lengths. MSE reduction is relative to quantized Pair-wise.

LengthICE FP16 MSE ↓Pair-wise MSE ↓ICE MSE ↓ReductionNative NIAH ↑Tying-only NIAH ↑ROKI NIAH ↑
2K3.505×10−63.505\times10^{-6}0.0403280.03167421.46%100.0%100.0%100.0%
32K2.589×10−42.589\times10^{-4}0.0358780.02937718.12%86.7%86.7%83.3%
64K7.369×10−47.369\times10^{-4}0.0374340.03097017.27%70.0%70.0%70.0%
128K5.005×10−35.005\times10^{-3}0.0331200.02961210.59%70.0%70.0%60.0%

Appendix E.1 reports 64K/128K NIAH for all five configurations (Native FP16, tying-only FP16, KIVI, Block-GTQ, and ROKI).

3. Decode latency comparison with FPTQuant and QServe (Fig. 4)

Thank you for suggesting this comparison. We measured end-to-end decode latency on Llama-3.1-8B-Instruct over 4K–128K contexts and added the results to Appendix F.5. Figure 4 originally focused on online Q/K transformation overhead; ROKI, FPTQuant, and QServe instead fold their Q/K transformations into projection weights for this Llama model, requiring no separate transformation during decoding.

ContextROKIFPTQuantQServe
4K19.7619.9619.99
16K21.9922.1522.08
32K24.7324.6324.75
64K29.9329.9229.90
128K40.0140.1640.13

End-to-end decode latency (ms/token), batch size 1. All three methods exhibit comparable latency, indicating that ICE expands the foldable transformation space without sacrificing this runtime advantage.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the reviewers for their constructive feedback and suggestions for additional experiments.

In post-RoPE KV-cache quantization, large activations in a few channels widen the quantization range and increase error. Because RoPE pairs have distinct frequencies, existing foldable Q/K rotations operate within individual pairs and cannot redistribute large activations across them. ICE, the core technique in ROKI, assigns a shared frequency to each group of nearby low-frequency RoPE pairs. This enables rotations across pairs within each group that commute with the modified RoPE. ROKI uses these rotations to redistribute large Q/K activations, assigns precision to key pairs based on their contributions to the QK dot product after transformation, and integrates the Q/K transformations into existing operations. At an average KV-cache precision of 3 bits, ROKI achieved the highest average LongBench-E score among the evaluated quantization methods on all three models.

Summary of Reviewer Responses

  • Reviewer 1 (aNmx) asked about the effect of frequency tying, retrieval at 128K, and comparisons with additional latency baselines. Frequency tying alone had little effect on perplexity, and ROKI achieved higher 128K NIAH accuracy than KIVI and Block-GTQ. ROKI also maintained decoding latency comparable to FPTQuant and QServe across context lengths from 4K to 128K.
  • Reviewer 2 (vZu2) asked how ROKI handles RMSNorm in Qwen3 and requested additional long-context retrieval results and an ICE ablation. In Qwen3, the Q/K transformations run within the existing RMSNorm kernels. Frequency tying alone matched the original FP16 model’s average retrieval accuracy at both 64K and 128K. With the same permutation strategy in both configurations, ICE reduced attention-output MSE by 15.8% relative to pairwise rotation.
  • Reviewer 3 (D5dx) asked about offline preparation costs and decoding latency at different batch sizes. We reported a breakdown of offline preparation time for all three models and compared decoding latency at batch sizes 1, 2, 4, and 8. At batch size 8, ROKI decoded in 41.71 ms per step, a setting in which FP16 ran out of memory.

Detailed results and experimental settings are available in our individual responses and the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.