acceptodds
Withdrawn from ICLR 2027

Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long inputs and extended generation increase the storage and access costs of the key–value (KV) cache. Low-bit quantization reduces storage and memory traffic, while query-channel pruning can further reduce key-cache reads. Rotation-based quantization redistributes the energy of key outliers across channels. To maintain computational invariance, the same orthogonal transform must be applied to queries, preserving query–key dot products. However, this rotation can disperse query energy, weakening the separation between a few large components to retain and many small ones to prune. We introduce Dual-QK, which uses paired non-orthogonal query and key transforms to address this conflict. Using calibrated query and key statistics, Dual-QK combines partial key whitening with a query-aligned basis to balance key scales for INT2 quantization and concentrate query energy for dynamic channel pruning. Channel-0 protection and bucket-relative RoPE support low-bit accuracy over long contexts. Experiments on four models across five generative benchmarks and long-context retrieval tasks show improved accuracy over OSCAR on most tasks at 40% query-channel sparsity. At a 128K context, Dual-QK provides KV-cache compression and an estimated reduction in KV read volume relative to unpruned BF16. Under the evaluated configurations, our SGLang implementation achieves up to the decoding throughput of unpruned BF16.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache, quantization, sparse attention, query-channel pruning, long context LLM inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_g5tcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dual-QK, which aims to simultaneously achieve low-bit KV cache quantization and dynamic channel pruning. The authors point out that while rotation-based quantization usually mitigates Key outliers by flattening their distribution, the shared rotation can also disperse Query energy, reducing the effectiveness of channel pruning based on Query magnitudes. To address this, Dual-QK uses paired non-orthogonal transforms that preserve the attention dot product: it applies partial whitening to the Key, and then applies PCA to the reversely compensated Query, making Key scales more balanced and Query energy more concentrated. During inference, the method dynamically selects the Key channels to read based on the channel energy of the current GQA Query, while retaining the full cache. The system also combines INT2 quantization, BF16 protection for the leading Key channel, diagonal scale calibration, and bucket-relative RoPE to improve quantization accuracy and long-context adaptability. The paper evaluates generation quality, long-context retrieval, and decoding throughput on four models. Under a uniform setting of 40% channel pruning, Dual-QK generally maintains accuracy better than the compared quantization baselines and achieves speedups by reducing Key cache reads. Its main contribution is a joint algorithm and system design targeting the compatibility of quantization and channel pruning.

Critical strengths

S1. The motivation of the paper is relatively novel and intuitive.

S2. The practical performance of the method is good.

Critical weaknesses

W1. Limited technical novelty: The method is essentially a combination of several existing techniques, which can be broken down as follows: 1) Using different transforms for Q and K (FPTQuant 2025); 2) Applying a fractional power transform to the Key eigenvalues to reduce variance, which is a well-established classical operation; 3) Applying PCA to the Query, which is also a standard classical operation; 4) Channel pruning, token protection, etc., for which the authors explicitly stated in the text that they referred to papers like OSCAR; 5) Bucket RoPE appears to be the only original component (no prior work found), but its contribution is not very fundamental. Therefore, the overall novelty of this work is questionable.

W2. Misleading experimental setup in the main table: The main comparison applies query-channel pruning uniformly to all SOTA baselines. However, methods like OSCAR are inherently not robust to this level of pruning, causing their scores to plummet. As shown in the ablation study (Table 4), when pruning is removed, OSCAR comprehensively outperforms this method on Qwen models. Comparing under a setting where baselines are forced to fail is a trick to make the proposed method look better.

W3. Insufficient experimental evaluation: The overall experiments are somewhat thin. Apart from the main table, the subsequent tables almost exclusively compare with OSCAR, missing several relevant baselines in the literature. For example, the authors could include comparisons with methods like Kitty and MixKVQ.

Questions for authors

Please explain the weaknesses above.

Reviewer_FZFmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies and resolves a fundamental distributional conflict between two complementary KV-cache optimizations: low-bit key quantization (INT2) and dynamic query-channel pruning.

  • Key quantization favors a flat channel distribution across keys so that channel outliers do not widen the shared quantization range. Rotation-based methods (e.g., QuaRot, SpinQuant, OSCAR, TurboQuant) achieve this by applying a shared orthogonal rotation Q→QH,K→KHQ \to QH, K \to KH, which preserves dot products (QH)(KH)⊤=QK⊤(QH)(KH)^\top = QK^\top but disperses query energy across all channels, destroying the sparsity needed for query-channel pruning.
  • Conversely, rotating into the principal components of query second moments (Q-PCA, Q→QU,K→KUQ \to QU, K \to KU) concentrates query energy into a few leading channels for pruning, but leaves key outliers intact, degrading INT2 quantization.

To resolve this conflict, the authors propose Dual-QK, which applies paired non-orthogonal reciprocal transforms A=Gα/2UDA = G^{\alpha/2} U D to queries and B=A−⊤=G−α/2UD−1B = A^{-\top} = G^{-\alpha/2} U D^{-1} to keys:

  1. Partial Key Whitening & Query-Aligned Basis: Using offline calibration statistics CQ=E[q⊤q]C_Q = \mathbb{E}[q^\top q] and CK=E[k⊤k]C_K = \mathbb{E}[k^\top k] per layer and KV head, Dual-QK forms G=CK+ϵIG = C_K + \epsilon I, scales keys by G−α/2G^{-\alpha/2} and queries by Gα/2G^{\alpha/2} (α∈[0,1]\alpha \in [0, 1]), and chooses orthogonal UU from the eigendecomposition Gα/2CQGα/2=UΛU⊤G^{\alpha/2} C_Q G^{\alpha/2} = U \Lambda U^\top, followed by a calibrated diagonal scaling DD that minimizes query-weighted INT2 key error.
  2. Channel-0 Protection & Dynamic Group-L2L_2 Channel Pruning: Because base channel 0 captures the largest expected query energy λ0\lambda_0 (48.3%–87.9% across models, Figure 5e–h), Dual-QK keeps key channel 0 in BF16 and excludes it from the per-token INT2 min/max range calculation. During decoding, each GQA group dynamically selects the top-mm channels by group L2L_2 query norm ej=∑h=1g(qh,j′)2e_j = \sqrt{\sum_{h=1}^g (q'_{h,j})^2} from a channel-major paged key layout.
  3. Bucket-Relative RoPE: Because non-orthogonal transforms A,BA, B do not commute with RoPE, Dual-QK partitions the context into w=2048w = 2048-token buckets, undoing the bucket-anchor rotation TbT_b before transforming stored keys (zn=knTbBz_n = k_n T_b B) and applying TbAT_b A to queries per bucket (q~b=qTbA\tilde{q}_b = q T_b A).

Evaluated on 4 models (Llama-3.1-8B-Instruct, Qwen3-4B-Thinking-2507, Qwen3-8B, Ministral-3-14B-Reasoning-2512) across 5 reasoning/coding benchmarks (GPQA-Diamond, HumanEval, LiveCodeBench v6, MATH-500, AIME25), RULER NIAH (4K–128K), and LongBench, Dual-QK substantially outperforms OSCAR, KIVI, TurboQuant, and ThinK at 40% query-channel sparsity, while achieving up to 3.75×3.75\times decoding throughput over unpruned BF16 in SGLang.

Critical strengths
  1. Clean Formulation of the Quantization–Pruning Basis Conflict via Reciprocal Transforms: Applying reciprocal transforms A=Gα/2UDA = G^{\alpha/2} U D and B=A−⊤=G−α/2UD−1B = A^{-\top} = G^{-\alpha/2} U D^{-1} (Equation 7: A0⊤CQA0=ΛA_0^\top C_Q A_0 = \Lambda and B0⊤CKB0=U⊤(G1−α−ϵG−α)UB_0^\top C_K B_0 = U^\top(G^{1-\alpha} - \epsilon G^{-\alpha})U) provides a straightforward linear-algebraic knob α∈[0,1]\alpha \in [0, 1] to trade off key-scale balancing against query-energy concentration. While non-orthogonal query-informed transforms and reciprocal channel scaling have close precedents in NOVA-KV (Fernández-Menduiña et al., 2026) and QServe (Lin et al., 2025b), combining partial key whitening with channel-0 BF16 protection and bucket-relative RoPE (w=2048w = 2048) is a practical engineering recipe that avoids the severe accuracy collapse suffered by orthogonal Hadamard rotations when subjected to 40% query-channel pruning (Table 3).

  2. Systems Implementation with Sector-Aligned Channel-Major Paging in SGLang: Rather than only simulating pruning masks in PyTorch, the authors implement a 128-token channel-major paged layout (where each INT2 key channel occupies a contiguous 32-byte sector-aligned row) in SGLang, demonstrating concrete memory-capacity savings and decoding throughput gains over unpruned BF16 at 90K context (Figure 4, Table 10).

Critical weaknesses
  1. Missing Unified Accuracy–Latency Comparison (40%-Pruned Accuracy in Table 1 vs. 0%-Pruned OSCAR Throughput in Figure 4, and Omission of 0%-Pruned Dual-QK Speed):

    • In the main accuracy tables (Tables 1, 2, and 12), OSCAR and TurboQuant are evaluated under 40% dynamic query-channel pruning in their Hadamard/randomly rotated coordinates. While this equal-sparsity comparison is a valid diagnostic test of pruning compatibility (confirming that orthogonal Hadamard mixing is incompatible with magnitude-based channel pruning, e.g., on Qwen3-4B-Thinking-2507, OSCAR's HumanEval score drops from 96.1% at 0% pruning in Appendix Table 11 to 33.9% at 40% pruning in Table 1, and AIME25 drops from 64.7% to 6.0%), it is an insufficient practical baseline because rotation-based quantizers are intended to be deployed unpruned.
    • Conversely, in the decoding throughput evaluation (Figure 4), OSCAR is run unpruned (0% sparsity) while Dual-QK is run at 40% sparsity—creating an apples-to-oranges mismatch between the accuracy tables and the throughput plots.
    • If we compare the two configurations as they are actually evaluated in Figure 4—namely, 0%-pruned OSCAR (Appendix Table 11) vs. 40%-pruned Dual-QK (Table 1)—then 0%-pruned OSCAR achieves higher accuracy than 40%-pruned Dual-QK across all 5 benchmarks on both Qwen3-4B-Thinking-2507 and Qwen3-8B at slightly lower storage cost (2.28 vs. 2.34 Capacity BPE):
      • On Qwen3-4B-Thinking-2507, 0%-pruned OSCAR achieves 63.5% GPQA, 96.1% HumanEval, 46.7% LCB v6, 96.4% MATH-500, and 64.7% AIME25, outperforming 40%-pruned Dual-QK (60.8%, 95.0%, 38.8%, 94.3%, and 48.0%) by +16.7 points on AIME25 and +7.9 points on LCB v6.
      • On Qwen3-8B, 0%-pruned OSCAR achieves 56.6% GPQA, 92.2% HumanEval, 46.1% LCB v6, 95.8% MATH-500, and 61.3% AIME25, outperforming 40%-pruned Dual-QK (55.7%, 90.6%, 36.2%, 95.0%, and 53.3%) by +9.9 points on LCB v6 and +8.0 points on AIME25.
      • Even when neither method is pruned (0% sparsity, Tables 4 and 11), OSCAR still outperforms Dual-QK on Qwen3-8B average accuracy (70.4% vs. 69.5%) and on Qwen3-4B-Thinking AIME25 (64.7% vs. 54.0%, a 10.7-point drop).
    • Crucially, Figure 4 and Table 10 omit Dual-QK at 0% pruning, making it impossible to separate how much of Dual-QK's throughput advantage over 0%-pruned OSCAR comes from kernel/layout differences versus 40% channel pruning, or whether the small speedup from 40% pruning justifies losing 6.0–8.0 points on AIME25 and 4.1–7.0 points on LCB v6.
  2. Marginal Realized DRAM Traffic Reduction (14–15%) from 40% Key-Channel Pruning vs. Accuracy Cost and Per-Bucket Overheads:

    • Because 40% query-channel pruning only skips INT2 key channels while reading 100% of value channels, quantization metadata, and BF16 sink/recent tokens, theoretical read volume drops by only 17.5% from 0% to 40% sparsity (2.34→1.932.34 \to 1.93 BPE), and Nsight Compute (Appendix D.1, line 953) measures only 14–15% fewer attention-kernel DRAM reads at 40% vs. 0% sparsity across 30K–90K contexts and batch sizes 1–64.
    • Furthermore, because channel masks must be recomputed per 2,048-token bucket (q~b=qTbA0\tilde{q}_b = q T_b A_0, Section 3.4), a 90K–128K context requires executing 45–64 separate bucket query transforms and top-mm group-L2L_2 selections per layer at every decode step (contributing 0.4–2.8 ms of step overhead in Table 10, which the authors acknowledge in Section 5 line 471 can offset savings at small batch sizes).
  3. Substantial Long-Context Retrieval Degradation at 64K–128K (Table 2), Sensitivity of Channel-0 Concentration, and Test-Prompt Reuse in the Table 5 Ablation:

    • 128K RULER NIAH degradation: In Table 2, Dual-QK at 128K drops by 19.5 to 35.9 percentage points below BF16 across Llama-3.1-8B (94.8%→58.9%94.8\% \to 58.9\%), Qwen3-4B-Thinking (94.0%→64.4%94.0\% \to 64.4\%), and Qwen3-8B (76.5%→57.0%76.5\% \to 57.0\%). As noted in Section 5 (lines 474–476) and Appendix C.4 (lines 930–931), while bucket-relative RoPE bounds stored-key offsets n−ab∈[0,2048)n - a_b \in [0, 2048), query offsets relative to older buckets can still exceed the 2K calibration window, leaving open how much of the 128K degradation stems from 40% pruning versus non-orthogonal quantization error at long relative distances.
    • Concentrating 48.3%–87.9% of query energy into a single channel makes the representation brittle: In Figure 5e–h, channel 0 alone carries up to 87.9% of expected query energy (Llama-3.1-8B) and 90.2% of query-weighted INT2 error before protection (Figure 5i). Without the BF16 channel-0 patch, HumanEval drops by 19.1 points and AIME25 drops by 28.9 points (Table 3).
    • Test-prompt reuse in the Table 5 calibration ablation: While all default results in Tables 1–4 properly calibrate on WikiText-2, Appendix B.2 (lines 797–798) states that the GPQA-D row in the Table 5 calibration-domain ablation reuses the 198 GPQA-Diamond evaluation prompts (questions and answer choices) to fit the calibration moments. Fitting calibration statistics directly on evaluation prompts in that ablation row is test-set leakage and should be avoided or explicitly labeled as an in-sample reference.
Questions for authors
  1. Throughput and Latency Breakdown of 0%-Pruned Dual-QK vs. 40%-Pruned Dual-QK and 0%-Pruned OSCAR: In Table 10 and Figure 4, what is the decoding latency (ms) and throughput (tok/s) of Dual-QK at 0% channel pruning? Given that Nsight Compute measures only a 14–15% reduction in attention-kernel DRAM reads between 0% and 40% sparsity (Appendix D.1) and that 40% pruning requires computing bucket-relative query transforms q~b=qTbA0\tilde{q}_b = q T_b A_0 and top-mm channel masks for every 2K bucket at every decode step, how much wall-clock speedup does 40% pruning actually provide over 0%-pruned Dual-QK, and why is 0%-pruned Dual-QK (or 40%-pruned Dual-QK) faster than 0%-pruned OSCAR in Figure 4 when both read ~1.93 vs. 2.28 BPE?
  2. Source of the 128K RULER NIAH Accuracy Drop (Quantization vs. Pruning): In Table 2, Dual-QK loses 29.6 to 35.9 points relative to BF16 at 128K on Llama-3.1-8B and Qwen3-4B-Thinking. What is the RULER NIAH accuracy at 64K and 128K for 0%-pruned Dual-QK and 0%-pruned OSCAR? Does the 128K drop come primarily from the 40% channel pruning at large query-to-bucket relative distances, or from the non-orthogonal transform B=G−α/2UD−1B = G^{-\alpha/2}UD^{-1} amplifying INT2 quantization error when qTbq T_b is far outside the 2K calibration window?
  3. Why Does Dual-QK Underperform OSCAR on AIME25 at 0% Pruning (Qwen3-4B-Thinking, Table 11)?: At 0% pruning on Qwen3-4B-Thinking-2507 (Table 11), OSCAR achieves 64.7±3.8%64.7 \pm 3.8\% on AIME25 and 46.7±1.3%46.7 \pm 1.3\% on LCB v6, whereas Dual-QK drops to 54.0±4.9%54.0 \pm 4.9\% on AIME25 (-10.7 points) and 42.9±1.0%42.9 \pm 1.0\% on LCB v6 (-3.8 points). Does non-orthogonal key whitening (α=0.5\alpha = 0.5) amplify quantization noise along low-variance key directions that become important during long chain-of-thought reasoning traces?

Suggestions for Future Work

  1. Multi-Range RoPE Query Calibration: In Equation (5), stored keys only see residual positions [0,w)[0, w), but bucket-adjusted queries q~b=qTbA\tilde{q}_b = q T_b A see anchor rotations TbT_b spanning up to 128K. Estimating CQ=E[(qTb)⊤(qTb)]C_Q = \mathbb{E}[(q T_b)^\top (q T_b)] averaged over a logarithmic grid of bucket offsets aba_b (rather than only within a 2K window) would make UU and DD robust to long-range relative RoPE rotations at 64K–128K.
Reviewer_3UZ9OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed Dual-QK, which introduces paired non-orthogonal query-key transforms to resolve conflicts between rotation-based quantization and query-channel pruning, achieving superior accuracy at 40% sparsity and significant KV-cache compression with up to substantial decoding throughput gains over unpruned BF16 at 128K context.

Critical strengths
  1. Using B=A−TB=A^{-T} to decouple the Q/K transforms preserves the dot product while separately serving quantization and pruning; the idea is simple and novel.
  2. The system design is fairly complete, considering INT2 quantization, channel-0 protection, bucket-relative RoPE, and channel-major layout.
  3. Under 40% pruning, generative tasks and long-context NIAH show clear improvements over OSCAR.
Critical weaknesses
  1. The throughput comparison is unfair: Dual-QK uses 40% pruning, while OSCAR/BF16 are unpruned, making the speed advantage hard to attribute.
  2. There is no direct comparison with query-aware sparse attention methods such as SparQ, Double Sparsity, Loki, and SparK.
  3. Bandwidth BPE is idealized; measured DRAM reads drop only about 14–15%, and 128K accuracy is still far below BF16.
Questions for authors
  1. Can the throughput of Dual-QK and OSCAR be compared under the same 40% pruning, same layout, and same kernel?
  2. Can a bandwidth-accuracy comparison with SparQ, Double Sparsity, Loki, SparK, and similar methods be added?
  3. Where does the error at 128K NIAH mainly come from, and what are the time and resource costs of full calibration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.