acceptodds
Under review as a conference paper at ICLR 2027

HayateQuant: More Geometry per Bit with Folded Rotations for KV Cache Quantization

Abstract

Compressing the key–value (KV) cache can substantially reduce memory traffic during language-model decoding. However, improved quantization geometry is useful only if it can be realized with low encoding and serving overhead. We investigate whether short-block quantization can improve a data-free cache codec while preserving a compact, fixed-rate representation. We introduce HayateQuant (HQ), which replaces independent scalar assignments with compact joint codebooks. A folded rotation promotes uniform within-block directions, while precomputed candidate lists keep online assignment efficient. To account for the distinct effects of key and value quantization errors, HQ further employs role-specific scaling and key and value bit allocation. Across diverse model families, HayateQuant achieves a favorable quality-compression tradeoff, particularly at moderate rates. Its optimized packed serving implementation also improves decoding throughput over the existing baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
kv cashe quantization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UyEyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

HayateQuant (HQ) replaces TurboQuant’s (TQ) scalar assignments with short-block joint codebooks for KV-cache quantization. It combines folded rotations, candidate-table search, role-specific scaling, and key-heavy bit allocation with a packed vLLM implementation. The results are promising at 3 bits, but the experiments do not isolate the benefit of joint geometry or establish superiority over closely related vector KV-cache codecs.

Critical strengths
  1. Well-integrated codec design.

The paper combines joint codebooks, offline-prepared rotations, efficient candidate search, and key/value-specific reconstruction within a fixed-rate cache representation. The mathematical analysis is careful about its assumptions and provides reasonable motivation for the main design choices.

  1. Measured quality and serving improvements.

Unlike work reporting only reconstruction error, the paper evaluates seven models and implements packed-cache attention in vLLM. At 3 bits, HQ improves PPL on six of seven models and increases B200 decoding throughput by approximately 11% over the scalar TQ reference.

Critical weaknesses
  1. Insufficient comparison and component isolation.

The experiments compare almost exclusively with a restricted scalar TQ path, despite identifying FibQuant and BlockQuant as closely related vector codecs. Moreover, HQ simultaneously changes the codebook geometry, rotation, KV allocation, scaling, and reassignment procedure. The current ablations therefore do not establish that joint block geometry is responsible for the reported improvements.

  1. Limited robustness and practical validation.

The aggregate PPL gains are strongly influenced by extreme R1-Qwen results; among non-R1 models, the 4-bit aggregate improvement is negligible, while 2-bit HQ reduces average downstream accuracy. Configuration choices also reuse models and the WikiText-2 test set from the final evaluation. Finally, the paper lacks long-context quality tests, broader serving settings, uncertainty estimates, and key candidate-table parameters needed for reproduction.

Questions for authors
  • Controlled component analysis. Under the same rotation, scaling, and key/value allocation, how does scalar quantization compare with exact block VQ and candidate-table block VQ? This would clarify how much of the gain comes specifically from joint geometry.

  • Closest baseline comparison How does HQ compare with the full TQ method and closely related vector KV-cache codecs such as FibQuant and BlockQuant at the same effective memory budget? Comparisons with practical methods such as KIVI or KVQuant would also clarify the broader quality–throughput trade-off.

  • Held-out and long-context evaluation Do the improvements remain when configuration choices are made on separate validation data and evaluated on held-out models? Please also report quality, memory, and throughput over longer contexts and generations.

  • Candidate-search validation What grid size, shortlist length, clipping range, and table memory are used? How does candidate search compare with exhaustive search in nearest-codeword recall, quality, encoding latency, and total throughput?

Satisfactory answers and supporting results for the questions above, would address my main concerns and possibly raise my recommendation.

5 comments
AuthorsOpenReview ↗

Response to Reviewer UyEy

Thank you for laying out the evidence that would address your concerns. We have completed the controlled comparisons, added the closest baselines, and evaluated both long-context quality and the cost of candidate search.

Controlled component analysis. With rotation, scales, allocation and reassignment held constant, joint coding lowers PPL relative to scalar coding in all four model–corpus pairs and for all 15 independent rotation seeds (general response). Replacing exhaustive search with candidate-table search changes PPL by less than 0.01 in each comparison below, using K4/V2, the same Haar matrix and FP16 distance scoring:

ModelCorpusExhaustiveTwo candidates
Llama-3.1-8BWikiText-26.83476.8308
Llama-3.1-8BC411.381711.3830
Qwen3-8BWikiText-29.05809.0632
Qwen3-8BC414.781114.7723

Closest baselines. Among HQ, FibQuant and matched BlockQuant, HQ has the lowest PPL in 7/8 comparisons at 3.125 bits and in all eight at 2.125 bits with the validation-selected allocation. It also improves over TQ-PROD's Algorithm 2 residual sketch in all eight comparisons for both balanced and key-heavy allocations. Our reply to gwMo details the implementations and comparisons with KIVI, KVQuant, GEAR and PolarQuant. The downstream results are also encouraging: HQ has the highest average accuracy among the compressed methods below on both Qwen3 models. The table reports five-task accuracy (%) at 3 bits, using balanced TQ-PROD and matched BlockQuant:

ModelFullKVHQTQ-PRODFibQuantBlockQuant
Llama-3.1-8B74.0973.0366.1072.6773.14
Qwen3-8B71.5369.2547.7867.2967.48
Qwen3-14B74.9173.7963.9773.5173.22
Mistral-7B-v0.376.7975.8672.8076.2575.87

Held-out and long-context evaluation. The new allocation is selected on validation data, and the controlled geometry study includes a previously unused checkpoint (reply to gwMo, W6). The general response reports long-context quality using the packed cache. A 3-bit HQ cache uses 100 bytes per head and token, compared with 102 for TQ and 512 for FullKV. With equal 6 GiB KV pools and 20K prompts, HQ achieves 1.09–1.18× FullKV end-to-end throughput. With an 8 GiB pool, it completes generation at 128K total tokens, extending the capacity available with FullKV (reply to v4Hy).

Candidate-search validation. For b=2b=2 the table is a 256 × 256 grid over the address range [−5,5]2[-5,5]^2 with two int16 codeword indices per cell (shortlist length 2), i.e., 256 KiB per table and 512 KiB for K and V, shared across layers. Clipping affects only the address; distances use the original block.

Bits per 2D codeMin. FP32 shortlist coverageMin. FP16 recall vs FP32Max. relative excess distortion
299.997%99.969%0.0000%
499.974%99.898%0.0018%
699.890%99.632%0.0144%
899.545%98.545%0.1535%
1098.076%94.442%1.4663%

The deployed K4/V2 setting uses 8- and 4-bit codes. Candidate search retains high recall and changes PPL by less than 0.01 in the controlled comparisons above. With packing, scales and decoder held fixed, it increases end-to-end throughput by 1.12× [1.11, 1.13] at 4K/batch 1, 1.09× [1.09, 1.09] at 16K/batch 4 and 1.07× [1.07, 1.08] at 32K/batch 8 (five paired engine blocks). The complete packed-store encoder, including rotation and scale fitting, is 1.44× faster for 32 vectors and 5.08× for 16,384 vectors (five process pairs). The benefit therefore carries through from encoding to end-to-end serving.

In the camera-ready paper, we will present these controls alongside the main results and expand the reproducibility details for candidate search, validation-based selection and packed serving. We hope the additional evidence addresses the concerns you identified and would appreciate your reconsideration of the rating.

Thanks for the detailed rebuttal. Most of my concerns are resolved, but the additional results and other considerations have not been reflected within the manuscript through revision. Please incorporate them and detail the changes.

Thank you for pointing out this omission. We have revised the manuscript to incorporate the additional results and clarifications from our rebuttal.

Specifically, Section 4 now summarizes the controlled component analysis, expanded baseline comparisons, and additional serving results. Appendix D (Tables 9–21) provides the detailed evidence, including rotation-seed robustness, candidate-search accuracy and efficiency, validation-selected and held-out evaluations, long-context quality, and throughput and memory measurements.

We also clarify experimental protocols, baseline limitations, and cases where HQ does not improve, and have updated the reproducibility statement accordingly.

The main text remains within the 10-page limit. We appreciate your constructive feedback and hope the revision addresses your remaining concern.

Reviewer_gwMoOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HayateQuant (HQ), a data-free fixed-rate codec for compressing the key–value cache during language-model decoding. Relative to the scalar reconstruction path of TurboQuant, HQ groups coordinates into short blocks and quantizes each block with a jointly fitted vector codebook. To make this practical, the paper combines several components: a folded hybrid rotation, grid-indexed candidate lists for approximate codeword assignment, key- and value-specific scale rules, key-heavy K/V bit allocation, and packed attention directly in rotated coordinates.

My main concern is that the contribution of the block codebook is not isolated: HQ changes several components simultaneously, highly relevant prior work such as PolarQuant is not cited, and the closest vector-codec methods are not experimentally compared. The headline aggregate perplexity gains (especially at 4 bits) also rely heavily on R1-Qwen operating points where both compressed methods are severely degraded relative to FullKV. Finally, despite the paper’s long-context motivation, no long-context quality benchmark is reported.

Critical strengths

The 3-bit setting shows promising quality improvements over the TQ reference across the evaluated model families

Critical weaknesses
  1. The paper does not cite PolarQuant (Wu et al., NeurIPS 2025), published well before this submission. PolarQuant separately quantizes the radius and angle of each RoPE-paired two-dimensional unit, yielding a structured polar product code closely related to HQ’s b=2b=2 geometry. Its query-dependent decoding LUT is also conceptually related to HQ’s packed attention. The authors should cite and discuss PolarQuant, and compare under matched total KV bytes (or matched key bits with identical value precision).

  2. The only compressed baseline is a restricted scalar TQ path without its residual/inner-product channel. There are no comparisons to KIVI, KVQuant, FibQuant, BlockQuant, GEAR, or PolarQuant.

  3. The gains are driven by mostly degraded operating points: e.g., R1-Qwen-7B has FullKV PPL 46.204, versus TQ/HQ 153,393/74,902 at 2 bits and 21,496/254 at 4 bits. The appendix’s robust summaries differ from the headline: the non-R1 HQ/TQ ratio at 4 bits is 0.998; at 2 bits HQ wins only 3/7 models with a median gain of -2.6%; Llama-2-7B regresses from 6.079 to 8.151 PPL and from 64.9% to 59.8% ACC. Win counts, medians, and non-R1 aggregates should be foregrounded, and central claims should rest on non-collapsed operating points.

  4. HQ simultaneously changes joint coding, key-heavy allocation, NC/LS scale rules, value reassignment, and the rotation rule. Tables 3–4 probe only rotation and allocation on one or two models. A b=1 vs. b=2 comparison under identical rotation, allocation, scale, and search conventions, plus a scalar control with the same split and scale rules and an exact-vs.-candidate-search control, are needed. The exclusive choice of b=2 is also unsupported by a block-size sensitivity study. Without these controls, the gains may come from allocation, scaling, or rotation rather than “more geometry per bit.”

  5. In terms of long-context, it is measured only on WikiText-2 with 2048-token windows and five short zero-shot tasks. There is no LongBench/RULER/needle evaluation, no long-generation quality evaluation, and no generative mathematical or chain-of-thought reasoning evaluation despite the inclusion of R1 distills. Several compressed R1-Qwen scores also cluster near the approximate chance-level average of this task suite.

  6. The rotation and allocation rules are selected on Llama-3.1-8B/Qwen3-8B and then evaluated on the same models; the paper does not clarify whether the same WikiText-2 test split was used for selecting these rules and reporting the final results. Some development margins are only 1–2%, with no seeds or variance. A-HQ is also not evaluated in serving and has worse aggregate PPL than fixed HQ at 3 and 4 bits.

Questions for authors
  1. Please compare against PolarQuant under matched total KV bytes, or under matched key-cache bits with the same value-cache precision

  2. Can the authors provide a b=1 versus b=2 HQ comparison with identical rotation, scale rules, K/V allocation, and search conventions?

  3. How much of the gain remains when scalar TQ uses the same K/V split and role-specific scale rules as HQ?

  4. Were the rotation and allocation rules selected using the same WikiText-2 test split reported in Table 1?

  5. Please provide LongBench/RULER or long-generation results and a broader serving evaluation across context lengths and batch sizes

9 comments
AuthorsOpenReview ↗

COMMENT: Reviewer gwMo (1/3)

Thank you for the detailed review. We have added the comparisons and controls you requested, isolated the contribution of joint geometry, and evaluated long-context quality and mathematical reasoning. These experiments also led to a better 2-bit allocation and an effective way to protect reasoning accuracy.

W1/Q1: PolarQuant. We appreciate the reference to PolarQuant (Wu et al., NeurIPS 2025) and will add it to the related-work discussion. PolarQuant codes the radius and angle of each RoPE pair and decodes with a query-dependent table. HQ uses joint Cartesian codebooks after rotation and a candidate table for encoding. For the comparison you requested, all methods use the same V2 value codec, key budget, metadata and 256-token block-retirement policy, totaling 3.126–3.132 bits per coordinate. The table reports test PPL:

ModelCorpusHQ keyScalar keyPolarQuant keyHQ / PolarQuant [95% CI]
Llama-3.1-8BWikiText-26.56096.56146.59630.9946 [0.9939, 0.9953]
Llama-3.1-8BC410.791810.796110.87770.9921 [0.9901, 0.9939]
Qwen3-8BWikiText-28.71458.74819.09350.9583 [0.9534, 0.9632]
Qwen3-8BC413.829213.875614.90390.9279 [0.9208, 0.9342]
Qwen3-14BWikiText-27.70527.72027.84640.9820 [0.9801, 0.9842]
Qwen3-14BC412.453212.479812.70540.9802 [0.9777, 0.9826]
Mistral-7B-v0.3WikiText-24.96514.96564.97990.9970 [0.9966, 0.9975]
Mistral-7B-v0.3C47.82437.82477.83410.9987 [0.9983, 0.9992]

HQ achieves lower PPL in all eight matched comparisons, with every 95% interval below one. The broader table below also includes PolarQuant's released policies, which use a 128-token block-retirement policy. HQ has lower PPL in 7/8 comparisons with the V4 version while using fewer bits; both V4 and V2 results are reported.

W2: broader baselines. We added TQ-PROD, KIVI, KVQuant and GEAR alongside the vector codecs. TQ-PROD includes Algorithm 2's residual sketch; the separate outlier-channel quantizers are not reproduced. FibQuant and BlockQuant are independent implementations, and released policies are ported to the common quality harness. Effective bits include metadata and full-precision cache portions. WikiText-2 / C4 PPL:

MethodEff. bitsLlama-3.1-8BQwen3-8BQwen3-14BMistral-7B
FullKV166.484 / 10.6728.687 / 13.8417.683 / 12.4544.936 / 7.824
HQ3.1256.821 / 11.3778.993 / 14.6777.840 / 12.6375.034 / 7.922
TQ-PROD, key-heavy3.1887.594 / 12.70422.268 / 62.5308.495 / 13.9975.253 / 8.212
FibQuant3.1256.915 / 11.46810.004 / 18.4347.997 / 13.0345.045 / 7.961
BlockQuant, matched3.1256.858 / 11.3549.545 / 16.1767.940 / 12.9245.036 / 7.958
BlockQuant, p=3p=33.0106.894 / 11.4469.050 / 14.8527.900 / 12.7865.031 / 7.932
PolarQuant, released3.256.846 / 11.4019.403 / 15.3938.009 / 12.9165.031 / 7.958
PolarQuant, released, V23.196.605 / 10.8939.133 / 15.0367.848 / 12.7084.986 / 7.840
KIVI-2, released3.416.856 / 11.3748.935 / 14.0817.848 / 12.6085.042 / 7.953
KVQuant-3, calibrated3.58–3.686.595 / 10.8738.831 / 14.0057.796 / 12.5815.004 / 7.887
GEAR-24.637.910 / 13.2139.682 / 15.0138.519 / 13.4465.414 / 8.666

HQ has lower PPL than key-heavy TQ-PROD and FibQuant in all eight comparisons, and than each BlockQuant variant in seven of eight. It also improves over balanced TQ-PROD, evaluated at 3.188 bits, in all eight comparisons. These results support HQ as a competitive data-free codec at 3.125 bits, with the table making the quality and storage trade-offs across methods explicit.

AuthorsOpenReview ↗

Response to Reviewer gwMo (2/3): component controls and R1

W4, Q2–Q3: component isolation. The first table in our general response compares b=1b=1 with b=2b=2 while holding rotation, scaling, K/V allocation, reassignment and exhaustive search constant. Joint coding lowers PPL in all four model–corpus pairs, for all 15 independent rotation seeds. Giving scalar coding the same K4/V2 split and NC/LS scales still leaves a 0.7–1.6% PPL improvement from joint coding at the reference seed. Exact and candidate search differ by less than 0.01 PPL (reply to UyEy). To study block size, we also compare b=1,2,4b=1,2,4 with balanced K3/V3 allocation, the same Haar matrix and scales, and exhaustive FP32 search:

ModelCorpusb=1b=1b=2b=2b=4b=4
Llama-3.1-8BWikiText-26.94046.86066.8316
Llama-3.1-8BC411.544811.362511.2968
Qwen3-8BWikiText-29.94669.84699.5610
Qwen3-8BC417.738717.409116.2706
Encoding, 16,384 vectors (µs)2562,31033,646

b=2b=2 captures a consistent quality improvement with a compact 64-entry codebook. Increasing to b=4b=4 improves PPL further, but its 4,096 entries make encoding about 15× slower. This supports b=2b=2 as an effective balance of quality and encoding cost.

W3: robust model-level results. The general response emphasizes individual models and non-R1 summaries, as you suggested. To address the R1 behavior directly, we used training data to identify a simple remedy: protecting the first layer's cache. With equal protection and 3.53–3.59 effective bits, HQ lowers test PPL relative to TQ by 5.8–24.4% across all six model–corpus pairs. For example, R1-Qwen-1.5B WikiText-2 PPL improves from 191.5 to 144.9. The matched scalar control also benefits, identifying sensitive-key protection as the main source of recovery.

W5: generative reasoning. We evaluated all 500 MATH-500 questions for every condition, using identical model-native prompts, greedy decoding and a 32,768-token cap. TQ here is the randomized-Hadamard scalar reference; HQ uses 3.125 stored bits per coordinate. The guarded variants retain the first layer's cache in full precision, using 3.527 bits on R1-Llama and 3.585 on the R1-Qwen models. Each cell gives accuracy (%) and the number of generations reaching the token cap:

ModelFullKVHQTQHQ + guardTQ + guardScalar + guard
R1-Qwen-1.5B56.2 / 1530.4 / 2900.2 / 30450.4 / 22748.4 / 21551.0 / 206
R1-Qwen-7B68.8 / 1050.2 / 3660.6 / 37465.0 / 12863.0 / 14662.4 / 150
R1-Llama-8B72.4 / 5169.4 / 5965.8 / 8269.6 / 6268.6 / 6969.0 / 78

Protecting one layer raises HQ accuracy from 0.4% to 50.4% on R1-Qwen-1.5B and from 0.2% to 65.0% on R1-Qwen-7B. With equal protection, HQ's point estimates exceed TQ by +2.0, +2.0 and +1.0 points (paired 95% intervals: [−2.2, +6.0], [−2.2, +6.2], [−2.6, +4.6]). The clearest result is the large recovery from first-layer protection, which also benefits the scalar control.

On R1-Llama, HQ reaches 69.4% without protection, versus 65.8% for TQ (+3.6 points, 95% CI [−0.2, +7.2]), with fewer cap hits (59 vs 82).

For five batches of five questions, including one FullKV batch, we disabled the cuDNN attention backend after backend errors. Prompts, token caps and scoring were unchanged. Excluding the affected ten questions from every method on each affected model preserves the direction of all reported differences.

AuthorsOpenReview ↗

Response to Reviewer gwMo (3/3): selection data, long context, A-HQ

W6/Q4: selection data. Yes, the original rotation/allocation study used a subset of the WikiText-2 test split. In the new study, we selected the allocation on WikiText-2 validation and a disjoint C4 development subset, then froze it before test evaluation. First-layer protection was selected using WikiText-2 train only. The selected 2-bit allocation changes K3/V1 to K2.5/V1.5 while keeping search, matrices and scale rules fixed. The table reports test PPL and five-task accuracy (%):

ModelWikiText-2 PPLC4 PPLFive-task accuracy
Llama-2-7B8.141 → 5.74511.262 → 7.84759.64 → 65.28
Llama-3.1-8B11.147 → 8.44918.318 → 14.08266.24 → 69.77
Qwen3-4B46.926 → 67.21757.801 → 85.81950.00 → 50.41
Qwen3-8B19.019 → 13.37328.466 → 22.06057.56 → 61.92
Qwen3-14B10.017 → 9.08415.881 → 14.75067.92 → 70.91
Mistral-7B-v0.36.431 → 5.5189.442 → 8.50069.76 → 73.40
R1-Llama-8B46.941 → 31.52577.781 → 52.91740.91 → 40.77
R1-Qwen-7B74655.334 → 97695.34087125.709 → 127893.88935.41 → 35.53
R1-Qwen-1.5B26907.819 → 24962.42534379.635 → 29855.34536.06 → 35.76

The selected rule improves PPL on 7/9 models and five-task accuracy on all six non-R1 models. In particular, it resolves the Llama-2-7B regression you identified: accuracy rises from 59.64% to 65.28%. At 2.125 bits, HQ also has lower PPL than FibQuant and matched BlockQuant on all eight model–corpus pairs. WikiText-2 / C4 PPL:

ModelHQ, selected allocationFibQuantBlockQuant, matched
Llama-3.1-8B8.449 / 14.0829.427 / 15.8079.369 / 15.672
Qwen3-8B13.373 / 22.060467.480 / 875.42731.661 / 60.800
Qwen3-14B9.084 / 14.75011.495 / 18.82111.153 / 18.013
Mistral-7B-v0.35.518 / 8.5005.662 / 8.8005.648 / 8.836

Joint coding also improves quality on the previously unused Mistral-7B-Instruct-v0.3 checkpoint: with the same Haar rotation, it lowers WikiText-2 PPL from 5.055 to 5.037 and C4 from 7.936 to 7.917. The Qwen3-14B gains are smaller, with intervals including zero. The 15-seed study in the general response measures variation on the development models.

W5/Q5: long context and serving. RULER and LongBench-E with the packed cache are in the general response. The context × batch serving grid and the memory-constrained results are in our reply to v4Hy. At 16K context, batch 4 and 1,024 generated tokens, the optimized HQ decoder is 2.93× faster than TQ end to end [95% CI 2.91, 2.95].

A-HQ. We will focus the main paper on fixed HQ, which provides the stronger 3/4-bit aggregate quality and the measured packed-serving gains. A-HQ will appear as an exploratory allocation study in the appendix, without a serving claim.

For the camera-ready revision, we will integrate the new baselines and controlled studies into the main evaluation, expand the analysis of allocation and layer sensitivity, and fully document data selection and serving protocols. The evidence now directly addresses the central questions about geometry, comparative quality and long-context use. We would be grateful if you would reconsider your rating in light of these results.

Reviewer_v4HyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

HayateQuant (HQ) is a data-free KV-cache quantization codec. It asks whether short-block (vector) quantization can improve a KV-cache codec within the code-and-scale budget of scalar quantization while preserving efficient packed serving (§1, p.1-2). Building on TurboQuant's model-independent rotation-plus-scalar-levels path (§2, p.3), HQ replaces per-coordinate scalar assignment with joint block codebooks of block size b=2 that keep the total code length at dβ bits with one scale per vector (§3.1, p.3). A folded rotation materializes a Hadamard×sign×per-block-Haar product into a single dense transform offline (Eq.6, §3.2, p.5); a precomputed candidate table replaces exhaustive codebook search (§3.2, p.5); and role-specific scales — norm-correcting for keys, least-squares for values — with a key-heavy bit split address the distinct roles of keys and values in attention (§3.3, p.5). Attention runs directly on packed entries in rotated coordinates (Eq.10, §3.4, p.6), and an optional Adaptive HQ selects between allocations from causal statistics (§3.5, p.6). Validation spans seven models at 2/3/4 bits: WikiText-2 perplexity and five-task zero-shot accuracy (Table 1, p.7), vLLM decode throughput on one B200 (Table 2, p.8), and rotation/allocation ablations (Tables 3-4, p.9), with supporting proofs in Appendix B.

Critical strengths

The folded rotation is a clever construction that avoids adding serving-path overhead. The Hadamard×sign×per-block-Haar product (Eq.6, §3.2, p.5) is materialized offline into one dense transform, so at serving time it is a single dense matmul with the same arithmetic structure as the dense-rotation reference and no separate block-mixing pass (§3.2, p.5; Prop B.5, p.15). This is consequential because, as the paper frames it (§1, p.1), improved geometry is useful only if it is realizable cheaply on the inference path. The codec built around it lowers aggregate WikiText-2 PPL at every tested rate (3-bit 118.4→68.3, 4-bit 84.4→35.8 vs TQ, Table 1, p.7) and improves measured decode throughput over the TQ controls at all three rates (Table 2, p.8). Treating keys and values differently is supported by both theory and practical value. Attention is linear in values but nonlinear in keys: a value error enters the output directly, while a key error acts through the softmax weights with sensitivity scaled by ∥q∥/√d (Eq.7, §3.3, p.5; Theorem B.16, p.19). This motivates norm-correcting scales for keys, least-squares scales for values (Eq.8, p.5), and a key-heavy bit split. The allocation ablation confirms the payoff: the key-heavy K4/V2 split gives the lowest PPL on both development models (Llama-3.1-8B 6.933, Qwen3-8B 8.648) over symmetric and value-heavy splits (Table 4, p.9). This grounds, in both analysis and measurement, a choice usually made heuristically.

Critical weaknesses

The evaluation includes no vector-codec baseline. In §5 (p.9) the paper names FibQuant and BlockQuant as "closely related vector codecs" and then, in the same paragraph, states "We evaluate this design against TQ's scalar reference." TQ is a scalar codec, so every main-table comparison (Table 1, p.7) pits HQ's joint short-block geometry against per-coordinate scalar assignment. The paper's central finding — that short-block/vector geometry improves a KV-cache codec — is therefore never tested against another method that also uses block/vector geometry. I searched the paper, including the appendices, for a stated reason for the omission (unavailable code, an incompatible setting, a different rate regime); none is given. The only adjacent statement is that HQ's focus is "the combined realization" of its components (§5, p.9). This matters because the demonstrated gains could stem from the joint-codebook idea HQ shares with those baselines rather than from HQ's specific realization. It is a bounded gap — the scalar comparison is valid and the method is sound — but it leaves the key comparative claim untested.

Questions for authors
  1. Table 2 (p.8) and §4.3 report decode and end-to-end throughput for HQ and TQ at matched code rate; there is no uncompressed-serving row, and FullKV appears only as a quality reference (Table 8, p.20). So the reported throughput gains are over TQ, not over uncompressed serving. At batch size 1 with a 4096-token prompt and 128 generated tokens, is HQ a decode-throughput win over uncompressed (FullKV) serving, or only over TQ at matched rate? Please report FullKV throughput under the same protocol.
  2. The deployed recipe uses dense Haar at ≤2 bits and the folded hybrid above that (§4.1, p.7), and both are materialized as a single dense d×d transform. In the rotation ablation (Table 3, p.9), dense Haar gives better 2-bit PPL (8.915 vs hybrid 9.123), while at 3 bits the hybrid leads by only 0.021 PPL (7.013 vs 7.034) but wins KV-MSE more clearly (0.0290 vs 0.0300). Does folding save any serving work relative to dense Haar, or only relative to a two-pass Hadamard-then-per-block-Haar implementation? Please state folding's cost and quality advantage over dense Haar precisely.
  3. Table 2 reports the median of three vLLM runs with no spread (§4.3, p.8). The decode gains over TQ are +4.5% / +11.5% / +3.7% at 2/3/4 bits; the 4-bit gap in particular is small. What is the run-to-run variability across the three runs (e.g., min/max or standard deviation)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer v4Hy

Thank you for recognizing the value of the folded rotation and the role-specific treatment of keys and values. Your questions helped us strengthen the comparisons and clarify the serving results.

Vector-codec baselines. HQ compares favorably with the two closest vector codecs. We evaluated independent implementations of FibQuant and a matched-budget BlockQuant variant at the same 3.125 bits per coordinate, including metadata. The table reports test PPL; lower is better:

ModelWikiText-2: HQ / FibQuant / BlockQuantC4: HQ / FibQuant / BlockQuant
Llama-3.1-8B6.821 / 6.915 / 6.85811.377 / 11.468 / 11.354
Qwen3-8B8.993 / 10.004 / 9.54514.677 / 18.434 / 16.176
Qwen3-14B7.840 / 7.997 / 7.94012.637 / 13.034 / 12.924
Mistral-7B-v0.35.034 / 5.045 / 5.0367.922 / 7.961 / 7.958

HQ has the lowest PPL in 7/8 cells and improves over FibQuant in all eight. It also has lower PPL in 7/8 comparisons with the outlier-aware p=3p=3 BlockQuant variant at 3.010 bits (full table in our reply to gwMo). Across the five zero-shot tasks, HQ has the highest average accuracy among these compressed codecs on Qwen3-8B and Qwen3-14B and is within 0.4 points of the best on Llama-3.1 and Mistral. At 2 bits, the validation-selected allocation gives HQ lower PPL than both vector codecs in all eight model–corpus pairs. All quality comparisons use the same evaluation harness.

Q1: FullKV throughput. HQ converts the memory saved by compression into higher throughput under a fixed KV budget. With equal 6 GiB KV pools, 20,480-token prompts and 192 generated tokens, HQ/FullKV end-to-end throughput is 1.086× [1.076, 1.098], 1.096× [1.091, 1.101] and 1.181× [1.175, 1.189] at 3, 6 and 12 concurrent requests. The intervals are based on five paired engine launches on the same B200 GPU, with three request repetitions each.

For your requested 4K/batch-1 setting, the original HQ and FullKV decode rates are 173.6 and 265.1 tokens/s (Q3 below). The optimized HQ decoder reaches 0.90× [0.87, 0.93] of FullKV end-to-end throughput.

The broader context and batch-size comparison, with 128 output tokens, gives the following optimized end-to-end throughput ratios:

ContextHQ / native TQ, batch 1 / 4 / 8HQ / FullKV, batch 1 / 4 / 8
4K1.51 / 1.67 / 1.760.90 / 0.83 / 0.78
16K2.19 / 2.22 / 2.300.88 / 0.81 / 0.78
32K2.61 / 2.46 / 2.540.86 / 0.81 / 0.77

HQ is 1.51–2.61× faster than native TQ across this grid, with all nine paired 95% intervals above one. Each of five independent comparison blocks runs the methods sequentially on the same GPU of an exclusively allocated B200 node. Compression also extends capacity: with equal 8 GiB KV pools, HQ completes generation at 131,072 total tokens, beyond FullKV's cache capacity.

Q3: variability. We repeated the original-decoder comparison with five independent engine launches per method on one isolated B200, using 4K input, batch 1 and 128 output tokens, with three request repetitions per launch. Below are median decode rates in tokens/s and the range of engine medians. FullKV achieves 265.1 [261.1, 267.0]:

Cache bitsTQHQ
2166.1 [164.8, 166.3]175.5 [175.0, 176.7]
3152.5 [152.3, 153.3]173.6 [172.2, 173.6]
4161.2 [161.2, 161.3]170.1 [168.6, 170.2]

At every rate, including 4 bits, HQ's slowest engine is faster than TQ's fastest.

Q2: what folding saves. Folding provides the hybrid rotation at the cost of one dense transform, matching dense Haar's arithmetic structure and eliminating a separate mixing pass. For 16,384 vectors (d=128d=128, FP32), the dense reference implementations take 14.7 µs folded, 15.6 µs for Haar and 110.3 µs in two passes (five-process medians). The quality choice depends on rate: Table 3 favors Haar at 2 bits and the hybrid at 3 bits, which motivates the deployed recipe. Folding makes the hybrid practical at the same one-transform cost.

In the camera-ready revision, we will make the vector-codec comparisons, FullKV measurements and timing variability part of the main evaluation, and clarify the cost and purpose of folding. We hope this evidence resolves your questions and supports a more favorable assessment of the paper.

Reviewer_v4HyOpenReview ↗

Response to the authors' rebuttal

Thank you for the detailed response. The FibQuant/BlockQuant comparisons and the 15-seed joint-vs-scalar control address my main weakness, and the five-launch measurements answer Q3. I am keeping my rating of weak acceptance. The following points from the response bound the paper's practical claims and should be reflected in the camera-ready:

  1. FullKV throughput (Q1). At a fixed workload HQ runs at 0.77–0.90× FullKV with the optimized decoder, and 0.65× with the decoder behind Table 2 (173.6 vs. 265.1 tok/s). It exceeds FullKV (1.09–1.18×) only with a 6 GiB KV pool on a B200, where HQ admits about 5× more tokens; that capacity turns into under 1.2× throughput. The abstract's "improves decoding throughput over the existing baselines" holds against TQ only and should say so. An FP8 KV cache, the default production option, is not compared at equal memory.
  2. Folding (Q2). Folding costs the same as dense Haar (14.7 vs. 15.6 µs) and saves work only relative to a two-pass implementation. Its quality edge over dense Haar is 0.021 PPL at 3 bits in a single development comparison, and dense Haar is better at 2 bits. Since the title names folded rotations, the paper should state plainly that dense Haar is an equally cheap alternative.
  3. Quality summaries. The 7-model geometric means are driven by the R1-Qwen models, where every codec collapses even at 4 bits (R1-Qwen-1.5B: FullKV 173.7, HQ 3,866). The non-R1 summary (10.8% at 3 bits, 0.2% at 4 bits) and the controlled 0.9–2.2% joint-coding gain are the informative numbers and should lead. On RULER 32K, HQ is 2.5–3.7 points below FullKV.
  4. Baselines. In Tables 13–14, released PolarQuant V2 at 3.19 bits has lower PPL than HQ on Llama-3.1-8B and Mistral-7B on both corpora, so the general response's "lower PPL than matched PolarQuant in all eight comparisons" needs the matched configuration specified. FibQuant and BlockQuant are your own implementations; please release them with the code.

Thank you for your constructive follow-up and valuable suggestions. We appreciate your acknowledgment that our additional experiments address the main concern.

We commit to addressing all your points without exception in the camera-ready version. Specifically, we will clarify the throughput claims and FullKV/FP8 comparisons, accurately describe the benefits and limitations of folded rotations, prioritize representative quality results and acknowledge RULER limitations, clarify the PolarQuant comparisons, and release our FibQuant/BlockQuant implementations.

We will carefully revise the paper to ensure all claims are accurate, transparent, and fully supported by the evidence.

Thank you again for your thoughtful feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the reviewers for their thoughtful feedback. The additional experiments provide stronger evidence for HQ's central contribution: better quantization geometry can improve quality while remaining efficient in a packed KV cache. Controlled comparisons show a consistent benefit from joint coding, and the expanded evaluation covers seven baseline families, long-context tasks and mathematical reasoning. HQ achieves lower PPL than matched PolarQuant in all eight comparisons and 1.09–1.18× FullKV throughput with equal 6 GiB KV pools.

Isolating the benefit of joint geometry. We compare scalar (b=1b=1) and joint (b=2b=2) coding with identical rotations, K4/V2 allocation, key/value scale rules, reassignment and exhaustive FP32 search. Both use FP16 scales; only the codebook changes. The table reports test PPL at 3.125 bits per coordinate, including scales:

ModelCorpusScalarJointJoint wins, 15 new rotation seedsJoint/scalar PPL, geo. mean [95% CI]
Llama-3.1-8BWikiText-26.88596.836315/150.9900 [0.9885, 0.9915]
Llama-3.1-8BC411.471711.380415/150.9911 [0.9891, 0.9931]
Qwen3-8BWikiText-29.21209.069115/150.9822 [0.9776, 0.9863]
Qwen3-8BC414.952314.765515/150.9783 [0.9733, 0.9830]

Joint coding lowers PPL by 0.9–2.2% across the new seeds, with all four confidence intervals for the joint/scalar ratio below one. Thus, a measurable gain remains when allocation, scaling and rotation are held constant. The 15 seeds were fixed after a separate five-seed pilot, which included two reversals on Qwen3. These intervals resample seeds; the other fixed-seed PPL intervals resample corpus blocks.

Long-context quality in packed serving. We evaluate the packed 3-bit caches in vLLM with identical prompts and decoding. RULER includes all 13 tasks with 100 examples per task; LongBench-E includes every example in its 13 tasks, with a 32K prompt cap. Higher scores are better.

ModelBenchmarkFullKVTQHQHQ, optimized decoder
Llama-3.1-8BRULER 8K90.1988.0689.4789.21
Llama-3.1-8BRULER 32K85.0383.0182.5482.69
Llama-3.1-8BLongBench-E54.5252.2154.5954.20
Mistral-7B-v0.3RULER 8K91.1790.9190.7390.63
Mistral-7B-v0.3RULER 32K84.0679.7780.4180.22
Mistral-7B-v0.3LongBench-E51.7551.3651.7851.79

HQ is within 0.1 points of FullKV on LongBench-E for both models and exceeds native TQ in four of six cells. On Llama-3.1 RULER 8K, its gain over TQ is +1.40 points [95% CI +0.55, +2.28]. The table also evaluates the optimized decoder used in the new speed measurements; its scores are within 0.4 points of original HQ.

Improvements across models. Following gwMo and UyEy's suggestion, we emphasize individual models and non-R1 summaries. At 3 bits HQ lowers WikiText-2 PPL relative to TQ on 6/7 models; among the four non-R1 models it wins 3/4, with a 10.8% geometric-mean and 5.5% median reduction. The non-R1 gain at 4 bits is 0.2%. At 2 bits, selecting the allocation on validation data improves PPL on 7/9 models and five-task accuracy on all six non-R1 models. At the same 2.125 bits, HQ also achieves lower PPL than FibQuant and matched BlockQuant in all eight model–corpus comparisons. Our reply to gwMo gives the full allocation results and MATH-500 evaluation.

If accepted, we will incorporate these results into the camera-ready paper and revise the main evaluation around the reviewers' questions. We will expand the discussion of related work and sensitivity analyses, document the selection and evaluation protocols, and provide the implementation details needed to reproduce the comparisons. The controlled geometry results and the gains in packed serving will be central to the revised presentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.