- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HayateQuant (HQ) replaces TurboQuant’s (TQ) scalar assignments with short-block joint codebooks for KV-cache quantization. It combines folded rotations, candidate-table search, role-specific scaling, and key-heavy bit allocation with a packed vLLM implementation. The results are promising at 3 bits, but the experiments do not isolate the benefit of joint geometry or establish superiority over closely related vector KV-cache codecs.
Critical strengths
- Well-integrated codec design.
The paper combines joint codebooks, offline-prepared rotations, efficient candidate search, and key/value-specific reconstruction within a fixed-rate cache representation. The mathematical analysis is careful about its assumptions and provides reasonable motivation for the main design choices.
- Measured quality and serving improvements.
Unlike work reporting only reconstruction error, the paper evaluates seven models and implements packed-cache attention in vLLM. At 3 bits, HQ improves PPL on six of seven models and increases B200 decoding throughput by approximately 11% over the scalar TQ reference.
Critical weaknesses
- Insufficient comparison and component isolation.
The experiments compare almost exclusively with a restricted scalar TQ path, despite identifying FibQuant and BlockQuant as closely related vector codecs. Moreover, HQ simultaneously changes the codebook geometry, rotation, KV allocation, scaling, and reassignment procedure. The current ablations therefore do not establish that joint block geometry is responsible for the reported improvements.
- Limited robustness and practical validation.
The aggregate PPL gains are strongly influenced by extreme R1-Qwen results; among non-R1 models, the 4-bit aggregate improvement is negligible, while 2-bit HQ reduces average downstream accuracy. Configuration choices also reuse models and the WikiText-2 test set from the final evaluation. Finally, the paper lacks long-context quality tests, broader serving settings, uncertainty estimates, and key candidate-table parameters needed for reproduction.
Questions for authors
-
Controlled component analysis. Under the same rotation, scaling, and key/value allocation, how does scalar quantization compare with exact block VQ and candidate-table block VQ? This would clarify how much of the gain comes specifically from joint geometry.
-
Closest baseline comparison How does HQ compare with the full TQ method and closely related vector KV-cache codecs such as FibQuant and BlockQuant at the same effective memory budget? Comparisons with practical methods such as KIVI or KVQuant would also clarify the broader quality–throughput trade-off.
-
Held-out and long-context evaluation Do the improvements remain when configuration choices are made on separate validation data and evaluated on held-out models? Please also report quality, memory, and throughput over longer contexts and generations.
-
Candidate-search validation What grid size, shortlist length, clipping range, and table memory are used? How does candidate search compare with exhaustive search in nearest-codeword recall, quality, encoding latency, and total throughput?
Satisfactory answers and supporting results for the questions above, would address my main concerns and possibly raise my recommendation.
5 comments
Thanks for the detailed rebuttal. Most of my concerns are resolved, but the additional results and other considerations have not been reflected within the manuscript through revision. Please incorporate them and detail the changes.
Reply
Thank you for pointing out this omission. We have revised the manuscript to incorporate the additional results and clarifications from our rebuttal.
Specifically, Section 4 now summarizes the controlled component analysis, expanded baseline comparisons, and additional serving results. Appendix D (Tables 9–21) provides the detailed evidence, including rotation-seed robustness, candidate-search accuracy and efficiency, validation-selected and held-out evaluations, long-context quality, and throughput and memory measurements.
We also clarify experimental protocols, baseline limitations, and cases where HQ does not improve, and have updated the reproducibility statement accordingly.
The main text remains within the 10-page limit. We appreciate your constructive feedback and hope the revision addresses your remaining concern.
Response to Reviewer UyEy
Thank you for laying out the evidence that would address your concerns. We have completed the controlled comparisons, added the closest baselines, and evaluated both long-context quality and the cost of candidate search.
Controlled component analysis. With rotation, scales, allocation and reassignment held constant, joint coding lowers PPL relative to scalar coding in all four model–corpus pairs and for all 15 independent rotation seeds (general response). Replacing exhaustive search with candidate-table search changes PPL by less than 0.01 in each comparison below, using K4/V2, the same Haar matrix and FP16 distance scoring:
Closest baselines. Among HQ, FibQuant and matched BlockQuant, HQ has the lowest PPL in 7/8 comparisons at 3.125 bits and in all eight at 2.125 bits with the validation-selected allocation. It also improves over TQ-PROD's Algorithm 2 residual sketch in all eight comparisons for both balanced and key-heavy allocations. Our reply to gwMo details the implementations and comparisons with KIVI, KVQuant, GEAR and PolarQuant. The downstream results are also encouraging: HQ has the highest average accuracy among the compressed methods below on both Qwen3 models. The table reports five-task accuracy (%) at 3 bits, using balanced TQ-PROD and matched BlockQuant:
Held-out and long-context evaluation. The new allocation is selected on validation data, and the controlled geometry study includes a previously unused checkpoint (reply to gwMo, W6). The general response reports long-context quality using the packed cache. A 3-bit HQ cache uses 100 bytes per head and token, compared with 102 for TQ and 512 for FullKV. With equal 6 GiB KV pools and 20K prompts, HQ achieves 1.09–1.18× FullKV end-to-end throughput. With an 8 GiB pool, it completes generation at 128K total tokens, extending the capacity available with FullKV (reply to v4Hy).
Candidate-search validation. For b=2 the table is a 256 × 256 grid over the address range [−5,5]2 with two int16 codeword indices per cell (shortlist length 2), i.e., 256 KiB per table and 512 KiB for K and V, shared across layers. Clipping affects only the address; distances use the original block.
The deployed K4/V2 setting uses 8- and 4-bit codes. Candidate search retains high recall and changes PPL by less than 0.01 in the controlled comparisons above. With packing, scales and decoder held fixed, it increases end-to-end throughput by 1.12× [1.11, 1.13] at 4K/batch 1, 1.09× [1.09, 1.09] at 16K/batch 4 and 1.07× [1.07, 1.08] at 32K/batch 8 (five paired engine blocks). The complete packed-store encoder, including rotation and scale fitting, is 1.44× faster for 32 vectors and 5.08× for 16,384 vectors (five process pairs). The benefit therefore carries through from encoding to end-to-end serving.
In the camera-ready paper, we will present these controls alongside the main results and expand the reproducibility details for candidate search, validation-based selection and packed serving. We hope the additional evidence addresses the concerns you identified and would appreciate your reconsideration of the rating.