- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies per-vector KV-cache quantization under three requirements: no calibration data or per-model fitting, a rate fixed in advance with all side information counted, and applicability across model families. HybridQuant applies three rounds of the randomized fast Walsh–Hadamard transform within each head, scales each row by its fp16 RMS, quantizes 8-dimensional blocks to the E8 lattice with the Conway–Sloane decoder, and codes the lattice points with a static, context-switched rANS model. The code tables, and the step size that meets a target rate, are computed offline by Monte Carlo on a Gaussian source. A dithered variant, HybridQuant-G, adds subtractive Voronoi dither. For models with key/value projection biases (Qwen2), the known biases (rotated by RoPE for keys) are subtracted before quantization and added back afterwards. On the theory side, Theorem 4.1 bounds the Kolmogorov distance to a Gaussian after one, two and three Hadamard rounds for a spike input. Theorem 4.2 applies the Zamir–Feder theorem to show that the dithered error does not depend on the input, and Theorem 4.3 bounds the concentration of the attention output independently of context length. Experiments on Qwen2.5-7B, Mistral-7B-v0.3 and Llama-3.1-8B report wikitext2/C4 perplexity at 2.1–5 effective bits against re-implemented TurboQuant-MSE and KIVI and a rotated entropy-coded scalar quantizer (Rot-ECSQ), with paired bootstrap tests. They add needle retrieval, LongBench-E subsets, one-at-a-time ablations, a sweep over the number of rounds, a check of the rate accounting against real rANS bitstreams, and a comparison under the protocol of two recent preprints.
Critical strengths
S1. Rate–distortion evaluation with fully counted rates and matched controls. Every reported rate includes the fp16 scale, coder flushes and escapes (Eq. 1). The accounting is checked against real rANS bitstreams that round-trip bit-exactly and come out 0.03–0.05% smaller than accounted (App. B.2), and measured rates fall within 0.003 bits of the target (Tables 2, 21). The ablations change one component at a time inside the same pipeline (Table 5: Z vs. E8, Rice vs. rANS, R = 0…3, centering). Rot-ECSQ and the Z-lattice control separate the contribution of E8 from that of entropy coding, and differences are tested with a paired bootstrap over windows and checked over five seeds. At ~2 bits the gains over fixed-rate quantizers are large and consistent across three model families and two datasets: wikitext2 perplexity is 8.8%, 13.3% and 31.3% higher for TurboQuant-MSE^c (Sec. 5.1; the percentages follow from Table 2). Together, these make the central empirical claim credible: a static entropy code designed for a Gaussian source works on real KV caches without calibration and beats fixed-rate scalar codes at equal rate.
S2. Exact projection-bias centering, and the diagnosis behind it. Sec. 3.2 and App. L.1 show that the k/v projection biases of Qwen2 make per-vector rotation quantizers collapse: TurboQuant-MSE without centering exceeds perplexity 4000 at every rate and finds 0/88 needles (Tables 3, 9). The fix is exact, needs no data, is consistent with RoPE and costs no bits (Prop. L.1). It also removes the divergence that recent work avoided by keeping layers in bf16 (Table 26). The fix is simple, but any per-vector KV quantizer can use it. The authors apply it to their baselines, so the comparison is not against a broken baseline.
Critical weaknesses
W1. The deployment setting the paper motivates — a variable-length, entropy-coded cache that attention decodes at every step — is not demonstrated. Requirement (ii) and the "drop-in component whose memory budget is known before the model is loaded" (Sec. 1) assume that the compressed cache can be stored and decoded online. Several points leave this open:
- All quality results simulate quantization: code lengths are computed on the GPU from the static table, and real rANS encoding and decoding run only on the CPU, on one window per model, as a check of the accounting (Sec. 3.3, App. B.2).
- There is no GPU decoder and no measurement of decode throughput. Table 23 excludes rANS (App. P.2). Every decoding step must decode all cached streams, each a sequential chain of 4608 symbols under a single 32-bit rANS state (App. B.2). Fixed-rate codes, by contrast, dequantize with a few independent operations per element.
- The accounting leaves out the per-stream offsets that any implementation needs (Sec. 3.3, about +0.008 bits). This contradicts "with every side bit counted" in the abstract.
- Only run-level average rates are reported. The spread across heads, layers and streams is not, yet it decides how much memory a fixed per-head budget in a paged cache must reserve.
- Storage of the partial (up to 31-token) stream before encoding is not discussed; rANS encodes in reverse order, so a stream can only be encoded once it is complete.
The Limitations section disclaims speed. Still, the paper's case against fixed-rate rotation quantizers depends on entropy decoding being affordable in this position, rather than only for storage or transfer, which is where CacheGen and KVTC use it. The TurboQuant paper itself considered entropy-coding its codebook indices and chose not to, "to maintain simplicity and speed" (Zandieh et al., 2026, paragraph "Entropy Encoding Codebook Pointers"). Decode cost is therefore exactly the question this paper has to answer. What would resolve it: a standalone GPU decode benchmark (e.g., one layer at 32k tokens, 8 KV heads) compared with fixed-rate 2-bit dequantization and with the attention kernel; the spread of rates per stream and per head (e.g., maximum and 99th percentile); and headline rates that include the offsets.
W2. The gains from the paper's new ingredients are modest, and the largest margins come from simplified or uncontrolled comparisons.
- Baselines. Both fixed-rate baselines are simplified re-implementations. TurboQuant-MSE leaves out the split into outlier and regular channels that the TurboQuant KV-cache experiments use, as well as the QJL residual. KIVI keeps no full-precision residual window (Sec. 5, App. Q.1). It is therefore unclear how much of the 8.8–31.3% margin at 2.125 bits survives against the published configurations. No calibration-free fixed-rate vector quantizer is included or cited, although one exists for exactly this setting. NSNQuant (Son et al., NeurIPS 2025) normalizes, centers and Hadamard-transforms each token so that it matches a standard normal, then quantizes with a single reusable codebook, with no calibration. This also makes inaccurate the statement in Sec. 2 that vector quantizers for the KV cache need calibration or training data. Without such a baseline, or at least a fixed-rate E8 codebook after the same rotation, the comparison cannot separate "vector vs. scalar" from "entropy-coded vs. fixed-rate".
- Prior use of the core principle. Sec. 1 presents its key idea as one entropy code, designed offline for a Gaussian source and applied after a random rotation. The same principle appears in distributed mean estimation: EDEN (Vargaftik et al., ICML 2022, Sec. 4.3) applies a randomized Hadamard transform and entropy-codes quantization intervals with probabilities computed from N(0,1). TurboQuant also discusses entropy-coding its indices with probabilities derived from the coordinate law. Neither is discussed. The paper should position what is new — the KV-cache setting, E8, data-free rate targeting and centering — against them.
- Matched control. The comparison that isolates the method's own components is Rot-ECSQ and the Z-lattice control, which share the rotation, rANS, rate targeting and centering. It gives 2.3%, 3.0% and 6.5% at 2.125 bits and 0.3–0.5% at 3.125 bits (Tables 2, 5), in line with the textbook 0.65 dB granular gain of E8.
- Seed and selection. Table 2 reports seed 0. At 2.125 bits HybridQuant scores 6.921 with that seed, about 1.4 standard deviations below its five-seed mean of 6.941 ± 0.014 (Sec. 5). Measured against the mean, the E8-over-Z gain on Qwen shrinks to about 2.0% and the cost of HybridQuant-G to about 0.9%. The paired-bootstrap intervals cover only variation over windows. The default configuration was also chosen on Qwen2.5-7B wikitext2, which is a reported test set (Sec. 5.3).
- Cross-paper comparison. The comparison with HyperQuant, OCTOPUS and TurboQuant (Sec. 5.1, Table 26) uses numbers reported under different harnesses (fp16 perplexity 7.25 vs. 10.03), with different protected layers, windows and compression ratios (7.5× vs. 2.9–3.0×). In the authors' own harness, TurboQuant-MSE^c loses 26.6% against the 63.0% reported, which shows how much harness and setting move these numbers.
- Concluding claim. The conclusion that calibration-free and entropy-coded quantization "need not be" a trade-off is never tested against a calibrated entropy coder. KVTC and CacheGen are excluded, and Remark J.4 states that the transform-coding gain given up was not measured.
Why it matters: the evidence currently supports a gain of roughly 2–6.5% perplexity at ~2 bits, and at most 0.5% at ~3 bits, for what the method adds over an entropy-coded rotated scalar quantizer. The larger margins rest on the comparisons above, and the enabling principle has precedents the paper does not discuss. What would resolve it: see Questions 2, 3 and 6, plus a related-work discussion of EDEN, TurboQuant's entropy-coding remark and NSNQuant.
Questions for authors
-
Can you report GPU rANS decode throughput for a full layer's cache (e.g., 32k tokens × 8 KV heads × K and V), next to fixed-rate 2-bit dequantization and the attention kernel time? Please also give the spread of the realized rate across streams and heads, and say how the partial stream is stored before it is encoded. If decoding is within a small constant factor of fixed-rate dequantization, or if you identify a target setting where decode cost is amortized and show the gains there, I would consider W1 largely resolved.
-
Can you compare against TurboQuant in the outlier-channel configuration of its KV-cache experiments (e.g., 2.5 and 3.5 bits) and KIVI with its 128-token full-precision window, at matched effective rates that count the fp16 window? A comparison with NSNQuant at 2 bits (its native setting, with side information counted), or with a fixed-rate E8 codebook after the same rotation, would separate the gain from vector quantization from the gain from entropy coding. If HybridQuant keeps a significant margin, for example over 5% perplexity at about 2.5 bits on Llama-3.1-8B, I would consider the baseline part of W2 resolved.
-
Please report the 2.125-bit rows of Table 2 averaged over the five seeds: HybridQuant, HybridQuant-G, the Z/Rot-ECSQ control, and TurboQuant-MSE^c, whose rotation is also seeded. Please also report tests that include seed variation. Is the default configuration still preferred when chosen on a split that is not used for reporting?
-
The attention output is a convex combination of the , so holds trivially. The bound in Thm 4.3 is with and , so it is at least . At the 2.125-bit step (, i.e., a design SNR of about 10.8 dB, cf. Table 20) this is about , weaker than the trivial bound once , i.e., whenever logits vary by order one. Prop. I.3 already gives a deterministic bound independent of T for any quantizer whose error stays in a bounded cell, with or without dither. What does Thm 4.3 add in practice? Could you evaluate the bound on real and compare it with the measured attention-output error? If there is a regime where the bound is informative, please state it; otherwise I suggest presenting the result as a qualitative statement, not a guarantee.
-
Thm 4.1 concerns the law of one coordinate over the random signs. The deployed code sees one fixed sign draw shared by all tokens, so for a systematic outlier that recurs in every token, the relevant object is the empirical distribution of one rotated pattern. Remark E.8 covers only the expected empirical CDF. Can you state, or measure, a version for a fixed draw? On real models, R = 0 gives a lower rate and a higher SNR than R ≥ 1 without dither (Table 20, keys: 2.108 bits / 10.86 dB vs. 2.124 / 10.73), and R = 1, 2, 3 cannot be told apart over five seeds (Sec. 5.3). Is there any real-model case, such as layers or heads with massive activations or another model family, where R = 1 or R = 2 measurably misses the target rate? A positive answer would raise my assessment of the theory's practical value.
-
[Suggestion] To support the conclusion that calibration-free and entropy-coded quantization need not trade off, report the transform-coding gain of Remark J.4, of the per-head K/V covariance eigenvalues, on the three models. Alternatively, compare with one calibrated entropy coder (KVTC or CacheGen) at matched effective bits.
6 comments
Response to Reviewer 3 (2/6)
Response to Reviewer 3 (2/6)
3. W2 (prior principle). EDEN, TurboQuant's entropy-coding remark, and the Sec. 2 claim about NSNQuant
Reviewer's point. The key idea, one entropy code designed offline for a Gaussian source after a random rotation, appears in EDEN and in TurboQuant's discussion of entropy coding, and NSNQuant contradicts the claim in Sec. 2 that KV vector quantizers need calibration.
Response: how it is resolved.
We concede both. NSNQuant is a calibration-free vector quantizer that came first, and Sec. 2 is corrected. EDEN already entropy-codes after a randomized Hadamard transform with probabilities, and TurboQuant describes entropy-coding its indices as an option without adopting it; both are now cited as precedents.
The revised paper positions the contribution against them: the KV-cache setting, E8 with a context-modelled static rANS code, data-free targeting of the total rate, centring, and attention computed on the compressed cache. HyperQuant is cited as concurrent work with a near-identical recipe.
Where in the revised paper: Sec. 1, Contributions (p. 2, red); Sec. 2, paragraph "Calibration and precedents" (p. 3, red); App. M, pointer to the revised Sec. 2 and the PCA-basis comparison (p. 46, red).
4. W2 (matched control, cross-paper comparison, concluding claim)
Reviewer's point. Against the matched Rot-ECSQ / Z-lattice control the gains are 2.3–6.5% at b and 0.3–0.5% at b, in line with E8's granular gain; the cross-paper comparison mixes harnesses; and the conclusion that calibration-free and entropy-coded need not trade off was never tested against a calibrated coder.
Response: how it is resolved.
Matched control: we agree that E8 itself adds roughly the textbook granular gain at about 2 b and little at about 3 b. The larger new gains come from components HQ-final adds; in the preregistered test-split ablation the fp16 window, the key channel scales and the sink each improve perplexity (real data).
Cross-paper comparison: we agree that harness differences dominate. The revised Sec. 5.1 relies on same-harness runs and keeps the HyperQuant / OCTOPUS protocol comparison only as an appendix check. Our HyperQuant re-implementation matches its paper at 3.0 bps but does not reproduce its reported 2.0-bps gains. An official HyperQuant implementation is public; we did not run it, so the non-reproduction has not been checked against the official code. This is a limitation of our comparison.
Concluding claim: it is now restricted to the calibration-free regime, and we concede that calibrated KVTC is better at low rates (see point 8).
Where in the revised paper: Table 24, component and rounds ablation (p. 64, red); Sec. 5.3, paragraph "Components and rounds" (p. 9, red); Sec. 5.1, paragraphs "Baselines" and "Perplexity" (p. 7, red); App. X.5 (p. 77, red); App. T.2, protocol of HyperQuant and OCTOPUS (p. 65); Sec. 6, Limitations and conclusion (pp. 9–10, red).
5. Q3. Five-seed means, seed-inclusive tests, and selection on a split not used for reporting
Reviewer's point. Table 2 reported seed 0, which flattered HybridQuant on Qwen; the paired intervals covered only windows; and the default was chosen on a reported test set. The reviewer asks for five-seed means of the 2.125-bit rows, tests that include seed variation, and selection on a split not used for reporting.
Response: how it is resolved.
All 2.125-b rows are now reported over 5 × 5 seeds, fully paired on one GPU, with intervals over windows and seeds (real data). Against TurboQuant-MSE$^c$ the submitted configuration is −7.3 / −12.4 / −20.3% (Qwen / Mistral / Llama, wikitext2), and against Rot-ECSQ −1.9 / −2.5 / −6.1%. All 12 cells exclude 0.
The paper's Llama margin of 31.3% came from seed 0, TurboQuant-MSE$^c$'s worst Llama draw; Sec. 5.1 and App. T.1 are corrected. The cost of the submitted HybridQuant-G is +0.90 to +4.04% over the six cells (5 seeds), every interval excluding 0.
Selection is now on validation splits only, confirmed on test, and the preregistered rule holds separately at each of seeds 0–4. Disclosure: test perplexities of the earlier centred mode had been seen before the rule was written.
Where in the revised paper: Table 34 (p. 82, red); Table 1 (p. 8, red); Sec. 5.1, paragraphs "Baselines" and "Perplexity" (p. 7, red); App. T.1 (p. 65, red); App. X.3 (p. 76, red); App. Q, disclosures (p. 57, red); Figure 7 (p. 63, red); Figure 9 (p. 69, red).
Response to Reviewer 3 (3/6)
Response to Reviewer 3 (3/6)
6. Q4. What does Theorem 4.3 add?
Reviewer's point. The Theorem 4.3 bound is weaker than the trivial bound whenever logits vary by order one, and Prop. I.3 already gives a deterministic T-independent bound. The reviewer asks what Theorem 4.3 adds, for an evaluation on real data, and otherwise to present it as a qualitative statement.
Response: how it is resolved.
(a) As stated, nothing: your calculation is right, and we replace the theorem. Because is a convex combination of the , holds trivially. The submitted bound with and is at least ; at b () this is about , weaker than trivial once , i.e. whenever logits vary by order one. With the paper's definitions it exceeds the trivial bound on all 552 measured head-contexts (simulation of a proved bound on real data).
(b) What replaces it: Theorem 4.3′ (proved; dithered modes only). Under (A1) subtractive Voronoi dither independent per position and block, (A2) no overload and (A3) exact arithmetic, each block error is uniform on and independent of the input (Zamir–Feder). Efron–Stein with the Payne–Weinberger inequality, applied block by block, gives for every unit vector
and the chain
with and . The chain shows where the submitted bound lost: it replaced each row's own step by the global , each row's deviation by the worst case , and the attention concentration by . Theorem 4.3′ keeps all three exact, and every member of the chain is independent of . Corollary A (proved) extends it to centred keys and adds the bias bound ; Corollary A′ (proved) extends Theorem 4.3′ up to the second inequality of the chain, Proposition I.11 and the bias bound to the guaranteed mode HQ-final-G, covering its fp16 sink and window rows, key scales and rate-exact steps. The last step, the comparison with the submitted constant , is not claimed.
(c) Evaluation on real (simulation of a proved bound on real data; 3 models, and ). Corollary A is below the trivial bound on 4968/4968 validation head-contexts and on 2208/2208 at 32k (evaluated for the earlier centred dithered mode, which shares the inner quantizer with HQ-final-G; not re-run for HQ-final-G itself). Theorem 4.3′ for the submitted dithered mode is below it on 4967/4968. The deterministic bound of Prop. I.3, in centred form (Corollary B), beats the trivial bound on only 8/1104.
(d) Where it is informative. Because each token is weighted by , the bound is informative when attention is spread over many coded tokens (small ), the typical case in our measurements, and approaches the trivial bound when attention concentrates on a single coded token with large logit noise (our reading of the formula; the paper does not measure the threshold). It holds per forward pass with the layer inputs fixed; during generation, later tokens depend on earlier dither draws through the sampled tokens.
For the undithered default the statement is qualitative, and we say so. No bound of this form can hold (Proposition N1, proved): duplicate value rows receive identical errors under the shared rotation, so their errors add coherently. On real caches the Corollary A expression is exceeded on 134/552 head-contexts, all in layer 0. The correlation-robust Theorem U, fully explicit, beats the trivial bound on only 38/552 head-contexts, and with Monte Carlo marginals it is a median 648× above the measured error. An informative attention-level guarantee therefore needs dither, which is why the paper offers it only in the guaranteed mode.
Response to Reviewer 3 (1/6)
Response to Reviewer 3 (1/6)
Thank you for an unusually concrete review. You stated what would resolve each point, that led us to answer against those criteria. W2 is answered in three parts (baselines, prior principle, matched control and conclusion).
1. W1 / Q1. Is the online, entropy-coded cache deployable?
Reviewer's point. Quality results simulated quantization; there was no GPU decoder or decode-throughput measurement; per-stream offsets were not counted; only run-level average rates were reported; and storage of the partial stream was not discussed. The reviewer asks for GPU decode throughput next to fixed-rate dequantization and the attention kernel, the rate spread across streams and heads, and how the partial stream is stored.
Response: how it is resolved.
Fused GPU kernels decode rANS inside attention, with no fp16 copy (real data). Against fixed-rate E8 decoding in the same kernel design, the entropy-coded kernel runs at 0.85× its speed at saturation and 0.72× at batch 1. The comparator is our own fixed-rate E8 kernel, not a standard 2-bit dequantizer; we believe 0.85× meets the "small constant factor" criterion.
Where the decode cost is amortized: capacity at the largest batch and storage/transfer. On the RTX 5090 (Llama-3.1-8B, 2.128 b stored) HQ-final holds 158 / 48 / 24 sequences at 4k / 16k / 32k against 162 / 44 / 21 for NSNQuant and 26 / 6 / 3 for fp16. Concessions: at batch 1 it is slower than fp16, and on the A100 its throughput is 0.54× / 0.78× / 0.59× fp16's.
Offsets: the block format stores int32 group offsets and charges them; the stored size of the submitted 2.125-b configuration is 2.128 b/elem against 2.124 b metered, so "every side bit counted" now holds for stored bytes.
Rate spread (real rANS bitstreams, submitted configuration, Qwen2.5-7B, one window): per-stream p99/mean is 1.0069 and max/mean 1.0117, so a paged allocator reserving 1.2% above the mean covers every stream measured. HQ-final's spread has not been measured.
Partial stream: rows that do not yet form a complete block are held in fp16, attended exactly and charged at 16 b (HQ-final's recent window). A block is rANS-encoded once it is complete, so reverse-order encoding is never a problem.
TurboQuant's remark that entropy coding costs simplicity and speed is fair, and we now cite it.
Where in the revised paper: App. Y.1, decode kernel against fixed-rate E8 and engine vs evaluation perplexity (p. 84, red); App. Y.2, rate spread per stream and head (p. 84, red); App. Y.3 (p. 85, red); Table 36 (p. 86, red); Sec. 3.3 (p. 6, red); App. B.2, Rate-exact rule and guaranteed mode (p. 21, red); Sec. 6, Limitations and conclusion (pp. 9–10, red); Sec. 2, paragraph "Calibration and precedents" (p. 3, red).
2. W2 (baselines) and Q2. TurboQuant's outlier configuration, KIVI's window, NSNQuant, and vector vs entropy-coded gains
Reviewer's point. Both fixed-rate baselines were simplified, and no calibration-free fixed-rate vector quantizer (NSNQuant) or fixed-rate E8 codebook was included, so "vector vs scalar" was not separated from "entropy-coded vs fixed-rate". The reviewer would consider the baseline part resolved by a margin over 5% at about 2.5 b on Llama-3.1-8B against TurboQuant's outlier configuration and KIVI with its window.
Response: how it is resolved.
TurboQuant (our re-implementation of its published channel split, with values at 2.125 b our choice; real data): at K 2.50 / V 2.125 b on Llama-3.1-8B the submitted configuration is −13.31% [−13.78, −12.88] (wikitext2) and −13.57% [−15.48, −12.01] (C4). This meets the >5 criterion even with the submitted configuration.
NSNQuant (official code at its native 2-bit setting, 2.238 b measured from its stored tensors): HQ-final is −6.44 / −9.21% on Llama-3.1-8B, −2.13 / −2.19% on Mistral and −7.89 / −3.61% on Llama-3.1-70B, with fewer bits; all intervals exclude 0. The submitted, uncentred configuration loses to NSNQuant; App. X.2 explains why, and centring alone is −0.7 to −2.7% against it at matched rate.
Fixed-rate E8 control (same rotation and centring, no entropy coder, exactly 2.125 b; real data): at a fixed lattice, rotation and centring, rANS vs fixed-rate E8 is −1.5% to −3.0% in perplexity.
KIVI with its window: see Reviewer 2, point 2.
Where in the revised paper: Table 1 (p. 8, red); Sec. 5.1, paragraphs "Baselines" and "Perplexity" (p. 7, red); App. X.2 (p. 75, red); App. X.4 (p. 77, red); App. X.5 (p. 77, red); App. X.9, entropy coding against a fixed-rate E8 codebook (p. 78, red); App. Q.1, Baseline implementations (p. 58, red).