acceptodds
Under review as a conference paper at ICLR 2027

HyperQuant: A Near-Entropy-Optimal Lattice Quantization Pipeline for Large Language and Diffusion Models

Abstract

Data-free post-training quantization has converged on rotation plus a fixed-rate codebook: whiten a weight tile toward Gaussianity, then round it to an MSE-optimal grid. That last step spends bits per index however unevenly the indices are used. We introduce HyperQuant (Hadamard, optimallY Packing, Entropy Rice-coding), which replaces that codebook with a lattice and a variable-length code in one calibration-free pipeline for both the linear layers and the KV cache of large language and diffusion transformers. A per-tile Walsh–Hadamard rotation whitens each tile, an optimal low-dimensional lattice (, , , or ) quantizes it, a lossless strip discards the bits lattice membership already fixes, and a Rice code carries the remainder to within bps of that lattice's rate–distortion ideal. Decoding is algebraic, so dimension costs no codebook memory. For the KV cache we add subtractive dither, which under an idealized sampler leaves every cached inner product unbiased per vector, and measurably so in practice, where rotation alone is unbiased only on average. Table1 collects operating points. On Llama-3.1-8B weights the default leads HIGGS at – bps, by PPL at over seeds. Neither OCTOPUS nor TurboQuant released usable code, so we reproduce both, validate each against published numbers, and run all three codecs in one harness under one itemized rate accounting over seeds. \method leads both at equal or lower effective rate, apart from a tie with both at bits and a loss to TurboQuant at bits on C4 under QJL; the leads hold at over shared seeds. The same pipeline quantizes the 19B-parameter LTX-2 video DiT with no visible artifacts. On one H100 at batch it cuts resident weight memory and the KV-cache encoding rate (computed rather than measured on a filled cache), though the decode kernel stays slower than bare bf16 GEMV.

Primary area:
optimization
Keywords:
Quantization, LLM, DiT

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mevrOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper introduces HyperQuant, a data-free post-training quantization pipeline for LLM weights and KV caches. It combines Hadamard rotation, low-dimensional lattice quantization, redundancy stripping, and variable-length Rice coding to reduce the average number of bits required for quantized representations. The paper also provides a theoretical analysis of the coding efficiency and introduces optional bias correction for KV-cache quantization. Experiments on language and diffusion models demonstrate improved rate–distortion performance over several existing quantization methods, while also revealing remaining limitations in inference throughput.

Critical strengths
  1. The unified quantization framework is practically relevant. HyperQuant applies the same core compression pipeline to both model weights and KV caches. The additional bias-correction analysis for KV quantization, including subtractive dither and its conditional unbiasedness guarantee, provides further technical depth.
  2. The paper makes a concrete effort toward GPU implementation. The authors identify the serial decoding bottleneck introduced by variable-length Rice codes and develop a parallel decoder to mitigate it. The implementation and performance analysis go beyond purely simulated quantization experiments.
Critical weaknesses
  1. The memory savings come with a substantial inference-throughput penalty. Although HyperQuant achieves promising compression ratios, the optimized parallel decoder remains approximately 1.85–1.92× slower than the BF16 baseline in the reported autoregressive decoding benchmark. This limits its immediate practicality for latency-sensitive LLM serving. A more comprehensive evaluation of the memory–throughput trade-off, particularly at larger batch sizes and against optimized quantized inference baselines, would strengthen the practical significance of the method.
  2. The presentation is mathematically dense and difficult to follow. Although the individual components are described in detail, the paper would benefit from a small, self-contained worked example illustrating the complete encoding and decoding process. In particular, showing how a low-dimensional lattice vector is quantized, stripped, Rice-coded, and reconstructed would make the method more accessible to readers without a background in lattice theory or information-theoretic coding.
Questions for authors
  1. Have the authors considered evaluating HyperQuant with KV caches stored in compressed form throughout autoregressive generation? In particular, how would the variable-length coding interact with FlashAttention-style kernels and long-context inference?

Besides, I'm still not confidence enough to say that I quite understanding the mathematical details of the paper. I've informed AC that my review on the mechanism of the paper may not be reliable.

Reviewer_y3weOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

HYPERQUANT is a data-free quantization pipeline for model weights and KV caches that combines Hadamard rotation, lattice quantization, and variable-length entropy coding. It also introduces optional dithering for unbiased inner-product estimation and a parallel GPU decoder. Experiments on two LLMs and a video diffusion model demonstrate competitive compression–quality tradeoffs against existing methods, supported by component ablations. Memory usage decreases, but inference remains slower than bf16.

Critical strengths
  • Combining lattice quantization with entropy coding improves on HIGGS at comparable rates, with gains persisting in dimension- matched comparisons. This supports the value of the coding approach beyond simply using higher-dimensional quantization.

  • Component ablations, effective-rate accounting, and repeated-seed comparisons help isolate the sources of improvement. Explicitly reporting baseline reversals and decoding slowdowns also makes the empirical claims more credible and their practical scope clearer.

Critical weaknesses
  • The optimized implementation remains roughly 1.9× slower than bf16, while KV-cache compression is calculated rather than measured on a populated cache. Thus, the paper demonstrates storage savings but does not establish a favorable memory–latency tradeoff in realistic long-context inference.

  • Comparisons rely on author reproductions, with a substantial unresolved discrepancy for 2-bit TurboQuant (+77.2% versus the published +63.0% perplexity increase). Matching synthetic distortion does not fully validate downstream behavior. Given the modest, configuration-dependent margins, this limits confidence in the claimed low-bit advantage.

Questions for authors
  • Which evaluations use physically compressed KV caches rather than pseudo-quantization? Do available measurements establish the actual memory–latency tradeoff with a populated cache, including encoding, decoding, and metadata overhead?

  • Which evaluations use physically compressed KV caches rather than pseudo-quantization? Do available measurements establish the actual memory–latency tradeoff with a populated cache, including encoding, decoding, and metadata overhead?

Reviewer_yiKkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

HyperQuant casts data-free post-training quantization as rate-distortion compression and replaces the prevailing "rotation + fixed-rate codebook" recipe with a single calibration-free pipeline for both linear-layer weights and the KV cache of large language and diffusion transformers. The encode path (Figure 1, §3, p.4-5) applies a per-tile Randomized Hadamard Transform to whiten each tile, quantizes it with an optimal low-dimensional lattice (E8, D4, A2, or Z), strips the bits that lattice membership fixes deterministically, and Rice-codes the remainder at a Gaussian-calibrated rate, decoding algebraically so dimension costs no codebook memory. For the KV cache it adds subtractive dither, proved (Appendix E) to make every cached inner product unbiased per vector. A parallel GPU Rice decoder is provided (§3.1, p.5). Validation (§4, p.6-9) is on Llama-3.1-8B-Instruct (WikiText-2, bf16 PPL 7.1606) and Qwen2.5-7B, against HIGGS (weights) and in-harness reproductions of TurboQuant/OCTOPUS (KV), plus an LTX-2-19B video-DiT stress test. The paper reports leads over HIGGS at 3-5 bps and over both KV baselines at equal or lower effective rate, with stated exceptions, and ~2.8×/~3.8× weight/KV compression on one H100. 、

Critical strengths

The reporting is unusually complete and honest, and the data are trustworthy. The paper states its benefits and bottlenecks plainly: Appendix G (p.37-38) collects the scope and threats to validity in one place, the abstract itself labels the headline KV figure "computed rather than measured" (p.1), the weight-vs-HIGGS gap is reported with multi-seed statistics and effect sizes rather than a single run (Table 17, p.24: 13.0σ at b=3, 1.9σ at b=4), the deterministic pipeline is shown to reproduce PPL exactly (Appendix D.10, p.24), and both KV reproductions are validated against references outside the paper (Appendices D.14-D.15). This matters because it lets a reader judge the true contribution without taking any claim on faith. The rate-distortion decomposition is precise. The gap over HIGGS is split into an entropy-coding component (recoverable by entropy-coding any fixed-rate codebook) and an unbounded-codebook component, quantified at matched dimension (A2 vs HIGGS-p2) in Table 18 (p.24) and corroborated by building actual Lloyd codebooks at p∈{2,4} in Appendix D.4 (Table 10, p.18). The analysis says exactly which part is the method's conceptual contribution and which follows from HIGGS's implementation cap — a level of precision that is itself a contribution.

Critical weaknesses

The headline practical gains are computed, not empirically measured. The abstract states the method "cuts resident weight memory ∼2.8× and the KV-cache encoding rate ∼3.8× (computed rather than measured on a filled cache)" (p.1). Appendix G (p.37) confirms the distinction: every KV-cache quality number comes from a harness that quantizes then immediately dequantizes to BF16, no experiment runs attention on the Rice-coded bitstream, and the KV resident-memory and throughput figures (Table 13, p.21) are computed from bitstream size and a standalone decode kernel. The 3.79× KV figure is a per-vector encoding rate, "distinct from a measured resident-memory footprint of a filled cache"; the actual HBM footprint at 32K/128K was not measured, and the KV-only resident-memory row is unchanged from baseline (14.99 vs 14.96 GiB, Table 13). The weight resident 2.8× (14.96→5.29 GiB) is measured at load, but the central KV-compression claim that motivates the work (§1) is projected, not demonstrated. It is slower than BF16, so the present practical benefit is likely minimal. §4.3 (p.7) states "this is not yet a throughput win": the parallel decoder is 1.85-1.92× slower end-to-end (bf16 39.5→12.7 tok/s single-thread, down to ~20.6-21.3), the kernel stays 2.0-2.2× off bare bf16 GEMV, and an unfused online-RHT cost of ≈13 ms/token (Appendix G, p.38) sits inside that gap. With no production-serving baseline (Appendix G, p.38), the paper itself concludes a practitioner cannot tell whether the memory win outweighs the throughput regression. Both weaknesses follow from the paper's own honest accounting, not from any error in it.

Questions for authors
  1. The margin over HIGGS is concentrated at low rate. At b=3 the E8-vs-HIGGS-p2 gap is 0.934±0.072 PPL (13.0σ, 4/4 seeds, Table 17, p.24), the figure the abstract leads with (p.1), but at b=4 — your default weight rate (Table 6, p.16) — it is 0.127±0.066 PPL, a 1.9σ effect (Table 17). At the rate you actually recommend for deployment, what justifies the variable-length lattice decoder and its throughput cost over a fixed-rate HIGGS codebook, given the quality margin there is under 2σ?
  2. What exactly is dimension-matched in the HIGGS comparison? The headline is E8 (8-D) vs HIGGS-p2 (2-D), whereas the decomposition isolating the entropy-coding contribution is A2-vs-HIGGS-p2 (both 2-D, Table 18, p.24), and Appendix D.4 (p.18) reports that higher-dimensional HIGGS codebooks cannot be built at b=3-5 (p=8, b=3 needs 2²⁴ codewords; p=8 is left to future work). At b=5 the matched-dimension decomposition attributes 1.79 dB of the 1.97 dB gap to unbounded-codebook structure and only 0.18 dB to entropy coding (Table 18). Please state precisely what is and is not matched in the headline E8-vs-HIGGS result, and how much of that lead reflects HIGGS's p≤2 implementation cap rather than the entropy-coding method — i.e., whether a same-dimension fixed-rate codebook would close most of it at the default rate.
  3. The abstract already labels the 3.79× KV figure "computed rather than measured on a filled cache" (p.1), and Appendix G (p.37) places a live filled-cache measurement among the items deferred to future work, so this concerns completeness rather than candor. Is there any measured resident-memory or throughput number — at any context length or batch size — that realizes the KV memory advantage end-to-end? If one can be produced, it would move the central KV-compression claim from projected to demonstrated and change the assessment.
  4. §3 asserts the RHT "folds into the preceding norm layer where possible," but Appendix D.8/G report no such fusion was found in the released integration, and the ≈13 ms/token figure assumes zero fold coverage. What fold coverage across the 224 call sites is actually achievable, and would the measured 1.85-1.92× end-to-end slowdown improve materially with it, or is it already near the floor?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.