- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces CanonQ, a QAT framework for compressing weights, activations, and KV cache (2-4 bits). It applies a Hadamard–Diagonal–Permutation transform plus normalization to map to each tensor block into canonical coordinates that approximate a standard Gaussian. Then, each block is quantized with a frozen scalar or vector codebook. The codebook only depends on bitwidth and d (dimensionality hyperparameter), so it is reused across layers and models. QAT adapts the network to the combined quantization errors. Two variants are compared: CanonQ-W (new transform applied to weights only) and CanonQ-Omni (applied to all 4 tensor types).
Experiments are reported on Llama3-1B (primarily), as well as Llama3-3B/8B, Qwen3-1.7B, and MobileLLM-Pro-Instruct-1B. The main evaluation is performed on wikitext-2 and 8 standard zero-shot tasks. It is briefly extented to code generation and reasoning tasks (for a weight-quantization case only).
Under aggressive quantization (W2A4KV2), CanonQ-Omni shows consistent and remarkable perplexity/accuracy gains compared to the reported quantized baselines.
Critical strengths
- the proposed quantization scheme uses a simple and reusable design: same codec construction applies to all tensor type (W/A/KV), and the resulting frozen tables are reused across layers and models. This design favors practical applicability of the technique.
- CanonQ-Omni is shown to be most successful in the hardest regime when activations and KV cache are quantized. The QAT baselines degrade dramatically while CanonQ-Omni retains reasonable performance (not achieving iso-accuracy but a remarkable improvement nonetheless)
Critical weaknesses
- novelty is limited: rotating, normalizing, using Gaussian-optimal codebooks is close to other works such as QuIP# [1], TurboQuant [2], HIGGS [3]. The primary contribution is applying known techniques together to all four tensor types (W, A, KV) in combination with QAT
- the CanonQ-W variant is a useful intermediate step between the ParetoQ baseline and CanonQ-Omni, showing the impact of the proposed approach applied only to weights. The results is that CanonQ-W beats ParetoQ only under some quantization settings, localizing the most relevant gains to activations and KV (keeping in mind that all runs a single seed, an additional limitation of this work). However, the W-to-Omni step introduces several changes together: rotation, norm-based scaling, an extra quantization level, Gaussian-optimal level placement. Intermediate QAT baselines, especially rotation + uniform quantizers on activations/KV, would have helped isolating the contribution of rotations (potentially very impactful) from the Gaussian-optimal codebook
- arguably the main barrier to QAT adoption is its compute budget. The paper has no kernels, throughput, or memory measurements, so the impact of on-the-fly rotation, normalization and nearest-codeword lookup is not assessed. At least, wall-clock training time or GPU-hours per model size / method should have been reported, along with CanonQ per-step overhead relative to ParetoQ.
[1] Tseng, Albert, et al. "Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks." Proceedings of machine learning research 235 (2024): 48630.
[2] Zandieh, Amir, et al. "Turboquant: Online vector quantization with near-optimal distortion rate." International Conference on Learning Representations. Vol. 2026. 2026.
[3] Malinovskii, Vladimir, et al. "HIGGS: Pushing the limits of large language model quantization via the linearity theorem." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.
Questions for authors
As per Critical Weaknesses, can the authors please comment on:
- isolating contribution of rotations from codebook on activations / KV
- training time, training steps, quality as a function training steps, CanonQ overheads
2 comments
Follow-up on our rebuttal
Thank you again for your review. As the discussion period draws to a close, we would appreciate your feedback on whether the additional results in our rebuttal address your concerns.
Novelty, Training Efficiency, and Convergence
We thank the reviewer for their careful assessment. We are encouraged by their recognition of CanonQ's reusable W/A/KV design and its gains under joint low-bit quantization. We address novelty, component contributions, and training cost below.
1. Novelty and supporting evidence
To our knowledge, CanonQ is the first QAT framework to jointly quantize weights, activations, and KV caches using a shared, frozen Gaussian-reference codebook design. Each codebook is designed once per bitwidth and tuple dimension and reused across layers and models. QuIP# and HIGGS focus on weight compression, while TurboQuant addresses online vector quantization, including KV caches.
The challenge is making fixed codebooks work across different tensor distributions that change during training. Weight updates alter activations and caches, and their quantization errors interact. CanonQ uses HDP and energy normalization to make the reference codebooks reusable, then jointly trains the network with all W/A/KV quantizers active. Theorems 1–3 connect distribution mismatch, reconstruction error, task loss, and gradient error (Secs. 3–4).
The W2A4KV2 results demonstrate the benefit of this joint design:
Keeping the canonical weight quantizer fixed, extending CanonQ-W to CanonQ-Omni improves 1B accuracy by 11.07 points and reduces PPL from 64.40 to 15.42 (Sec. 5.2, Table 1).
2. Stronger baselines and controlled component ablations
At matched 240K updates (31.5B tokens), CanonQ achieves 15.42 PPL / 53.56 mean accuracy, versus 27.57 / 48.04 for ParetoQ + QuaRot-style A/KV quantization: a 5.52-point accuracy gain.
To separate the component contributions, the following 120K-update controls keep the canonical weight codec, training settings, and alphabet cardinality fixed (16 A4 levels and 4 KV2 levels):
Gaussian codebooks contribute beyond rotation and normalization. With HDP and energy fixed, Gaussian levels improve PPL from 17.11 to 16.27 and accuracy from 51.21 to 52.28. Under absmax and HDP, they also improve both metrics: 23.33 → 19.02 PPL and 49.16 → 50.82 accuracy. The complete configuration achieves the best results across all six controls (Sec. 5.4, Table 3; Appendix C.2).
Separately, three 240K CanonQ runs on 1B achieve 53.61 ± 0.67 mean zero-shot accuracy (mean ± sample SD, n = 3; Appendix C.2).
3. Training cost and convergence
On eight A100-80GB GPUs, with 131,072 tokens per update, the measured W2A4KV2 training costs are:
Measurements use 600 updates, excluding the first 100 for warmup. CanonQ uses 19.1% less time per update and 5.0% less peak memory than the rotated baseline; memory is also 21.7% lower than the uniform baseline (Appendix C.8).
Quality at matched training checkpoints is:
CanonQ at 15K updates already exceeds the rotated baseline at 240K on both metrics, using one-sixteenth as many updates. It also leads at every matched checkpoint (Appendix C.1).