- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ApexQuant, a calibration-free elastic quantization framework that progressively quantizes residual errors using independent random rotations and reusable codebooks, supported by a theoretical analysis of geometric error decay. Experiments across language, remote-sensing, and medical models demonstrate competitive quantization performance.
Critical strengths
- The paper provides a theoretically grounded formulation of residual re-isotropization and progressive error decay.
- ApexQuant supports multiple precision levels through a single nested representation and accommodates scalar, E8 lattice, and trellis quantizers. This flexibility is valuable for memory-constrained deployments
- The experiments cover several language models and two additional application domains, with comparisons against multiple data-free quantization methods.
Critical weaknesses
- Despite achieving the lowest KL divergence among the evaluated data-free baselines at approximately 2 bits per weight, ApexQuant still suffers substantial degradation in downstream performance on several models.
- The central theoretical guarantees rely on independently sampled Haar-random orthogonal rotations, whereas the implementation adopts randomized Hadamard transforms for computational efficiency. The authors empirically demonstrate that the discrepancy is small on evaluated models, but no corresponding theoretical error bound is established for the practical transform.
- The reported 1.56× decoding speedup is obtained using a simulated Mistral block stack rather than a complete autoregressive inference pipeline. The reported speedup may not directly translate into comparable improvements in real-world LLM serving.
- The empirical comparison with existing elastic quantization methods is not fully controlled. Several comparisons in Appendix G rely on previously published results rather than unified reproduction.
Questions for authors
- Given the substantial performance degradation observed on several models at approximately 2 bits per weight, could the authors clarify which practical deployment scenarios can tolerate this level of degradation?
- Can the authors provide theoretical error bounds or a more systematic sensitivity analysis for replacing Haar-random rotations with randomized Hadamard transforms?
- How does ApexQuant perform in a complete LLM inference pipeline, including attention, KV-cache operations, and prefill? Could the authors report end-to-end latency, throughput, and peak GPU memory consumption under realistic batch sizes and mixed-precision serving workloads?
- Could the authors provide a more controlled comparison against representative elastic quantization methods under consistent models, evaluation datasets, and storage budgets?
2 comments
End-to-end serving and elastic baselines
W3. End-to-end serving
We agree that the block-stack benchmark in Table 11 offers only a limited view of end-to-end serving. It measures the linear layers of a 32-block stack at a single decode step and does not include attention, the KV cache or prefill. We now state this in the main text. We also make clear that the speedup is measured against storing one artifact per precision and not against FP16.
To evaluate the complete pipeline we integrated ApexQuant into vLLM as a quantization plugin. Attention, KV cache, prefill and batching are therefore handled by vLLM itself. The table below summarizes the results, which we report in a new appendix. On both models the 4-bit and 6-bit tiers stay close to BF16. On a single B200 GPU the 70B weights take 38.0 GiB at four bits instead of 131.4 GiB. This frees memory for the KV cache, which grows from 94k to 318k tokens and allows larger batches. For the 70B model the first layer has to be kept in BF16, which adds 0.15 bits per weight. Without it the 2-bit and 4-bit tiers collapse and we report this in the Limitations section.
| Model | Precision | MMLU | Weights | KV cache (tokens) |
|---|---|---|---|---|
| Llama-3.1-8B | BF16 | 0.683 | – | – |
| 4-bit | 0.671 | – | – | |
| 6-bit | 0.682 | – | – | |
| Llama-3.1-70B | BF16 | 0.825 | 131.4 GiB | 94k |
| 4-bit | 0.823 | 38.0 GiB | 318k | |
| 6-bit | 0.825 | 54.3 GiB | 303k |
The plugin does not yet use the fused kernels of Table 11 and in its current form it is slower than BF16. We therefore do not report its throughput and our latency claims remain limited to the fused-kernel measurement. Porting the fused kernels into vLLM is future work and we have noted this in the Limitations section.
W4. Controlled comparison with elastic methods
Of the elastic methods, only Any-Precision LLM has released code, and we reproduced it with the authors' own pipeline under our evaluation protocol (Table 10). At four bits ApexQuant reaches a perplexity of 5.613 against 5.624, without any data, while at two bits Any-Precision LLM is better (35.21 against 71.80). The same reproduction shows why published numbers must be used with care: the values reported for Any-Precision LLM by Wang et al. (2026) (6.05 and ) do not match the 5.624 and 35.21 we obtain. MoBiQuant and AnyBCQ have not released code, so we can only report their published values, which we flag in the table. MatQuant requires quantization-aware training on 10–100M tokens and is therefore outside the post-training setting. In a new appendix we also compare all methods under one protocol on Llama-3.1-8B and two MoE models, including the calibrated GPTQ and AWQ and the data-free DASHQuant, SINQ and HQQ. At four bits ApexQuant has the lowest KL among methods with comparable storage.
We thank you again for the constructive questions. We hope the new end-to-end results and clarifications address your concerns.
References
- A. Gersho and R. M. Gray, Vector Quantization and Signal Compression, Kluwer, 1992.
- J. Chee, Y. Cai, V. Kuleshov, and C. De Sa, "QuIP: 2-Bit Quantization of Large Language Models With Guarantees," NeurIPS, 2023.
- N. S. Jayant and P. Noll, Digital Coding of Waveforms, Prentice Hall, 1984.
- V. K. Goyal, "Theoretical Foundations of Transform Coding," IEEE Signal Processing Magazine, 18(5), 2001.
- R. Ben-Basat, W. Kuszmaul, M. Mitzenmacher, A. Portnoy, and S. Vargaftik, "Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven," arXiv:2605.06014, 2026.
- T. Zilca and G. Mendelson, "Approximating Uniform Random Rotations by Two-Stage Structured Hadamard Rotations in High Dimensions," arXiv:2604.23418, 2026.
2-bit cost and Hadamard vs. Haar
Thank you for the thorough reading and the constructive questions. We are glad you found the theoretical formulation grounded and the flexibility of the nested representation valuable for memory-constrained deployments.
W1. The cost of data-free quantization at 2 bits
You are right about substantial downstream degradation at two bits for some models. After diving deeper in this direction, we realized that at this bit-width, quality depends as much on the geometry of the weights as on the quantization method, and this has an information-theoretic explanation. For a linear layer with weights W and inputs x, the output distortion of a quantized layer W^ is the proxy loss Ex∥(W^−W)x∥2=tr((W^−W)H(W^−W)⊤), where H=Ex[xx⊤] is the second-moment matrix (Hessian) of the layer inputs. A calibrated quantizer such as QTIP estimates H and can approach a distortion that scales with the geometric mean λ~ of the eigenvalues of H, which lower-bounds what Hessian-aware rounding achieves (Gersho & Gray, 1992; Chee et al., 2023). A data-free quantizer, however, cannot know which directions are sensitive. The random rotations in ApexQuant deliberately isotropize the error, spreading it evenly across all directions. The distortion therefore scales, up to a correction along the direction of the weights themselves, with the average eigenvalue λˉ=tr(H)/d. The ratio λˉ/λ~ is therefore a fixed penalty, and because each additional bit reduces distortion by a factor of four, the cost of forgoing calibration is approximately G=21log2(λˉ/λ~) bits (Jayant & Noll, 1984; Goyal, 2001). This is a high-rate approximation and at two G is an upper bound on the penalty.
This may explain why the penalty varies across models. We expect it to depend on how spread out the eigenvalues of H are. This would explain why the encoders Clay and UNI keep most of their accuracy at two bits (Table 4) while LLaMA2 does not, although we have not measured this spread directly. For LLaMA2-7B (Table 10), ApexQuant at 4 bits (perplexity 5.61) performs comparably to QTIP at 2 bits (perplexity 5.86), suggesting that the penalty G for this model is at most roughly two bits, since this gap also includes ApexQuant's per-pass rate loss and QTIP's inter-block feedback.
We have added this discussion to the Limitations section, together with a recommendation of four bits and above as standalone tiers, where accuracy stays close to full precision without any data. The 2-bit tier remains useful as a prefix of the 4-bit model, for example as a fallback under memory pressure or for progressive download, and on some models it retains most of the performance (84.9% of the full-precision above-chance accuracy margin on Qwen2.5-7B in our zero-shot results). In our new experiments the 2-bit prefix also remains usable on the MoE model Qwen3.5-35B-A3B with an MMLU of 0.718, while the other data-free methods fall to chance. On Llama-3.1-8B it is not usable, and on the BFCL tool-calling benchmark it halves the score. These results are consistent with this recommendation.
W2. Haar in theory, Hadamard in practice
Thank you for this comment. Recent work has closed most of this gap and we have updated the manuscript to highlight this (Section 3). A single randomized Hadamard transform behaves like a random rotation for flat inputs, and two composed transforms do so for every input (Ben-Basat et al., 2026; Zilca & Mendelson, 2026). For our scalar stage these results carry over directly to the per-pass error and therefore to the decay law. Only the first pass sees the raw weights; after it the residual is approximately flat, which is the regime the single-transform bound covers. This matches our measurements: the per-pass ratio departs from the Haar prediction by at most 1.6% on individual layers at the first pass and by less than three parts in ten thousand at every later pass. Composing two transforms per pass is a cheap option where a guarantee for the first pass is needed. On real weights the difference is small and replacing the Hadamard transform by a dense random rotation changes the layer error by at most 0.2%, on both Clay and Phi-1.5 and at both group sizes we use (d=256 and 512). For the E8 lattice the same guarantee holds with three composed transforms (Ben-Basat et al., 2026). For the trellis no such result is known and the agreement remains empirical, which we state as a limitation. The first layer of Llama-3.1-70B is the one case where we needed an exception (see W3), and we have not yet tested whether composing transforms removes it.