- Rating
- 3
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors aim to maximize inference speed while preserving accuracy when quantizing LLM weights to 2 bits. To this end, they propose two techniques. Information-Aligned Analytic Coding (IAC) performs 2-bit quantization without sacrificing decoding speed, which includes Hadamard rotation for reshaping the weight distribution followed by an iterative optimization of symmetric per-row quantization levels. Knowledge-Intensive Calibration (KIC) selects sequences with large token-wise KL divergence from the teacher and uses them for effective distillation. As shown in Figure 1, FidelityQ achieves the lowest WikiText2 perplexity among the compared methods. It also decodes about 3.4 times faster than QuIP#, the prior method with comparable perplexity.
Critical strengths
A practical method that achieves both accuracy and speed
- FidelityQ achieves both accuracy and decoding speed with 2-bit scalar quantization. It outperforms existing per-row scalar methods across all models and settings in Table 1. Except on C4, it also surpasses the codebook-based AQLM, which decodes about two times slower.
- Moreover, its decoding speed matches DB-LLM while being much more accurate and is faster than QuIP# and AQLM as reported in Table 8.
Thorough analysis that justifies each design choice
- The paper supports the need for each component with analysis. It shows that rotation reshapes the weight distribution into a form that is easier to quantize (Fig. 3). It also shows that quantization error concentrates on a small fraction of tokens, which makes KIC effective (Fig. 4).
- The ablations on level placement, sequence selection, and calibration data (Tables 5-7) are also comprehensive.
Critical weaknesses
Limited novelty over existing techniques
- Most components of FidelityQ already exist in prior work: rotation in QuaRot, Lloyd-style non-uniform levels in SqueezeLLM and LeanQuant, the two-binary decomposition of 2-bit weights in DB-LLM, and teacher-guided calibration in BitDistiller.
- KIC is ineffective without IAC in Table 2, and the gain of its new part, sequence selection, is small in Table 6.
Slow prefill limits practicality
- Prefill reconstructs fp16 weights before computation, so it takes 7.32 ms/token, much slower than FP16 (0.27), QuIP# (1.12), and AQLM (1.98) as shown in Table 9.
- Since the speed advantage is limited to batch-1 decoding, it is unclear whether FidelityQ remains advantageous for long prompts or batch serving.
Questions for authors
(1) Novelty. Could you clarify which parts of FidelityQ are new compared with existing techniques (e.g. QuaRot, DB-LLM, LeanQuant, BitDistiller), and how much each contributes to the performance?
(2) Contribution of KIC. In Table 6, KIC's sequence selection shows only a small gain over random selection. Could you explain why, and report the mean/variance over multiple random seeds?
(3) Latency. Given the slow prefill speed, could you report end-to-end latency across prompt lengths (e.g., 1K-8K tokens) to show whether FidelityQ remains effective for long prompts?
4 comments
Q2:Why does KIC's sequence selection show a small gain over random selection in Table 6, and what are the mean and variance over several seeds?
A2: The gain in Table 6 is small on the corpus average because selection targets the high-divergence tail, and it grows with how much of that tail the selected set captures: across the four rules of Table 6, as the tail-KL mass of the selected set rises from 1.00× to 1.45× that of a random draw, Wiki2 falls monotonically from 17.05 to 16.69. On the worst 1% of held-out tokens, where selection aims, tail perplexity falls from 96.7 to 79.8 against a random draw of the same size (Fig. 4b). These tokens are a small share of the corpus, and selection acts on top of teacher-guided amplitude optimization, which already provides most of KIC's recovery (Table 2), so the whole-corpus increment is smaller than the local one. KIC's amplitude optimization trains the magnitudes of a mirrored grid, so it brings a similar gain on the MSE-clipped mirrored grid (Table R2 in our response to Reviewer tvS3), and Base's zero-including grid in Table 2 stays at its starting point by construction.
Table R4 gives the mean and standard deviation over three seeds per arm. Selection lowers the mean Wiki2 and C4 in every setting, and at W2A4KV4 on LLaMA-2-7B every selection seed beats every random seed on both metrics. On Qwen3-8B-Base, selection also raises the 7-task average and MMLU. We will report the contributions of amplitude optimization and selection separately in Sec. 4.3, and note in Table 2 that Base has no mirrored magnitudes for KIC to train.
Table R4. Tail-divergence selection against random selection, three seeds per arm, mean ± standard deviation (also in our response to Reviewer tvS3).
| Model, W-A-KV, budget | Random (Wiki2 / C4) | Tail-divergence (Wiki2 / C4) |
|---|---|---|
| LLaMA-2-7B, 2-16-16, 1024 | 7.15 ± 0.02 / 9.72 ± 0.01 | 7.12 ± 0.02 / 9.70 ± 0.02 |
| LLaMA-2-7B, 2-4-4, 1024 | 7.53 ± 0.01 / 10.12 ± 0.01 | 7.50 ± 0.01 / 10.11 ± 0.01 |
| Qwen3-8B-Base, 2-4-4, 512 | 10.75 ± 0.01 / 16.74 ± 0.02 | 10.73 ± 0.05 / 16.70 ± 0.09 |
Q3:Does FidelityQ remain effective in end-to-end latency for prompts of 1K to 8K tokens?
A3: The end-to-end measurements in Table R8 show that FidelityQ remains effective for long prompts: on LLaMA-2-7B at batch size 1, it is faster than FP16 at every prompt length from 1K to 8K tokens and runs at nearly the speed of the uniform 2-bit grid. Each run processes the prompt and then generates 256 tokens with the full 32-layer model, online Hadamard transforms, and causal attention, with decoding in CUDA graphs on an A100-SXM4-80GB. The advantage holds because each layer's weights are reconstructed once per prefill forward, so the cost is amortized over all prompt tokens, while every decoding step benefits from the faster 2-bit kernel. The 7.32 ms per token of Table 9 comes from the HuggingFace eager-mode fallback with a 128-token prompt, where this reconstruction is spread over few tokens; the prefill here costs about 0.08 ms per prompt token at every tested length (Table R8). Batch serving, which you also raise, behaves the same way: FidelityQ runs at a speed comparable to the uniform 2-bit grid at small batch sizes and is 21% faster at batch 128 (Table R7 in our response to Reviewer yBSp). We will add Table R8 and its protocol to Sec. 4.5 and App. D.
Table R8. End-to-end latency on LLaMA-2-7B, A100-SXM4-80GB, batch size 1, 256 generated tokens.
| Prompt | Prefill FP16 (ms) | Prefill FidelityQ (ms) | Prefill per prompt token, FP16 / FidelityQ (ms) | E2E FP16 (s) | E2E uniform W2 (s) | E2E FidelityQ (s) | Speedup |
|---|---|---|---|---|---|---|---|
| 1K | 65 | 86 | 0.063 / 0.084 | 2.42 | 1.28 | 1.32 | 1.84× |
| 2K | 125 | 157 | 0.061 / 0.077 | 2.54 | 1.42 | 1.45 | 1.75× |
| 4K | 259 | 310 | 0.063 / 0.076 | 2.80 | 1.69 | 1.73 | 1.62× |
| 8K | 553 | 645 | 0.068 / 0.079 | 3.34 | 2.28 | 2.31 | 1.45× |
Thank you for your detailed response. I think I would need to see a explicit technical contribution, or novelty beyond prior works to increase the score, but I didn't find in your answer. Therefore, I'll keep my rating at Weak Accept.
Dear Reviewer krzj,
Thank you for your timely reply and for your recognition of our work. Your comments have helped us improve the paper, and we sincerely appreciate your time and careful reading.
Best regards,
Authors
We thank you for the positive assessment. We are encouraged that you found FidelityQ "a practical method that achieves both accuracy and speed" and found our analysis "thorough" in justifying "each design choice". Following your questions, we compared FidelityQ part by part with QuaRot [1], DB-LLM [2], LeanQuant [3], and BitDistiller [4] (Q1), separated KIC's magnitude optimization from sequence selection over three seeds (Q2), and measured end-to-end latency for prompts of 1K to 8K tokens and batch sizes up to 128 (Q3). Each part of FidelityQ adds a measurable gain over the closest prior method, selection brings a stable gain over random selection, and FidelityQ is faster than FP16 end to end at every tested prompt length. We address each point below.
A1:The comparison below shows that each part of FidelityQ serves predictive fidelity with a technique that differs from the named method, and that each part adds a measurable gain over its alternative in our pipeline. FidelityQ targets predictive fidelity under extreme compression, keeping the predictions of the full-precision model at 2.007 bits per weight with per-row scales and lookup-free decoding.To our knowledge, FidelityQ is the first to obtain the minimum-distortion member of the binary-coding family, with a structure determined by the source distribution, and to calibrate only the two per-row amplitudes that this member exposes, so that calibration leaves the decode unchanged.
Each part contributes measurably on LLaMA-2-7B at W2A16. Without rotation, 2-bit RTN diverges (Table 10), and with it plain squared error matches the layer loss to about 10⁻⁴ (App. A). Cumulatively, IAC and KIC lower Wiki2 from 9.56 for Base, which pairs QuaRot's rotation with a uniform grid, to 7.78 and then 7.11 (Table 2). Among lookup-free mirrored grids trained identically, IAC's placement is lower on C4 in every seed than the clipped members of the same family (Table R2 in our response to Reviewer tvS3), and without a lookup table it keeps the speed of uniform scalar quantization, comparable at small batch sizes and 21% faster at batch 128 (Table R7 in our response to Reviewer yBSp). Replacing the mirrored basis by DB-LLM's {0, 1} basis in our pipeline raises C4 at both W2A16 and W2A4KV4 (Table 5). KIC trains only the level magnitudes in half the time of EfficientQAT [5], which trains all weights, and leads it on all three metrics at the same per-row granularity (Table R3 in our response to Reviewer tvS3). We will state these differences and contributions in Sec. 2 and Sec. 4.3.
[1] QuaRot: Outlier-free 4-bit inference in rotated LLMs, NeurIPS, 2024.
[2] DB-LLM: Accurate dual-binarization for efficient LLMs, Findings of ACL, 2024.
[3] LeanQuant: Accurate and scalable large language model quantization with loss-error-aware grid, ICLR, 2025.
[4] BitDistiller: Unleashing the potential of sub-4-bit LLMs via self-distillation, ACL, 2024.
[5] EfficientQAT: Efficient quantization-aware training for large language models, ACL, 2025.