- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes SQuAT, a quantization-aware training (QAT) method for vector quantization. It decomposes weight vectors into shape and gain components and applies separate straight-through estimators (STEs) to the two quantizers. The method uses E8-derived spherical lattice codebooks, including the proposed E8S codebook. It is evaluated against several post-training quantization (PTQ) and QAT baselines across multiple models and benchmarks.
Critical strengths
- The factored STEs are well motivated and have a clear geometric interpretation under idealized SGD updates.
- The evaluation results demonstrate competitive accuracy compared with existing methods across different models and benchmarks.
- The implementation is efficient, with training time approaching that of BF16 in the reported benchmark.
Critical weaknesses
- The evaluation is limited to E8-derived codebooks (E8S and E8P12). The generalizability to other vector quantization formats remains unclear. In addition, E8S currently lacks an efficient dequantization kernel for inference, limiting its practical deployment.
- The method introduces two changes simultaneously: spherical shape-gain parameterization and factored STEs. Table 10 compares three configurations: Euclidean parameterization with joint STE, Euclidean parameterization with factored STE, and spherical parameterization with factored STE (SQuAT). However, the fourth configuration, spherical parameterization with joint STE, is missing. Although it is analyzed theoretically in Appendix B.1, an empirical comparison would help disentangle the contributions of the parameterization and STE design.
- Some results appear inconsistent across tables. For example, Tables 1 and 10 report identical Spherical results but different Standard VQ-QAT results for Llama-3.2-1B. Similarly, Tables 2 and 5 report different results for 1B-token INT-QAT using WSD and Hadamard rotation for Llama-3.2-1B.
Questions for authors
None. Please reply to the weaknesses.
3 comments
Thank you for the reply. I look forward to the pending experimental results. Overall, I maintain my positive assessment of the paper.
Thank you for your positive assessment. As promised, the runs with spherical parameterization and a joint STE have now finished. The table completes all four combinations of parameterization and STE with Wiki2 perplexity after 1B tokens at 2.25 bpw
| Parameterization | STE | L1B | Q4B |
|---|---|---|---|
| Euclidean | joint (Standard VQ-QAT) | 14.22 | 10.17 |
| Euclidean | factored (App. B.2) | 14.01 | 10.15 |
| Spherical | joint (App. B.1) | 13.86 | 10.16 |
| Spherical | factored (SQuAT) | 13.85 | 10.07 |
Both the parameterization and the STE choice are needed for the best results. SQuAT gives the lowest perplexity on both models, and on Q4B it is the only configuration that improves over Standard VQ-QAT. SQuAT also improves over the spherical joint STE variant on downstream tasks for Q4B
| Q4B | CSR | MMLU | GSM8K | IFEval |
|---|---|---|---|---|
| Spherical, joint STE (App. B.1) | 70.15 | 67.07 | 76.19 | 81.01 |
| SQuAT | 71.69 | 67.29 | 77.03 | 82.26 |
As noted in our rebuttal and in App. B.1, the joint STE variant is somewhat artificial since the Euclidean weight is formed only to place the STE. We will add these results to Table 12.
General Response + Response to Weaknesses and Questions
We thank all reviewers for their constructive feedback. We are glad that the reviewers found the factored STE and the spherical parameterization novel and well motivated (rztw, J7ZN), the gradient analysis clear (gQL3), and the implementation efficient, with practical training speedups (gQL3, J7ZN). We have revised the paper accordingly, and all changes are highlighted in yellow. Since the revision merges Tabs. 1 and 2 and adds new tables, this rebuttal uses the numbering of the submitted version and marks new tables as “revised Table X”.
Decoding kernels. As requested by gQL3 and J7ZN, we developed fast decoding kernels for E8S and added them to the revised paper (Sec. 5 and App. F). Following standard practice for low-bit weight quantization, we measure at batch size one, the memory-bound regime of autoregressive decoding, on an NVIDIA L40S (the memory bandwidth class of the RTX 4090 used by QuIP#). We report end-to-end decode throughput at context length 512, averaged over three seeds, against the official QuIP# E8P12 kernel. Percentages give the change relative to E8P12 at 2.0 bpw.
At 2.0 bpw E8S decodes as fast as E8P12, and both run 3.1 to 3.2 times faster than FP16. At 2.25 bpw E8S-SG decodes 4.8% to 5.1% faster than E8P12-SG. App. F gives the kernel design, table sizes, GEMV latencies and comparisons with INT2 kernels.
Seeds on all three models. As requested by rztw, we repeated the comparison of SQuAT and Standard VQ-QAT with three seeds on L3B and Q4B at 1B tokens (App. E.2, revised Tables 9 and 10). SQuAT reaches a perplexity of 10.94 ± 0.03 on L3B and 10.02 ± 0.02 on Q4B, against 11.09 ± 0.04 and 10.19 ± 0.02 for Standard VQ-QAT. Together with L1B (Table 8), the perplexity differences exceed the standard deviation for all models and budgets, and the ranges do not overlap in any 1B-token experiment. We have started 5B-token runs and will post the results here if they finish in time, or include them in the final version.
Response to weaknesses and questions
W1. Only tested E8 codebooks, and E8S has no efficient inference kernel.
We agree that the evaluation covers only E8-derived codebooks, and Sec. 7 of the submitted version stated this limitation (line 467), together with the lack of an efficient E8S dequantizer (line 471). The second point no longer holds, as described in the General Response. We have since built batch-one decoding kernels for E8S that decode as fast as E8P12 at 2.0 bpw (Sec. 5 and App. F). We revised the limitations part and kept the accuracy claims limited to the two codebooks we test.
The method itself does not depend on E8. SQuAT is a training rule for any spherical codebook with exact nearest-neighbor assignment and a scalar gain quantizer. The two codebooks we test are also built differently. E8P12 is a structured subset designed for fast decoding, while E8S consists of whole lattice shells. Shape-gain training with SQuAT improves both (Tab. 3, and Tabs. 12 and 13 without Hadamard rotations).
W2. Missing Spherical parameterization with a joint STE.
We agree that Tab. 10 does not include this configuration, and we analyze it in App. B.1. It is a less natural choice, since with spherical parameters the Euclidean weight exists only as an intermediate quantity formed to place the STE. App. B.1 shows that it yields the same tangent update for the direction and radial update for the gain as SQuAT, with one difference. Its gradients use the unquantized direction and gain, whereas SQuAT uses the quantized values from the forward pass, so SQuAT’s updates are consistent with the model actually seen during training. This consistency is the specific contribution of the factored STE on top of the spherical parameterization. We are running this configuration on L1B at 1B tokens and will post the results before the end of the discussion period if they finish in time, and add them to Tab. 10 of the final version otherwise.
W3. Some results are inconsistent across tables. We checked the experiment logs and confirmed that the main text has the most recent results. Appendix Tabs. 5 and 10 came from earlier reruns and were not updated. Code, hyperparameters, seeds and setup were unchanged, and we attribute the differences to non-deterministic distributed training across A100 and H100 nodes. The conclusion is unchanged, since SQuAT (13.85 / 58.8 / 40.3) outperforms the Standard baseline in both Tab. 1 (14.22 / 58.4 / 39.3) and the submitted Tab. 10 (14.10 / 58.7 / 37.5). We updated Tab. 10 (revised Tab. 12) and are recomputing Tab. 5, which we will post here if it finishes in time and otherwise include in the final version.