- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes E8-QAT for 2-bit weight quantization. It starts from E8-PTQ and uses arithmetic E8 projection with an identity STE during QAT. The trained weights are stored using 16-bit core indices and sparse records for points outside the core. Results on Qwen3.6-35B-A3B and Llama-3.1-8B show better accuracy than LC-QAT, with less memory and supports a larger batch size.
Critical strengths
- E8 projection jointly quantizes eight weights without explicit codebook search during QAT. Sparse escape records preserve the selected points at export with little additional storage in the reported experiments.
- The paper clearly describes the core codebook, escape encoding, and storage overhead, and provides decoding checks in the appendix.
- E8-QAT improves over LC-QAT on both models. And the evaluation covers basic tasks, code generation, function calling, and instruction following.
- The quantized model substantially reduces memory use and supports a larger batch size on one GPU.
Critical weaknesses
- E8-PTQ already outperforms UniSVQ before QAT, as shown in Figure 4 and Table 1. The comparison with LC-QAT therefore does not isolate the benefit of the proposed training strategy.
- Exact escape storage adds little memory, but its impact on GPU kernel efficiency during inference is not quantified or demonstrated.
- Figure 3 compares BF16 and E8-QAT throughput at their respective maximum batch sizes. This comparison mainly reflects the larger batch enabled by reduced memory use. Single-request inference efficiency is not reported.
Questions for authors
- Why are the 8,655 boundary points selected lexicographically? How sensitive are PTQ accuracy and escape rates to alternative selections from the same shell?
- Could author evaluate additional dense and MoE models from other model families to assess whether the gains generalize across architectures?
2 comments
Response to Reviewer MzgK
Q1: “Why are the 8,655 boundary points selected lexicographically? How sensitive …?”
A: We thank the reviewer for their interest in the lexicographic tie-break. We conducted an experiment on the core-selection rule. Across six rules on post-Hadamard weights from three Llama layers, the reconstruction SNR changes by 0.089, 0.041, and 0.041 dB, which is 4–9% of the 1.022 dB gain.
Lexicographic order makes the core and the index numbering deterministic. Every rule keeps the same 56,881 points with , 86.8% of the core. The tie-break only fills the remaining 8,655 slots from the 60,480 points on the shell . Those candidates have one common norm and form a single orbit of , so the rule only rotates the kept subset. Stage 1’s incoherence transform makes the weights approximately isotropic, so that rotation does not matter in expectation.
On post-Hadamard weights, 120,000 groups from each of three Llama layers and projections, six rules — lexicographic, reverse lexicographic, three random draws, and whole sign-orbits — differ by 0.089, 0.041, and 0.041 dB in reconstruction SNR, and by less than 0.3 percentage points in the out-of-core rate. That spread is 4–9% of the 1.022 dB gain. Whole orbits give 65,529 points, not exactly .
Section 4 is revised to:
Lexicographic order fixes a reproducible core and index numbering. Every rule keeps the same 56,881 points with . The tie-break only fills the remaining 8,655 places from the 60,480 equal-norm points on the shell , one orbit of . On post-Hadamard weights, 120,000 groups from each of three Llama layers and projections, lexicographic order, reverse lexicographic order, three random draws, and whole sign-orbits differ by 0.089, 0.041, and 0.041 dB in reconstruction SNR and by less than 0.3 percentage points in the out-of-core rate. The spread is 4–9% of the 1.022 dB gain. Whole orbits give 65,529 points rather than . These are Stage-1 distortion differences.
Q2: “Could the authors evaluate additional dense and MoE models from other model families?”
A: We thank the reviewer for their interest in whether the gain extends to further models. We conducted an experiment on Mistral-7B-Instruct-v0.3 and Qwen3-4B with the same 375-step recipe. On Mistral-7B-Instruct-v0.3, E8-QAT leads by 2.89 on avg4, 67.51 against 64.62, and lowers WikiText from 14.35 to 11.45. On Qwen3-4B, E8-QAT is higher on all six accuracy metrics.
Mistral-7B-Instruct-v0.3 has the same feed-forward shape as Llama-3.1-8B, so Stage 2 uses the same 375-step schedule and the learning rate given by the rule. Both legs quantize the feed-forward projections, share the Hessian, and train with effective batch 16 and sequence length 32,768, the model's native window. The corpus is re-rendered with the Mistral chat template; all 2,000 sequences are kept and 79.8% of tokens remain. The scalar Stage-1 scale is the Llama value 0.83. The table compares the two trained endpoints.
| Method | ARC-C | ARC-E | HellaSwag | PIQA | WinoGrande | BoolQ | WikiText↓ | Avg4 | Avg6 |
|---|---|---|---|---|---|---|---|---|---|
| LC-QAT | 43.69 | 69.07 | 70.73 | 74.92 | 69.14 | 79.69 | 14.35 | 64.62 | 67.87 |
| E8-QAT | 45.90 | 74.07 | 74.58 | 78.45 | 71.11 | 83.88 | 11.45 | 67.51 | 71.33 |
E8-QAT is higher on all six accuracy metrics, and its WikiText perplexity is lower. Table 10 reports these endpoints.
Qwen3-4B is an additional dense model in the Qwen family. Both legs use the same Hessian and the same calibration corpus, then 375-step QAT with every setting shared except the codebook. The learning rate was fixed by the rule before any result. E8-QAT is higher on all six accuracy metrics and lower in WikiText perplexity (sign test ). Table 9 reports the six metrics.
Response to Reviewer MzgK
We thank Reviewer MzgK for a careful and well-organized review. We are glad the search-free projection, the escape format and decoding checks, the gains over LC-QAT, and the memory reduction came through. The revisions below are in the manuscript.
W1: “E8-PTQ already outperforms UniSVQ … The comparison with LC-QAT therefore does not isolate the benefit of the proposed training strategy.”
A: We thank the reviewer for their interest in how initialization and training each contribute. We conducted an experiment on the initialization. With the starts matched to within 0.04, E8-QAT leads LC-QAT by 1.18 at the endpoint, at 7.4σ.
In the measured data the E8-PTQ lead is 63.44 against 60.54 on avg4, before any QAT. To compare the QAT gain, we rolled the E8 initialization back to the UniSVQ start and repeated the training. E8-PTQ assigns codewords with Hessian-weighted LDLQ. To roll that start back, we replaced a random fraction of its 8-dimensional groups with the nearest in-core codeword chosen without the Hessian, a deliberately worse assignment, and swept the fraction until the step-0 avg4 matched the UniSVQ start (60.54). Row scalings, column scalings, and the QAT recipe stayed fixed. The only change was which codewords were used.
The starts are within 0.04. E8-QAT then reaches 64.08, against 62.90±0.11: +1.18, 7.4σ, 95% CI [+0.71,+1.65]. That gain is not inherited from the initialization.
We also conducted an experiment on the latent position inside the cell, keeping the quantized initialization fixed. The E8 avg4 mean rises from 65.217 to 65.985.
Section 5.2 is revised to:
W2: “Its impact on GPU kernel efficiency during inference is not quantified or demonstrated.”
A: We are grateful for the reviewer's attention to the kernel cost of the sparse records. We conducted an experiment on gather time for 223 groups, with the escape patch fused into the kernel. The variation in collection times of the experimental results indicates that the speed is close to that of LC-QAT.
Theoretically speaking, at 750 steps the escape rate is ρ=8.556×10−5, and the payload is b=2+20ρ≈2.00171 bits/weight, which is 0.00171 bit/weight above the 2-bit core. Inference is memory-bound, so speed tracks the bits read. Table 4 lists the quantized weights at 2.03 bits once codebooks and channel scales are included. The escape patch is fused into the gather kernel, so the extra traffic is not a second pass. The two measured rates are:
On the codebook-only experiment, three Llama-3.1-8B seeds raise the four-task average by 2.31 (p<10−4) and two Qwen3.6-35B-A3B seeds raise it by 0.76 (p<0.05). Qwen3-4B is higher on all six accuracy metrics, and WikiText perplexity is lower in all three.
Section 5.2 is revised to:
W3: “Figure 3 … mainly reflects the larger batch enabled by reduced memory use. Single-request inference efficiency is not reported.”
A: We appreciate the reviewer's close reading of the throughput comparison. E8-QAT lowers peak memory by 80% and delivers 1.97× the BF16 throughput at the largest feasible batch. The GPU is filled on purpose. Under load, tokens per second are set by the largest batch that fits, and that is the point Figure 3 measures. Decode is memory-bound, so the smaller footprint is the mechanism: peak memory is 80% lower, the batch is 4× larger, and throughput at that batch is 1.97× BF16.
The Figure 3 caption is revised to: