- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents CoBIT, a post-training quantization framework for ultra-low-bit large language model quantization. It aims to optimize the weight codebook, codeword assignment and activation quantization format via a unified layer output reconstruction objective.
For weight quantization, the authors leverage full Hessian information to guide non-uniform codebook updates and optimize codeword assignment with a GPTQ-style sequential error compensation mechanism. For activation quantization, the authors propose PhaseLUT, which dynamically selects the exponent and mantissa configurations of FP4 based on the floating-point mantissa information of the absolute maximum activation value of each token. In addition, the authors implement dedicated CUDA Kernels that fuse activation encoding, codebook lookup and low-precision computation to improve practical inference efficiency.
Experiments are conducted on multiple Qwen3 models and several inference models. The results demonstrate that the proposed method can significantly improve model performance under A4W2 and A4W3 settings, and achieves a certain degree of end-to-end inference acceleration under the A4W4 setting.
Critical strengths
- CoBIT achieves results that are significantly better than the compared baselines under the A4W2 and A4W3 settings across multiple models. Especially in the 2-bit weight clustering scenario, some baselines suffer severe performance degradation, while CoBIT can still maintain good downstream task accuracy.
- The authors compared different combinations such as Full/Diagonal Hessian and Nearest/Sequential Assignment, and further tested the effect of applying sequential compensated assignment to the codebooks of existing methods. These experiments help identify the actual contribution of different components to the final accuracy.
- The paper not only reports quantization accuracy, but also designs the corresponding CUDA Kernel and provides kernel-level and end-to-end latency evaluations. This improves the engineering integrity and practical application value of the method.
Critical weaknesses
W1. The proposed "unified optimization" is closer to staged optimization under a shared objective than genuine joint optimization.
Although the paper derives a unified layer-output reconstruction objective consisting of weight quantization error, activation quantization error, and their interaction, the actual optimization procedure does not jointly optimize these components.
Instead, the interaction term is omitted, and weight clustering and activation composition are optimized separately. The overall pipeline also follows a staged procedure, optimizing weight representations before calibrating PhaseLUT, rather than jointly or alternately optimizing all three variables.
Therefore, I believe the term "unified optimization" may overstate the degree of coupling in the proposed algorithm.
While using a common reconstruction objective to guide different components is reasonable, the authors should more clearly distinguish unified objective formulation from genuine joint optimization and justify the advantages of the proposed staged procedure.
W2. The main accuracy improvements at extremely low bit widths appear to be largely driven by the established GPTQ-style sequential error compensation mechanism.
The sequential assignment procedure closely follows the second-order error compensation principle introduced by GPTQ. Its primary modification is applying this mechanism to learned non-uniform codebooks rather than fixed quantization grids.
More importantly, Tables 21 and 22 in the appendix reveal that applying sequential assignment to existing SqueezeLLM codebooks achieves performance comparable to the complete CoBIT weight optimization.
For example, on Qwen3-8B under the 2-bit weight-only setting, SqueezeLLM codebooks combined with sequential assignment achieve an average accuracy of 0.7291, compared with 0.7246 for CoBIT with full-Hessian codebooks and sequential assignment.
This suggests that, at least in some extreme low-bit settings, most of the performance gain may originate from sequential error compensation rather than the proposed codebook construction or alternating optimization.
Although this combination is empirically effective, its independent algorithmic contribution over GPTQ, SqueezeLLM, and GuidedQuant/LNQ remains insufficiently established.
I would like the authors to clarify what constitutes the essential methodological novelty beyond extending established error compensation to learned clustering codebooks.
W3. The calibration-to-holdout stability analysis does not sufficiently justify omitting the weight–activation interaction term.
The paper decomposes the layer-output reconstruction error into three components but omits the interaction term during the subsequent weight and activation optimization stages.
In Appendix A.1.3, the authors examine the transferability of statistics associated with this interaction term between calibration and held-out data and observe relatively low similarity.
However, this observation mainly indicates that directly optimizing the interaction term using calibration statistics may suffer from generalization issues. It does not establish that the interaction term itself is negligible, nor does it demonstrate that omitting it causes no significant performance loss.
I therefore suggest providing additional evidence, including:
The relative magnitudes of the three error components across models, layers, and quantization bit widths.
A comparison of layer reconstruction errors and downstream performance with and without explicitly accounting for the interaction term.
At least on smaller models, a comparison between the proposed staged procedure and an alternating or joint optimization strategy incorporating the interaction term.
These experiments would strengthen the justification for both the proposed objective decomposition and the resulting optimization strategy.
W4. The limited quantization scope weakens the generality of the extreme low-bit inference claims.
As stated in the paper, only the Q/K/V and Gate/Up projections are quantized, while other modules, including O-proj and Down-proj, remain in BF16.
Consequently, the reported A4W2 and A4W3 settings describe the precision of selected projections rather than the effective precision of the entire model. Since a non-negligible fraction of model parameters remains at higher precision, these results should not be directly interpreted as full-model W2A4 or W3A4 quantization.
I suggest reporting the effective model-wide average bit width, actual compression ratio, and memory footprint, together with experiments quantizing all major linear projections.
Furthermore, while the most impressive accuracy improvements are obtained under W2 and W3 settings, the inference acceleration experiments primarily focus on A4W4.
Different codebook sizes and index widths may lead to different memory-access patterns and computational efficiency. Therefore, the A4W4 speedups do not necessarily generalize to W2 or W3.
Kernel-level and end-to-end inference measurements under W2/W3 would be important for validating the practical benefits of the paper's main extreme low-bit settings.
W5. The performance advantage over highly optimized inference kernels is relatively limited, and more detailed profiling is needed.
CoBIT demonstrates substantial speedups over FP16 cuBLAS. However, its advantage becomes considerably smaller when compared with highly optimized low-bit inference kernels such as Marlin, and the proposed implementation is slower in some larger-batch configurations.
This suggests that a substantial portion of the reported acceleration may come from reduced weight-memory traffic, while the independent runtime contributions of PhaseLUT, dynamic activation composition, and fused computation remain insufficiently isolated.
I recommend a more detailed latency breakdown covering:
-
Amax reduction and PhaseLUT lookup.
-
FP4 activation encoding and format conversion.
-
Weight-index decoding and codebook lookup.
-
MMA/FMA computation.
-
Memory access and output storage.
In addition, evaluating a broader range of batch sizes, prompt lengths, and generation lengths would provide a clearer understanding of the workloads under which the proposed kernel offers meaningful advantages.
Questions for authors
1.What is the fundamental distinction between the proposed unified optimization and a staged optimization framework guided by a shared reconstruction objective? Why is the term "unified optimization" appropriate when the optimization stages are largely decoupled?
2.Given that existing SqueezeLLM codebooks combined with GPTQ-style sequential assignment achieve performance comparable to the full CoBIT method under W2, what are the unique and indispensable contributions of full-Hessian codebook refinement and alternating optimization?
3.Why does the observed calibration-to-holdout instability of interaction-related statistics sufficiently justify omitting the interaction term? Could the authors compare the reconstruction error and downstream performance with and without explicitly optimizing this term?
4.Could the authors report the effective model-wide bit width, compression ratio, and memory footprint, together with results quantizing additional projections such as O-proj and Down-proj? Can they also provide actual inference speedups under W2/W3 settings?
5.Could the authors provide a more detailed runtime breakdown to isolate the contributions of PhaseLUT, activation encoding, codebook-based computation, and operator fusion, and compare against optimized kernels such as Marlin under a broader range of inference workloads?
7 comments
Response to Reviewer gTZR, Part 2
Response to Weakness 2:
We acknowledge that frozen SqueezeLLM codebooks with our assignment procedure can achieve comparable performance in some settings. We further find that frozen SqueezeLLM codebooks are more sensitive to calibration-set size. CoBIT performs better on average and degrades less as calibration size decreases. Table R1 compares the two methods on Qwen3-4B. To isolate the codebook comparison, both use fixed E2M1 FP4 activations without PhaseLUT. From 512 to 32 samples, Wiki2 perplexity increases by 1.09 for CoBIT versus 1.41 for SqueezeLLM, and C4 perplexity increases by 1.73 versus 2.55. FKL increases by approximately 17.0% versus 18.2%. These results exclude PhaseLUT, which further improves perplexity and KL in our main experiments.
Table R1. Calibration-size comparison on Qwen3-4B (A4W2).
| Calibration size | Selection seed | Wiki2 CoBIT | Wiki2 SQ | C4 CoBIT | C4 SQ | FKL CoBIT | FKL SQ |
|---|---|---|---|---|---|---|---|
| 512 | 42 | 20.55 | 21.14 | 27.95 | 28.60 | 0.681 | 0.676 |
| 512 | 0 | 20.71 | 22.75 | 28.26 | 29.85 | 0.673 | 0.701 |
| 512 | 19 | 20.44 | 21.96 | 28.42 | 29.88 | 0.672 | 0.695 |
| 512 | Mean | 20.57 | 21.95 | 28.21 | 29.44 | 0.675 | 0.691 |
| 128 | 42 | 21.18 | 21.30 | 29.72 | 29.66 | 0.688 | 0.693 |
| 128 | 0 | 21.33 | 22.06 | 29.58 | 29.51 | 0.708 | 0.708 |
| 128 | 19 | 20.31 | 20.79 | 28.48 | 28.68 | 0.700 | 0.700 |
| 128 | Mean | 20.94 | 21.38 | 29.26 | 29.28 | 0.699 | 0.700 |
| 32 | 42 | 22.09 | 24.35 | 30.21 | 32.65 | 0.799 | 0.824 |
| 32 | 0 | 21.34 | 23.16 | 29.95 | 32.20 | 0.765 | 0.805 |
| 32 | 19 | 21.56 | 22.58 | 29.66 | 31.12 | 0.805 | 0.821 |
| 32 | Mean | 21.66 | 23.36 | 29.94 | 31.99 | 0.790 | 0.817 |
Since our method targets activation–weight quantization, we also compare A4W3/A4W2 performance on Qwen3-8B under the same rotation and scaling. Table R2 shows that CoBIT improves FKL, perplexity, and average accuracy over frozen SqueezeLLM codebooks with our assignment procedure at A4W3, with a 1.1-percentage-point accuracy gain. At A4W2, the average accuracy gain is small, while FKL and perplexity also improve. Figure 3 in Section 4.2 further shows that the off-diagonal Hessian terms matter more as the codebook gets smaller. These results support the benefit of full-Hessian codebook updates.
Table R2. A4W2/A4W3 results with shared R/S (Qwen3-8B).
| Setting | Method | FKL ↓ | Wiki2 ↓ | C4 ↓ | ARC ↑ | PIQA ↑ | Hella ↑ | MMLU ↑ | Avg ↑ |
|---|---|---|---|---|---|---|---|---|---|
| A4W2 | CoBIT | 0.545 | 13.90 | 20.75 | 0.819 | 0.563 | 0.595 | 0.595 | 0.643 |
| A4W2 | LNQ | 0.575 | 13.93 | 20.19 | 0.794 | 0.599 | 0.588 | 0.544 | 0.631 |
| A4W2 | SqueezeLLM codebook + CoBIT assignment | 0.575 | 14.37 | 21.24 | 0.828 | 0.554 | 0.593 | 0.593 | 0.642 |
| A4W3 | CoBIT | 0.266 | 11.58 | 17.31 | 0.884 | 0.816 | 0.736 | 0.729 | 0.791 |
| A4W3 | LNQ | 0.275 | 11.65 | 17.39 | 0.881 | 0.794 | 0.750 | 0.715 | 0.785 |
| A4W3 | SqueezeLLM codebook + CoBIT assignment | 0.271 | 11.70 | 17.49 | 0.884 | 0.783 | 0.737 | 0.717 | 0.780 |
LNQ provides prior work on closed-form full-Hessian codebook updates. In CoBIT, weight optimization accounts for quantization errors accumulated from preceding layers, while a shared output-reconstruction objective guides codebook refinement, assignment, and activation-format selection. Tables 21–25 further examine the roles of codebook construction, assignment, and Hessian information. Beyond the weight-only comparison in Table 3, we evaluate LNQ under the same rotation and scaling at A4W2/A4W3 in Table R2. CoBIT achieves better FKL and average accuracy in both settings, with lower perplexity except for C4 at A4W2.
Response to Reviewer gTZR, Part 3
Response to Weakness 3:
We first report the relative contributions of the three terms on Qwen3-4B and Qwen3-8B at W2/W3/W4. As shown in Table R3, the interaction term has a magnitude below 5% throughout the reported ranges.
Table R3. Relative contributions of the error terms.
| Model | Bits | Weight term | Activation term | Interaction term |
|---|---|---|---|---|
| Qwen3-4B | W2 | 77–80% | 23–27% | −4.3% to −3.4% |
| Qwen3-4B | W3 | 46–50% | 52–57% | −2.9% to −2.4% |
| Qwen3-4B | W4 | 18–21% | 80–83% | −1.2% to −1.0% |
| Qwen3-8B | W2 | 76–79% | 24–29% | −4.7% to −3.7% |
| Qwen3-8B | W3 | 44–49% | 54–59% | −2.9% to −2.4% |
| Qwen3-8B | W4 | 17–20% | 81–84% | −1.2% to −1.0% |
We further compare optimization with and without the interaction term on Qwen3-8B at A4W2 under two settings. The accumulated setting used in the paper accounts for quantization errors from preceding layers. The local setting considers only activation quantization in the current layer. In the accumulated setting, including the interaction term slightly reduces held-out reconstruction error but worsens FKL and perplexity. Downstream results are mixed. In the local setting, including the interaction term worsens FKL, perplexity, and held-out error. The local results show that explicitly optimizing the interaction term can hurt performance. We revised the paperto reflect these results.
Table R4. Including the interaction term (Qwen3-8B, A4W2).
| Setting | FKL ↓ | Wiki2 ↓ | C4 ↓ | ARC ↑ | PIQA ↑ | Hella ↑ | MMLU ↑ | Avg ↑ | Held-out error, Wiki2 / C4 ↓ |
|---|---|---|---|---|---|---|---|---|---|
| Accumulated, without interaction | 0.545 | 13.90 | 20.75 | 0.819 | 0.563 | 0.595 | 0.595 | 0.643 | 0.01439 / 0.01193 |
| Accumulated, with interaction | 0.584 | 13.95 | 21.36 | 0.822 | 0.560 | 0.596 | 0.583 | 0.640 | 0.01435 / 0.01161 |
| Local, without interaction | 0.554 | 13.76 | 20.94 | — | — | — | — | — | 0.01477 / 0.01223 |
| Local, with interaction | 0.860 | 18.15 | 25.50 | 0.745 | 0.546 | 0.469 | 0.389 | 0.537 | 0.01639 / 0.01330 |
Response to Reviewer gTZR, Part 1
Thank you for your detailed comments on the optimization framework, methodological contributions, interaction term, quantization scope, and inference performance. We provide a brief summary for quick reading:
Response to Weakness 1:
We agree with the reviewer and revised our wording to avoid confusion. By “unified,” we mean a shared objective, rather than a joint solver. Each stage of CoBIT optimizes part of the same layer-output reconstruction objective in Eqs. 1–2. We use a staged procedure for practical reasons. The joint problem mixes discrete choices, such as integer assignments and activation formats, with orthogonal rotations, which makes it hard to solve directly. Splitting the problem into stages gives subproblems that can each be solved efficiently. The main term the split ignores is the interaction between weight and activation quantization errors, which accounts for less than 5% of the objective in our settings. Rotation and scaling are continuous and already learned with AdamW, but the codebook and assignment steps are discrete. Optimizing them jointly with rotation and scaling has no direct closed-form solution, so it would rely heavily on gradient-based fine-tuning, which is less efficient. Such methods can still work well. In Table 1 of the paper, CodeQuant, which uses gradient-based optimization, is mostly the second-best method, and CoBIT consistently outperforms it.