acceptodds
Under review as a conference paper at ICLR 2027

CoBIT: Unified Optimization of Codebooks, Assignments, and Activation Composition for Extreme Low-Bit LLM Quantization

10 views

Abstract

Post-training quantization is essential for efficient large model inference, yet preserving accuracy at ultra-low budget below 4 bits remains challenging. Clustering-based quantization provides adaptive non-uniform representations, but existing methods use limited information to construct codebooks and assign weights, while activation quantization is typically handled separately. We present CoBIT, a post-training quantization framework that optimizes weight clustering and activation quantization under a common layer-output reconstruction objective. For weights, CoBIT alternates Hessian-aware codebook construction with error-compensated assignment. For activations, CoBIT selects a per-token exponent–mantissa composition based on layer-output error. We introduce PhaseLUT to replace runtime composition search with a lookup, and a hardware-aware kernel that integrates composition selection, activation quantization, and low-precision computation. Across all four general-purpose models, CoBIT improves average accuracy over the strongest baseline in each setting by 4.8% and 35.9% points with 4-bit activations and 3- and 2-bit weight clustering, respectively. With 4-bit weight indices, it also achieves up to end-to-end decoding speedup over dense FP16.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Quantization, Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gTZROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents CoBIT, a post-training quantization framework for ultra-low-bit large language model quantization. It aims to optimize the weight codebook, codeword assignment and activation quantization format via a unified layer output reconstruction objective.

For weight quantization, the authors leverage full Hessian information to guide non-uniform codebook updates and optimize codeword assignment with a GPTQ-style sequential error compensation mechanism. For activation quantization, the authors propose PhaseLUT, which dynamically selects the exponent and mantissa configurations of FP4 based on the floating-point mantissa information of the absolute maximum activation value of each token. In addition, the authors implement dedicated CUDA Kernels that fuse activation encoding, codebook lookup and low-precision computation to improve practical inference efficiency.

Experiments are conducted on multiple Qwen3 models and several inference models. The results demonstrate that the proposed method can significantly improve model performance under A4W2 and A4W3 settings, and achieves a certain degree of end-to-end inference acceleration under the A4W4 setting.

Critical strengths
  1. CoBIT achieves results that are significantly better than the compared baselines under the A4W2 and A4W3 settings across multiple models. Especially in the 2-bit weight clustering scenario, some baselines suffer severe performance degradation, while CoBIT can still maintain good downstream task accuracy.
  2. The authors compared different combinations such as Full/Diagonal Hessian and Nearest/Sequential Assignment, and further tested the effect of applying sequential compensated assignment to the codebooks of existing methods. These experiments help identify the actual contribution of different components to the final accuracy.
  3. The paper not only reports quantization accuracy, but also designs the corresponding CUDA Kernel and provides kernel-level and end-to-end latency evaluations. This improves the engineering integrity and practical application value of the method.
Critical weaknesses

W1. The proposed "unified optimization" is closer to staged optimization under a shared objective than genuine joint optimization.

Although the paper derives a unified layer-output reconstruction objective consisting of weight quantization error, activation quantization error, and their interaction, the actual optimization procedure does not jointly optimize these components.

Instead, the interaction term is omitted, and weight clustering and activation composition are optimized separately. The overall pipeline also follows a staged procedure, optimizing weight representations before calibrating PhaseLUT, rather than jointly or alternately optimizing all three variables.

Therefore, I believe the term "unified optimization" may overstate the degree of coupling in the proposed algorithm.

While using a common reconstruction objective to guide different components is reasonable, the authors should more clearly distinguish unified objective formulation from genuine joint optimization and justify the advantages of the proposed staged procedure.


W2. The main accuracy improvements at extremely low bit widths appear to be largely driven by the established GPTQ-style sequential error compensation mechanism.

The sequential assignment procedure closely follows the second-order error compensation principle introduced by GPTQ. Its primary modification is applying this mechanism to learned non-uniform codebooks rather than fixed quantization grids.

More importantly, Tables 21 and 22 in the appendix reveal that applying sequential assignment to existing SqueezeLLM codebooks achieves performance comparable to the complete CoBIT weight optimization.

For example, on Qwen3-8B under the 2-bit weight-only setting, SqueezeLLM codebooks combined with sequential assignment achieve an average accuracy of 0.7291, compared with 0.7246 for CoBIT with full-Hessian codebooks and sequential assignment.

This suggests that, at least in some extreme low-bit settings, most of the performance gain may originate from sequential error compensation rather than the proposed codebook construction or alternating optimization.

Although this combination is empirically effective, its independent algorithmic contribution over GPTQ, SqueezeLLM, and GuidedQuant/LNQ remains insufficiently established.

I would like the authors to clarify what constitutes the essential methodological novelty beyond extending established error compensation to learned clustering codebooks.


W3. The calibration-to-holdout stability analysis does not sufficiently justify omitting the weight–activation interaction term.

The paper decomposes the layer-output reconstruction error into three components but omits the interaction term during the subsequent weight and activation optimization stages.

In Appendix A.1.3, the authors examine the transferability of statistics associated with this interaction term between calibration and held-out data and observe relatively low similarity.

However, this observation mainly indicates that directly optimizing the interaction term using calibration statistics may suffer from generalization issues. It does not establish that the interaction term itself is negligible, nor does it demonstrate that omitting it causes no significant performance loss.

I therefore suggest providing additional evidence, including:

The relative magnitudes of the three error components across models, layers, and quantization bit widths.

A comparison of layer reconstruction errors and downstream performance with and without explicitly accounting for the interaction term.

At least on smaller models, a comparison between the proposed staged procedure and an alternating or joint optimization strategy incorporating the interaction term.

These experiments would strengthen the justification for both the proposed objective decomposition and the resulting optimization strategy.


W4. The limited quantization scope weakens the generality of the extreme low-bit inference claims.

As stated in the paper, only the Q/K/V and Gate/Up projections are quantized, while other modules, including O-proj and Down-proj, remain in BF16.

Consequently, the reported A4W2 and A4W3 settings describe the precision of selected projections rather than the effective precision of the entire model. Since a non-negligible fraction of model parameters remains at higher precision, these results should not be directly interpreted as full-model W2A4 or W3A4 quantization.

I suggest reporting the effective model-wide average bit width, actual compression ratio, and memory footprint, together with experiments quantizing all major linear projections.

Furthermore, while the most impressive accuracy improvements are obtained under W2 and W3 settings, the inference acceleration experiments primarily focus on A4W4.

Different codebook sizes and index widths may lead to different memory-access patterns and computational efficiency. Therefore, the A4W4 speedups do not necessarily generalize to W2 or W3.

Kernel-level and end-to-end inference measurements under W2/W3 would be important for validating the practical benefits of the paper's main extreme low-bit settings.


W5. The performance advantage over highly optimized inference kernels is relatively limited, and more detailed profiling is needed.

CoBIT demonstrates substantial speedups over FP16 cuBLAS. However, its advantage becomes considerably smaller when compared with highly optimized low-bit inference kernels such as Marlin, and the proposed implementation is slower in some larger-batch configurations.

This suggests that a substantial portion of the reported acceleration may come from reduced weight-memory traffic, while the independent runtime contributions of PhaseLUT, dynamic activation composition, and fused computation remain insufficiently isolated.

I recommend a more detailed latency breakdown covering:

  • Amax reduction and PhaseLUT lookup.

  • FP4 activation encoding and format conversion.

  • Weight-index decoding and codebook lookup.

  • MMA/FMA computation.

  • Memory access and output storage.

In addition, evaluating a broader range of batch sizes, prompt lengths, and generation lengths would provide a clearer understanding of the workloads under which the proposed kernel offers meaningful advantages.

Questions for authors

1.What is the fundamental distinction between the proposed unified optimization and a staged optimization framework guided by a shared reconstruction objective? Why is the term "unified optimization" appropriate when the optimization stages are largely decoupled?

2.Given that existing SqueezeLLM codebooks combined with GPTQ-style sequential assignment achieve performance comparable to the full CoBIT method under W2, what are the unique and indispensable contributions of full-Hessian codebook refinement and alternating optimization?

3.Why does the observed calibration-to-holdout instability of interaction-related statistics sufficiently justify omitting the interaction term? Could the authors compare the reconstruction error and downstream performance with and without explicitly optimizing this term?

4.Could the authors report the effective model-wide bit width, compression ratio, and memory footprint, together with results quantizing additional projections such as O-proj and Down-proj? Can they also provide actual inference speedups under W2/W3 settings?

5.Could the authors provide a more detailed runtime breakdown to isolate the contributions of PhaseLUT, activation encoding, codebook-based computation, and operator fusion, and compare against optimized kernels such as Marlin under a broader range of inference workloads?

7 comments
AuthorsOpenReview ↗

Response to Reviewer gTZR, Part 1

Thank you for your detailed comments on the optimization framework, methodological contributions, interaction term, quantization scope, and inference performance. We provide a brief summary for quick reading:

  • Unified objective: “Unified” refers to the shared reconstruction objective. CoBIT optimizes its components in stages, and we revised the paper to make this clear.
  • Codebook refinement: Our additional experiments show lower sensitivity to calibration size and improvements in FKL, perplexity, and average accuracy under activation–weight quantization.
  • Interaction term: Its magnitude is below 5% in the reported settings. Explicitly optimizing it produces mixed results, and we revised our explanation accordingly.
  • Quantization scope: We report effective model-wide bit widths, memory usage, results quantizing all seven projections, and W2/W3 inference measurements.
  • Kernel performance: Our optimized kernel outperforms vLLM's Marlin implementation across the tested batch sizes from 1 to 32. We also provide component profiling, a fusion ablation, and additional inference workloads.

Response to Weakness 1:

We agree with the reviewer and revised our wording to avoid confusion. By “unified,” we mean a shared objective, rather than a joint solver. Each stage of CoBIT optimizes part of the same layer-output reconstruction objective in Eqs. 1–2. We use a staged procedure for practical reasons. The joint problem mixes discrete choices, such as integer assignments and activation formats, with orthogonal rotations, which makes it hard to solve directly. Splitting the problem into stages gives subproblems that can each be solved efficiently. The main term the split ignores is the interaction between weight and activation quantization errors, which accounts for less than 5% of the objective in our settings. Rotation and scaling are continuous and already learned with AdamW, but the codebook and assignment steps are discrete. Optimizing them jointly with rotation and scaling has no direct closed-form solution, so it would rely heavily on gradient-based fine-tuning, which is less efficient. Such methods can still work well. In Table 1 of the paper, CodeQuant, which uses gradient-based optimization, is mostly the second-best method, and CoBIT consistently outperforms it.

AuthorsOpenReview ↗

Response to Reviewer gTZR, Part 2

Response to Weakness 2:

We acknowledge that frozen SqueezeLLM codebooks with our assignment procedure can achieve comparable performance in some settings. We further find that frozen SqueezeLLM codebooks are more sensitive to calibration-set size. CoBIT performs better on average and degrades less as calibration size decreases. Table R1 compares the two methods on Qwen3-4B. To isolate the codebook comparison, both use fixed E2M1 FP4 activations without PhaseLUT. From 512 to 32 samples, Wiki2 perplexity increases by 1.09 for CoBIT versus 1.41 for SqueezeLLM, and C4 perplexity increases by 1.73 versus 2.55. FKL increases by approximately 17.0% versus 18.2%. These results exclude PhaseLUT, which further improves perplexity and KL in our main experiments.

Table R1. Calibration-size comparison on Qwen3-4B (A4W2).

Calibration sizeSelection seedWiki2 CoBITWiki2 SQC4 CoBITC4 SQFKL CoBITFKL SQ
5124220.5521.1427.9528.600.6810.676
512020.7122.7528.2629.850.6730.701
5121920.4421.9628.4229.880.6720.695
512Mean20.5721.9528.2129.440.6750.691
1284221.1821.3029.7229.660.6880.693
128021.3322.0629.5829.510.7080.708
1281920.3120.7928.4828.680.7000.700
128Mean20.9421.3829.2629.280.6990.700
324222.0924.3530.2132.650.7990.824
32021.3423.1629.9532.200.7650.805
321921.5622.5829.6631.120.8050.821
32Mean21.6623.3629.9431.990.7900.817

Since our method targets activation–weight quantization, we also compare A4W3/A4W2 performance on Qwen3-8B under the same rotation and scaling. Table R2 shows that CoBIT improves FKL, perplexity, and average accuracy over frozen SqueezeLLM codebooks with our assignment procedure at A4W3, with a 1.1-percentage-point accuracy gain. At A4W2, the average accuracy gain is small, while FKL and perplexity also improve. Figure 3 in Section 4.2 further shows that the off-diagonal Hessian terms matter more as the codebook gets smaller. These results support the benefit of full-Hessian codebook updates.

Table R2. A4W2/A4W3 results with shared R/S (Qwen3-8B).

SettingMethodFKL ↓Wiki2 ↓C4 ↓ARC ↑PIQA ↑Hella ↑MMLU ↑Avg ↑
A4W2CoBIT0.54513.9020.750.8190.5630.5950.5950.643
A4W2LNQ0.57513.9320.190.7940.5990.5880.5440.631
A4W2SqueezeLLM codebook + CoBIT assignment0.57514.3721.240.8280.5540.5930.5930.642
A4W3CoBIT0.26611.5817.310.8840.8160.7360.7290.791
A4W3LNQ0.27511.6517.390.8810.7940.7500.7150.785
A4W3SqueezeLLM codebook + CoBIT assignment0.27111.7017.490.8840.7830.7370.7170.780

LNQ provides prior work on closed-form full-Hessian codebook updates. In CoBIT, weight optimization accounts for quantization errors accumulated from preceding layers, while a shared output-reconstruction objective guides codebook refinement, assignment, and activation-format selection. Tables 21–25 further examine the roles of codebook construction, assignment, and Hessian information. Beyond the weight-only comparison in Table 3, we evaluate LNQ under the same rotation and scaling at A4W2/A4W3 in Table R2. CoBIT achieves better FKL and average accuracy in both settings, with lower perplexity except for C4 at A4W2.

AuthorsOpenReview ↗

Response to Reviewer gTZR, Part 3

Response to Weakness 3:

We first report the relative contributions of the three terms on Qwen3-4B and Qwen3-8B at W2/W3/W4. As shown in Table R3, the interaction term has a magnitude below 5% throughout the reported ranges.

Table R3. Relative contributions of the error terms.

ModelBitsWeight termActivation termInteraction term
Qwen3-4BW277–80%23–27%−4.3% to −3.4%
Qwen3-4BW346–50%52–57%−2.9% to −2.4%
Qwen3-4BW418–21%80–83%−1.2% to −1.0%
Qwen3-8BW276–79%24–29%−4.7% to −3.7%
Qwen3-8BW344–49%54–59%−2.9% to −2.4%
Qwen3-8BW417–20%81–84%−1.2% to −1.0%

We further compare optimization with and without the interaction term on Qwen3-8B at A4W2 under two settings. The accumulated setting used in the paper accounts for quantization errors from preceding layers. The local setting considers only activation quantization in the current layer. In the accumulated setting, including the interaction term slightly reduces held-out reconstruction error but worsens FKL and perplexity. Downstream results are mixed. In the local setting, including the interaction term worsens FKL, perplexity, and held-out error. The local results show that explicitly optimizing the interaction term can hurt performance. We revised the paperto reflect these results.

Table R4. Including the interaction term (Qwen3-8B, A4W2).

SettingFKL ↓Wiki2 ↓C4 ↓ARC ↑PIQA ↑Hella ↑MMLU ↑Avg ↑Held-out error, Wiki2 / C4 ↓
Accumulated, without interaction0.54513.9020.750.8190.5630.5950.5950.6430.01439 / 0.01193
Accumulated, with interaction0.58413.9521.360.8220.5600.5960.5830.6400.01435 / 0.01161
Local, without interaction0.55413.7620.94—————0.01477 / 0.01223
Local, with interaction0.86018.1525.500.7450.5460.4690.3890.5370.01639 / 0.01330
Reviewer_BbNCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to design a joint optimization method for codebook quantization of model weights with FP4 activation quantization. This paper proposes a method that uses a layer-output error decomposition to update the codebook based on Hessians and error-compensated weight assignment. Then, the method uses a per-token exponent–mantissa lookup table for activation quantization. Additionally, this paper writes a fused GPU kernel that implements the activation lookup, encoding, and clustered computation. On four Qwen3 models, A4W3 and A4W2 improve mean downstream accuracy over the strongest tested quantized baseline in each setting by 4.8 and 35.9 points, averaged across models. The A4W4 implementation achieves up to 1.53 times decoding speedup (at batch size 1) over dense FP16 models.

Critical strengths
  • The idea of joint optimization of both weights and activation quantization is sound. The low-bit accuracy gains are relatively substantial in the tested settings. Tables 1 and 17 cover four Qwen3 model sizes at A4W2 and A4W3. Table 3 separately shows that the 2-bit weight quantizer remains competitive with stronger weight-only methods.
  • System-level performance is evaluated. The system's results measure the complete A4W4 inference path. Tables 7 and 31 report batch-1 decoding speedups on two GPUs with activation encoding included.
Critical weaknesses
  • I wonder how directly applying joint optimization on top of existing weight-only quantization baselines performs and compares to the proposed method. It would be better to include such a comparison with joint optimization variants of weight-only baselines.
  • I wonder how the joint optimization (in the proposed method) reduces outliers compared to previous methods (or directly combines a previous activation quantization).
  • The inference experiments do not cover the W2 and W3 settings that yield the most accuracy improvement. I wonder how this applies to W2/W3 settings.
Questions for authors

The results show that frozen SqueezeLLM codebooks plus sequential assignment nearly match CoBIT at K=4. What benefit does alternating the full-Hessian codebook update add under this setting?

6 comments
AuthorsOpenReview ↗

Response to Reviewer BbNC, Part 1

Thank you for your questions about baseline comparisons, outlier reduction, low-bit inference, and alternating optimization. We provide a brief summary for quick reading:

  • Baseline comparisons: We add A4W2/A4W3 comparisons with LNQ and frozen SqueezeLLM codebooks under the same rotation and scaling.
  • Outlier reduction: Our transformations substantially reduce activation outliers. Compared with QuaRot + GPTQ, CoBIT achieves much lower weight and output errors with only slightly higher activation error.
  • Low-bit inference: We add W2/W3 timings and observe lower decoding latency as the bit width decreases, especially on the RTX 5060 Ti.
  • Alternating optimization: The A4W2/A4W3 comparisons show gains in FKL, perplexity, and average accuracy. Further ablations support calibration robustness and alternating refinement with early stopping.

Response to Weakness 1:

Our joint optimization is built around our own weight quantizer: the shared objective drives both the codebook update and the sequential assignment, so applying it to a baseline would replace the baseline’s codebook and assignment procedure, and the result would essentially become CoBIT rather than an enhanced version of that baseline. As a result, the full optimization cannot be added orthogonally on top of other methods. Rotation and scaling, however, are general transformations. We therefore apply the same rotation and scaling to a strong weight-only baseline and compare A4W2/A4W3 performance. We also include frozen SqueezeLLM codebooks with our assignment procedure. We use LNQ as the weight-only baseline. We also attempted A4 quantization with ParoQuant, but encountered divergence in some layers using its released code, leading to model collapse in our experiments.

Table R1. Baseline comparisons on Qwen3-8B with shared R/S.

SettingMethodFKL ↓Wiki2 ↓C4 ↓ARC ↑PIQA ↑Hella ↑MMLU ↑Avg ↑
A4W2CoBIT0.54513.9020.750.8190.5630.5950.5950.643
A4W2LNQ0.57513.9320.190.7940.5990.5880.5440.631
A4W2SqueezeLLM codebook + CoBIT assignment0.57514.3721.240.8280.5540.5930.5930.642
A4W3CoBIT0.26611.5817.310.8840.8160.7360.7290.791
A4W3LNQ0.27511.6517.390.8810.7940.7500.7150.785
A4W3SqueezeLLM codebook + CoBIT assignment0.27111.7017.490.8840.7830.7370.7170.780

Response to Weakness 2:

We interpret this question as referring to our learned rotation and scaling. We compare these transformations with no preprocessing, SmoothQuant, and QuaRot. Looking first at activations alone, Table R2 shows that CoBIT brings average kurtosis close to 2.95, near the Gaussian value of 3. This is consistent with the approximately Gaussian activation distributions that motivate PhaseLUT. For outliers, we compare the channel max-to-median ratios after transformation. CoBIT is only slightly worse than QuaRot's Hadamard transformation, and their relative FP4 (E2M1) activation quantization errors are nearly identical. This is because our objective accounts for quantized weights and minimizes output error, rather than activation quantization error alone.

Table R2. Activation statistics and FP4 error.

TransformKurtosis (attn)Kurtosis (FFN)Channel max/median (attn)Channel max/median (FFN)FP4 rel. error (attn)FP4 rel. error (FFN)
None26333822.124.20.4830.384
SmoothQuant56.772.56.806.540.3100.274
QuaRot2.812.811.531.530.1130.113
CoBIT R/S2.952.951.631.620.1150.115

Our goal is to preserve the overall output, rather than minimize activation outliers alone. To examine this distinction, we further compare activation, weight, and output errors against QuaRot + GPTQ. As shown in Table R3, CoBIT's activation error is only 0.0005–0.0006 higher, while its weight error and total output error are substantially lower.

Table R3. Reconstruction error on Qwen3-8B (A4W2).

MethodTotal (attn)Total (FFN)Weight term (attn)Activation term (attn)Weight term (FFN)Activation term (FFN)
QuaRot + GPTQ0.13520.22350.12640.00940.20960.0152
CoBIT0.03890.06190.03060.00990.04930.0158
AuthorsOpenReview ↗

Response to Reviewer BbNC, Part 2

Response to Weakness 3:

Our reported timings use FP4-quantized activations converted back to FP16 for computation, matching the configuration used for our accuracy results. Native FP4 matrix multiplication would also require FP4 weights. In codebook quantization, the assignments use low-bit indices while the centroids remain in FP16. Using native FP4 computation would therefore require further quantizing the centroids. We do not apply this additional quantization because the resulting timings would no longer correspond to the accuracy reported in Tables 1 and 2.

The W2/W3 timings below capture the benefits of lower-bit memory access while retaining FP16 computation. We also further optimized our kernel via GPU configuration tuning, storing codebooks in registers, and overlapping asynchronous weight loading with computation during the rebuttal period. Table R4 reports timings with the updated kernel, and Table R5 compares it with the original kernel. As Table R4 shows, reducing the weight bit width lowers decoding latency on both the A6000 and RTX 5060 Ti. The reduction is larger on the RTX 5060 Ti, which has lower memory bandwidth.

Table R4. Qwen3-8B decoding latency (ms/token).

Weight bitsA6000RTX 5060 Ti
W416.5022.89
W315.9321.68
W215.3920.35

Note: All settings use A4 activations.

Table R5. Kernel optimization at A4W4 (ms/token).

ModelBatchA6000 originalA6000 optimizedRTX 5060 Ti originalRTX 5060 Ti optimized
Qwen3-8B117.1816.4823.2822.81
Qwen3-8B218.2717.1725.1224.21
Qwen3-8B420.4718.5427.9726.63
Qwen3-8B825.2221.2035.2631.65
Qwen3-8B1636.3426.3748.1240.88
Qwen3-8B3248.8537.1668.9360.13
Qwen3-4B111.2210.7814.6414.06
Qwen3-4B212.0311.3516.0315.66
Qwen3-4B413.7812.6918.6017.81
Qwen3-4B817.6215.3424.6022.64
Qwen3-4B1626.3820.3736.2731.84
Qwen3-4B3236.6630.6456.1251.17
AuthorsOpenReview ↗

Response to Reviewer BbNC, Part 3

Response to Question 1:

CoBIT is slightly worse by 0.004 in the cited accuracy result, but this does not indicate an overall disadvantage. We find that frozen SqueezeLLM codebooks with our assignment method are more sensitive to calibration-set size. Table R6 compares the two methods under this scenario on Qwen3-4B. To isolate the codebook comparison, both use fixed E2M1 FP4 activations without PhaseLUT. CoBIT performs better on average and degrades less as calibration size decreases. From 512 to 32 samples, Wiki2 perplexity increases by 1.09 for CoBIT versus 1.41 for SqueezeLLM, and C4 perplexity increases by 1.73 versus 2.55. FKL increases by approximately 17.0% versus 18.2%. These results exclude PhaseLUT, which further improves perplexity and KL in our main experiments.

Table R6. Effect of calibration size (Qwen3-4B, A4W2).

Calibration sizeSelection seedWiki2 CoBITWiki2 SQC4 CoBITC4 SQFKL CoBITFKL SQ
5124220.5521.1427.9528.600.6810.676
512020.7122.7528.2629.850.6730.701
5121920.4421.9628.4229.880.6720.695
512Mean20.5721.9528.2129.440.6750.691
1284221.1821.3029.7229.660.6880.693
128021.3322.0629.5829.510.7080.708
1281920.3120.7928.4828.680.7000.700
128Mean20.9421.3829.2629.280.6990.700
324222.0924.3530.2132.650.7990.824
32021.3423.1629.9532.200.7650.805
321921.5622.5829.6631.120.8050.821
32Mean21.6623.3629.9431.990.7900.817

As shown in Table R1 (Response to Weakness 1), CoBIT also improves FKL, perplexity, and average accuracy over frozen SqueezeLLM codebooks with our assignment procedure at A4W3 and A4W2, further supporting the benefit of codebook updates.

We further examine alternating codebook and assignment optimization on Qwen3-8B at A4W2, comparing one fixed round, five fixed rounds, and the default early-stopping procedure used in the paper (as illustrated in paper’s Algorithm 3). Table R7 shows that one round already achieves reasonable performance, while running all five rounds can worsen some metrics. We suspect this may reflect overfitting to the small calibration set. Our early-stopping procedure achieves the best overall results, supporting both the benefit of alternating updates and the robustness of early stopping.

Table R7. Alternating optimization on Qwen3-8B (A4W2).

MethodFKL ↓Wiki2 ↓C4 ↓ARC ↑PIQA ↑Hella ↑MMLU ↑Avg ↑
CoBIT, 1 round0.56314.1221.190.8160.5660.5850.5830.637
CoBIT, early stopping0.54513.9020.750.8190.5630.5950.5950.643
CoBIT, 5 rounds0.57114.3720.980.8050.5840.5610.5620.628
Reviewer_ooDvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoBIT is a PTQ framework for LLMs with clustered weights and FP4 activations. For weights, it alternates betwen a full-Hessian closed-form codebook update and a GPTQ-style error-compensated assignment. For activations, it selects a per-token FP4 exponent-mantissa composition from a mantissa-indexed lookup table (PhaseLUT), and it also fuses activation encoding into a clustered linear kernel. On Qwen3 4B-32B, it reports large accuracy gains at 2-3 bits and up to 1.53x decode speedup over FP16 at 4 bits.

Critical strengths

The authors broadly evaluate across four model sizes and reasoning models, with perplexity, FKL, and downstream metrics. The paper performs informative diagnostics, including particularly the codebook transplant study (Tables 21-22) and the collapsed-checkpoint analysis (A.6.2). It also comes with a working fused kernel with end-to-end latency measurements rather than simulated estimates.

Critical weaknesses
  1. The incremental contribution of each proposed component is unclear The objective is framed as unified, but the stages are optimized sequentially and the interaction term is dropped. For weight, a frozen SqueezeLLM codebook with one sequential-assignment pass matches full CoBIT at 2 bits (0.729/0.821 vs. 0.725/0.821 on Qwen3-8B/14B, Tables 21-22). This suggests that much of the improvement comes from GPTQ-style assignment rather than iterative codebook refinement. Moreover, closed-form full-Hessian codebook updates were already used in LNQ. For activation, PhaseLUT improves only 0.1-0.2 over fixed E2M1 (Table 5). In Table 27, per-token E2M1 without PhaseLUT scores higher on 14B (0.850 vs. 0.848).

  2. The extreme-low-bit and efficiency claims lack evidence o_proj and down_proj layers stay in BF16, which is a substantial amount of linear parameters in Qwen3-8B. Thus, the reported 2-bit weight results do not represent the effective precision of the full model. All latency measurements also use 4-bit indices, so it unclear whether the 2-/3-bit configurations provide storage or inference benefits. The kernel also decodes FP4 activations back to FP16, so activation quantization lowers accuracy (0.823 to 0.791 on Qwen3-8B, Table 5) without a demonstrated efficiency gain. Against weight-only Marlin, CoBIT is only 1.01-1.07x faster at batch 1 and slower at batch 8 (25.80 vs. 22.60 ms/token, Qwen3-8B, A6000).

Overall, I lean toward rejection. The experiments do not establish how much of the gains can be attributed to the proposed components beyond existing techniques.

Questions for authors
  1. Under A4W2/A4W3, how does a frozen SqueezeLLM codebook plus one sequential pass compare with CoBIT? How does LNQ compare when given the same R and S?
  2. Would calibrating PhaseLUT against the full output error, with quantized weights fixed, change the result?
  3. Does the kernel support 2-/3-bit index packing? If not, what do the 2-/3-bit settings save over 4 bits?
  4. How do the Table 1 margins change under log-likelihood scoring, given that several baselines score below chance?
10 comments
AuthorsOpenReview ↗

Response to Reviewer ooDv, Part 1

Thank you for your detailed feedback on the contributions of individual components, quantization coverage, inference efficiency, and evaluation protocol. We summarize our main responses below.

  • Component contributions: Controlled comparisons show improvements in quantization quality and calibration robustness. Additional ablations support PhaseLUT and our choice to omit the interaction term.
  • Coverage and efficiency: We add all-projection quantization results, W2/W3 memory and latency measurements, and an optimized kernel that outperforms Marlin across the tested configurations.
  • Calibration objective: Full-output calibration changes only 2 of 1,152 PhaseLUT entries, with comparable overall performance.
  • Evaluation protocol: CoBIT retains its advantage under log-likelihood scoring, although the A4W2 margins decrease.

Response to Weakness 1 CoBIT uses a shared output-reconstruction objective, but optimizes its components one after another rather than jointly. Each stage reduces one part of the same layer-output error. This lets us update the codebooks in closed form and calibrate weight assignment and activation quantization efficiently. SpinQuant and QuaRot also fix the transformations before weight quantization. We will revise the manuscript to make clear that the objective is shared but the optimization is sequential.

For SqueezeLLM, CoBIT is slightly worse in accuracy in the cited W2 comparison, but this does not indicate an overall disadvantage. We find that the SqueezeLLM-based variant (SqueezeLLM codebook with our assignment method) is more sensitive to calibration-set size. Table R1 compares the methods under the calibration size scenario on Qwen3-4B at A4W2. To isolate weight optimization, both use fixed E2M1 activations without PhaseLUT. CoBIT achieves better mean results and lower sensitivity to calibration size. Reducing the calibration size from 512 to 32 increases WikiText2 perplexity by 1.09 for CoBIT versus 1.41 for the SqueezeLLM-based variant, and C4 perplexity by 1.73 versus 2.55. FKL increases by approximately 17.0% versus 18.2%. These results do not include PhaseLUT, which further improves perplexity and KL divergence in our paper’s experiments.

Table R1. Calibration sensitivity on Qwen3-4B (A4W2).

Calibration sizeSelection seedWikiText2: CoBITWikiText2: SQC4: CoBITC4: SQFKL: CoBITFKL: SQ
5124220.5521.1427.9528.600.6810.676
512020.7122.7528.2629.850.6730.701
5121920.4421.9628.4229.880.6720.695
512Mean20.5721.9528.2129.440.6750.691
1284221.1821.3029.7229.660.6880.693
128021.3322.0629.5829.510.7080.708
1281920.3120.7928.4828.680.7000.700
128Mean20.9421.3829.2629.280.6990.700
324222.0924.3530.2132.650.7990.824
32021.3423.1629.9532.200.7650.805
321921.5622.5829.6631.120.8050.821
32Mean21.6623.3629.9431.990.7900.817

Since CoBIT targets weight–activation quantization, we additionally compare A4W3 and A4W2 on Qwen3-8B using the same rotation and scaling (Table R2). At A4W3, CoBIT achieves lower FKL and perplexity and improves average accuracy by 1.1 percentage points. At A4W2, the average accuracy gain is small, while FKL and perplexity also improve. Together, these results show benefits beyond combining SqueezeLLM codebooks with our assignment method.

AuthorsOpenReview ↗

Response to Reviewer ooDv, Part 2

Response to Weakness 1 (continue) We acknowledge that LNQ already studies closed-form full-Hessian codebook updates. Our weight optimization accounts for accumulated quantization errors from preceding layers, and CoBIT uses a shared output-reconstruction objective to guide codebook refinement, assignment, and activation-format selection. We separately contribute analyses of codebook–assignment combinations and Hessian choices in Tables 21–25. Table 3 also shows our advantage in the weight-only comparison. We further provide direct A4W3/A4W2 comparisons in Table R2, where CoBIT achieves higher average accuracy and lower FKL and WikiText2 perplexity in both settings.

Table R2. Qwen3-8B comparisons with shared R/S.

SettingMethodFKL ↓WikiText2 ↓C4 ↓ARC ↑PIQA ↑HellaSwag ↑MMLU ↑Avg. ↑
A4W2CoBIT0.54513.9020.750.8190.5630.5950.5950.643
A4W2LNQ0.57513.9320.190.7940.5990.5880.5440.631
A4W2SqueezeLLM codebook + CoBIT assignment0.57514.3721.240.8280.5540.5930.5930.642
A4W3CoBIT0.26611.5817.310.8840.8160.7360.7290.791
A4W3LNQ0.27511.6517.390.8810.7940.7500.7150.785
A4W3SqueezeLLM codebook + CoBIT assignment0.27111.7017.490.8840.7830.7370.7170.780

We also compare the number of alternating updates on Qwen3-8B at A4W2. Compared with one round, our early-stopping procedure improves average accuracy from 63.7% to 64.3% and reduces FKL from 0.563 to 0.545. Running five fixed rounds instead gives 62.8% accuracy. These results support the benefit of alternating refinement with early stopping.

AuthorsOpenReview ↗

Response to Reviewer ooDv, Part 3

Response to Weakness 1 (continue) For activation quantization, Section 4.3 shows that PhaseLUT mainly improves KL divergence and perplexity. We see two possible reasons why these improvements do not consistently translate into accuracy gains. First, FP4 offers only three candidate compositions, limiting the search space. Second, perplexity and KL divergence measure how closely the quantized model follows the original model, whereas most of the evaluated QA tasks require only a few output tokens without reasoning.

Based on these observations, we test Qwen3-4B and Qwen3-8B at A8W3 on GPQA, which requires longer reasoning outputs. At A8, PhaseLUT mostly selects E2M5, followed by E4M3 and a small proportion of E3M4. We therefore compare against fixed E2M5 and E4M3. PhaseLUT achieves higher mean accuracy on both models (Table R3). It also provides the flexibility to select compositions automatically, without evaluating every fixed composition separately to find a suitable one. Both this automatic selection and the observed improvements support its usefulness.

Table R3. GPQA accuracy (%) at A8W3.

ModelActivation formatSeed 42Seed 0Seed 19Mean
Qwen3-4BFixed E4M347.9841.9237.8842.59
Qwen3-4BFixed E2M546.4641.9242.4243.60
Qwen3-4BPhaseLUT46.4645.4544.9545.62
Qwen3-8BFixed E4M346.9748.4846.9747.47
Qwen3-8BFixed E2M549.4947.4748.4848.48
Qwen3-8BPhaseLUT49.4948.9947.9848.82

Table R4 reports the relative contributions of the three error terms on Qwen3-4B and Qwen3-8B at W2/W3/W4. The interaction term’s magnitude is below 5% in these measurements.

Table R4. Contributions of the three error terms.

ModelWeight bitsWeight termActivation termInteraction term
Qwen3-4BW277–80%23–27%−4.3% to −3.4%
Qwen3-4BW346–50%52–57%−2.9% to −2.4%
Qwen3-4BW418–21%80–83%−1.2% to −1.0%
Qwen3-8BW276–79%24–29%−4.7% to −3.7%
Qwen3-8BW344–49%54–59%−2.9% to −2.4%
Qwen3-8BW417–20%81–84%−1.2% to −1.0%

We further compare optimization with and without the interaction term on Qwen3-8B at A4W2 in two settings. The accumulated setting follows our paper and accounts for upstream quantization errors. The local setting considers only activation quantization within the current layer.

In the accumulated setting, including the interaction term slightly reduces held-out reconstruction error but worsens FKL and both perplexity scores. Downstream effects are mixed: ARC improves slightly, while MMLU decreases from 59.5% to 58.3%. In the local setting, including the term worsens all available paired metrics: FKL, both perplexity scores, and held-out error. Downstream results for local optimization without the term are unavailable, so we do not draw a paired downstream conclusion. These results show that including the interaction term can hurt performance in some settings. In our paper’s accumulated setting, however, held-out error improves while FKL and perplexity worsen, and downstream effects are mixed. We will revise the description to reflect these results without attributing them to overfitting.

Table R5. Interaction ablation on Qwen3-8B (A4W2).

SettingInteraction termFKL ↓WikiText2 PPL ↓C4 PPL ↓ARC ↑PIQA ↑HellaSwag ↑MMLU ↑Avg. ↑Held-out error: WikiText2 ↓Held-out error: C4 ↓
AccumulatedWithout0.54513.9020.750.8190.5630.5950.5950.6430.014390.01193
AccumulatedWith0.58413.9521.360.8220.5600.5960.5830.6400.014350.01161
LocalWithout0.55413.7620.94—————0.014770.01223
LocalWith0.86018.1525.500.7450.5460.4690.3890.5370.016390.01330

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

We thank the reviewers for their detailed feedback. For the Area Chair's convenience, we summarize the main concerns, additional evidence, and planned revisions below.

  • Contributions beyond existing weight quantization. CoBIT targets weight–activation quantization at extremely low weight bit widths. In this setting, it improves average accuracy over the strongest baseline in each setting by 4.8 points at A4W3 and 35.9 points at A4W2 across four Qwen3 models (Table 1). The comparisons raised by the reviewers verify whether each component contributes on its own. In these experiments, the baselines also use CoBIT's learned rotation and scaling, and the SqueezeLLM codebook also uses CoBIT's assignment. Under this shared setting, CoBIT still has the advantage over the SqueezeLLM codebook at both A4W2 (FKL 0.545 vs. 0.575) and A4W3 (accuracy 79.1% vs. 78.0%), and outperforms LNQ by 1.2 and 0.6 points. In the weight-only setting, CoBIT reaches similar accuracy to the SqueezeLLM codebook with lower PPL and KL, and is stronger than LNQ in the direct comparison. Beyond LNQ, CoBIT additionally models the effect of accumulated upstream quantization error, and we provide a detailed breakdown of the weight component in the appendix to help the community better understand this problem.
  • Benefits of activation optimization. PhaseLUT's gains appear mainly in KL and PPL, because A4 allows only three compositions and QA tasks decode very few tokens. To verify this, we evaluated PhaseLUT at A8 on GPQA, where it outperforms the best fixed composition by 2.0 points on Qwen3-4B. With all seven projections quantized, PhaseLUT also improves FKL, PPL, and all four accuracies over a fixed composition.
  • Optimization framework and interaction term. We revised the term "unified": all stages are derived from one shared objective, and our decomposition removes the need for gradient descent, making CoBIT both faster and more accurate than CodeQuant. New experiments support this design. Calibrating PhaseLUT on the full output error gives nearly identical results to our objective. Including the interaction term clearly hurts the model without upstream quantization (FKL 0.554 to 0.860), and with upstream quantization it keeps accuracy similar but worsens FKL (0.545 to 0.584). Finally, our early-stopping procedure outperforms both one round and five rounds of alternating optimization.
  • Quantization coverage. We limited the quantization scope to avoid the overhead of online rotation and scaling, and applied the same scope to all baselines. When all seven projections are quantized, ParoQuant's released code did not converge in our A4 setup, while CoBIT outperforms LNQ by 2.2 points with fixed activations and 2.7 points with PhaseLUT.
  • Practical inference benefits. CoBIT achieves up to 1.59× end-to-end decoding speedup over FP16. Our optimized kernel is now faster than Marlin at every batch size from 1 to 32 on both GPUs, and uses less GPU memory. W2 and W3 further reduce decoding latency compared with W4, and we provide a detailed latency breakdown.
  • Evaluation protocol. Under log-likelihood scoring, CoBIT still leads the second-best method by 18.4 points on Qwen3-8B and 18.8 points on Qwen3-4B at A4W2. On Qwen3-14B, where the gap was small, our lead grows slightly under log-likelihood.

We incorporated these results and clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.