- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CSQuant combines Cross-Block Smoothing (CBS) and LiteBoA for low-bit quantization of large language models. CBS replaces the channel permutation in the rotation–permutation–rotation (RPR) pipeline with a cross-block Hadamard transform, allowing the two local rotations to be merged. Fused kernels further reduce online overhead. LiteBoA uses standard GPTQ for Q/K and an attention-aware Hessian shared across heads for V. The paper evaluates LLaMA-1/2/3 under W4A4KV4 and weight-only quantization, reporting perplexity, zero-shot accuracy, inference efficiency, and quantization cost.
Critical strengths
- CBS effectively reduces online transformation overhead.
Its factorized structure allows the two local rotations to be merged, while kernel fusion further reduces reads and writes of intermediate tensors. In the single-Transformer-block W4A4KV4 benchmark on an RTX 3090, with a batch size of 4 and a sequence length of 2048, CBS is approximately 20% faster than RPR on both LLaMA-2 7B and 13B. Table 3 also reports lower perplexity for CBS in the evaluated settings. These results support the practical value of the design.
- LiteBoA shows potential for reducing quantization cost.
In Table 1, LiteBoA and Full BoA have the same reported perplexity of 5.89, while LiteBoA reduces quantization time from 6260.26 to 1494.93 seconds and peak GPU memory usage from 23.21 to 12.62 GB. This is useful for quantizing models with limited computational resources.
Critical weaknesses
- The large 2-bit performance gap relative to DuQuant and RUQuant needs a clearer explanation.
In Table 3, RPR achieves W2A16 perplexities of 12.04 and 8.78 on LLaMA-2 7B and 13B, respectively, whereas the corresponding results for DuQuant and RUQuant in Table 2 are orders of magnitude higher. Appendix B.4 describes the native baseline configurations, but does not clearly explain how they differ from the RPR ablation configuration in Table 3. With LiteBoA enabled, replacing RPR with CBS only reduces perplexity from 10.13 to 9.99 and from 8.11 to 8.07. It is therefore necessary to distinguish the contribution of the proposed modules from the effects of differences in weight reconstruction procedures or other implementation settings when interpreting the much larger gains over these two baselines in the main table. This is my main concern about the attribution of the reported accuracy gains.
- The effect of sharing the Value Hessian across heads has not been evaluated in isolation.
Table 1 does compare the attention-aware objectives for Q/K with standard GPTQ. However, it lacks a configuration that uses GPTQ for Q/K while retaining per-head attention-aware Hessians for V. The comparison between Full BoA and LiteBoA changes both the treatment of Q/K and whether the Value Hessian is shared. It therefore does not isolate the effect of head sharing on accuracy and cost. This leaves an important design choice insufficiently validated, especially under the 2-bit settings emphasized in the paper.
Questions for authors
- What are the implementation differences between RPR in Table 3 and DuQuant/RUQuant in Table 2?
Please clarify the differences in the weight quantization backend, quantization range selection, and rotation construction. If different weight reconstruction procedures are used, how much of the W2A16 performance gap remains when these settings are matched?
- With CBS fixed and Q/K quantized using standard GPTQ, how do per-head and shared Value Hessians compare?
Please compare their perplexity, quantization time, and peak GPU memory usage, and provide results for at least one W2A16 configuration to clarify the effect of head sharing under ultra-low-bit quantization.
- Why construct two rotations and then merge them, rather than directly learn the target rotation?
The paper uses a local rotation–cross-block mixing–local rotation pipeline. However, Eq. (8) shows that the final transformation can be written as a single orthogonal transform, , where . It is unclear to me why two local rotations need to be constructed and then merged, rather than directly learning the target transformation, following the rotation-learning approach of SpinQuant. The paper explains that the two rotations adapt to the distributions before and after mixing, respectively. However, please clarify whether the main advantage of this design over directly learning a rotation is lower calibration cost, more efficient block-wise computation and kernel fusion, or better quantization accuracy. Which advantages arise from the final structured transformation, and which specifically require the two-stage construction?
- What practical advantages does CBS offer over FlatQuant, given its more constrained transformation structure?
FlatQuant uses learnable invertible Kronecker factors and fuses the transformations with quantization. In contrast, CBS uses a fixed Hadamard factor and an adaptive orthogonal factor. Since orthogonal transformations are a special case of affine transformations, does the transformation family used by CBS form a subset of that used by FlatQuant? If so, what advantages does CSQuant offer?
5 comments
Response to Reviewer GTpw (Part 2/2)
Q4: About practical advantages of CBS over FlatQuant.
A4: We clarify the structural differences between CBS and FlatQuant, and conduct additional experiments to evaluate their practical efficiency and quantization accuracy.
(1) Structural Differences and Design Advantages. With matching Kronecker factor dimensions and channel layout, CBS is a restricted subset of the transformation family used by FlatQuant. Our goal is to combine layer-specific adaptation with efficient deployment. CBS constructs layer-specific rotations independently for each linear layer without gradient-based optimization. Its structured cross-block mixing allows two local rotations to merge exactly, while the fixed Hadamard factor supports fast butterfly computation and kernel fusion. This design preserves layer-specific adaptation while reducing both rotation-construction cost and online overhead.
(2) Experimental Comparison with FlatQuant. To evaluate these advantages, we conduct additional experiments comparing FlatQuant with the complete CSQuant pipeline, including LiteBoA, on LLaMA-2 7B. As shown in Table 4, CSQuant reduces quantization time by 73.44% and peak GPU memory by 59.03% under W4A4KV4, with only a 0.11 increase in PPL. Under W2A16KV16, CSQuant achieves substantially lower PPL while reducing quantization time by 64.47% and peak GPU memory by 42.80%. We further compare W4A4KV4 inference efficiency during prefill and decode. As shown in Table 5, CSQuant consistently achieves higher inference speedups than FlatQuant across all tested prefill and decode settings. These results demonstrate that, despite its more constrained transformation structure, CSQuant offers substantially lower quantization cost and faster inference, while maintaining competitive accuracy under W4A4KV4 and achieving better accuracy under W2A16KV16.
Table 4: WikiText2 perplexity (PPL ), quantization time (s ), and peak GPU memory (GB ) of FlatQuant and CSQuant on LLaMA-2 7B.
| Setting | Method | PPL | Time(s) | Peak Memory(GB) |
|---|---|---|---|---|
| W4A4KV4 | FlatQuant | 5.78 | 5628.61 | 30.80 |
| CSQuant | 5.89 | 1494.93 | 12.62 | |
| W2A16KV16 | FlatQuant | 44.10 | 4067.95 | 20.96 |
| CSQuant | 9.99 | 1445.51 | 11.99 |
Table 5: W4A4KV4 inference speedups over FP16 () of FlatQuant and CSQuant across prefill and decode stages.
| Phase | Batch | Length | FlatQuant | CSQuant |
|---|---|---|---|---|
| Prefill | 4 | 1024 | 2.21× | 2.32× |
| 4 | 2048 | 2.17× | 2.25× | |
| 4 | 4096 | 2.10× | 2.18× | |
| Decode | 16 | 4096 | 0.88× | 1.28× |
| 32 | 4096 | 1.60× | 1.98× | |
| 64 | 4096 | 2.01× | 2.43× |
Overall, these results show that CSQuant combines layer-specific adaptation with lower quantization cost and faster inference, achieving a favorable accuracy–efficiency trade-off in the tested settings. We have clarified these design advantages and included the additional comparisons in Appendix C.7 of the revised paper.
Thank you for the detailed responses and additional experiments. The matched reconstruction comparisons and the Value-Hessian ablation address my main concerns about gain attribution and validation. The direct-learning comparison also clarifies the meaningful reduction in offline quantization cost. These results strengthen the empirical support for the paper.
My remaining reservation concerns the extent of the methodological contribution. CBS provides an efficient construction within a constrained Kronecker rotation family, while LiteBoA simplifies the existing reconstruction procedure. With LiteBoA fixed, the additional PPL improvement from CBS is modest in the reported W2A16KV16 experiments: 10.13 to 9.99 and 8.11 to 8.07. I consider these useful but incremental advances.
Table 23 also shows that CSQuant is slightly slower than the fusion-enhanced QuaRot and OSTQuant baselines in all four reported TTFT configurations. I acknowledge that the transferable fusion optimizations and lower quantization cost are valuable contributions. However, after considering the full response, I maintain my rating.
Response to Reviewer GTpw (Part 1/2)
Thanks very much for your helpful and constructive comments.
W1 & Q1: About implementation differences and the W2A16KV16 gain attribution.
A1: In the following, we provide the implementation differences and the conduct additional controlled experiments to isolate the contributions of CBS and weight reconstruction.
(1) Implementation differences. RPR in Table 3 of the main paper is a transformation ablation within our quantization framework, rather than the complete native DuQuant/RUQuant pipeline evaluated in Table 2 of the main paper. Specifically, Table 2 uses the native round-to-nearest (RTN) weight quantization and quantization settings of DuQuant/RUQuant, whereas Table 3 uses GPTQ or LiteBoA with symmetric per-output-channel quantization, no weight grouping, and MMSE-based range selection. Regarding rotation construction, Table 2 retains the respective native transformations of DuQuant and RUQuant, while the RPR ablation in Table 3 uses RUQuant-based local rotations and cross-block permutation.
(2) Comparison under Matched Reconstruction Settings. To isolate these effects, we conduct additional experiments comparing DuQuant, RUQuant (RPR), and CBS under the same weight reconstruction methods, GPTQ and LiteBoA, respectively. Within each reconstruction setting, we match quantization-range selection, weight precision and granularity, calibration samples, and evaluation protocols, only the rotation construction and cross-block transformation differ. As shown in Table 1, under matched reconstruction settings, CBS consistently achieves lower PPL than DuQuant and RUQuant (RPR). These results demonstrate that the W2A16KV16 accuracy improvements arise from both weight reconstruction and CBS transformation.
Table 1: WikiText2 perplexity (PPL ↓) of different transformations under GPTQ and LiteBoA with W2A16KV16 quantization.
We have clarified these implementation differences in the caption of Table 3 of the main paper and added the comparison results in Appendix C.6 of the revised paper.
W2 & Q2: About the ablation of Value Hessian sharing across heads.
A2: We conduct experiments to compare per-head and shared Value Hessians on LLaMA2 7B and 13B under W4A4KV4 and W2A16KV16. We keep CBS fixed and use standard GPTQ for Q/K with all other settings unchanged.
Table 2: WikiText2 perplexity (PPL ↓), quantization time (s ↓), and peak GPU memory (GB ↓) with per-head and shared Value Hessians on LLaMA2 models.
As shown in Table 2, head sharing reduces quantization time and memory while keeping PPL almost unchanged. Under W2A16KV16, it reduces quantization time by 8.41–11.33% and peak memory by 15.15–23.41%, with PPL increasing by only 0.18 and 0.12 on the 7B and 13B models, respectively. We have included these results in Section 4.3 of the revised paper.
Q3: About two-stage rotation construction versus direct learning.
A3: We compare our two-stage construction with directly learning the local orthogonal factor R in the same structured family H⊗R using SpinQuant-style optimization, keeping the remaining quantization settings fixed.
Table 3: Comparison of direct learning and two-stage construction under W2A16KV16 on LLaMA-2 7B.
As shown in Table 3, our two-stage construction reduces quantization time by 47.35% compared with direct learning, while maintaining comparable PPL and peak memory usage. It uses gradient-free, closed-form updates to adapt to activations before and after cross-block mixing, avoiding iterative gradient optimization. The demonstrated advantage is therefore lower offline cost. Block-wise execution and kernel fusion follow from the final structured transformation and are available to both approaches. We have included the comparison and discussion in Appendix C.6 of the revised paper.