acceptodds
Under review as a conference paper at ICLR 2027

CSQuant: Cross-Block Smoothing for Accurate and Efficient Low-Bit LLM Quantization

Abstract

Low-bit quantization is essential for efficient large language model (LLM) inference, but aggressive quantization remains vulnerable to activation outliers that amplify quantization error. Orthogonally rotating activations alleviates quantization error but incurs additional online computation, particularly in adaptive methods that use intermediate permutations, which prevent multiple rotations from being combined and introduce explicit data movement. We propose CSQuant, an accurate and efficient post-training quantization framework comprising two key modules: Cross-Block Smoothing and LiteBoA. Cross-Block Smoothing replaces intermediate permutations with a normalized cross-block Hadamard transform, enabling cross-block outlier redistribution while allowing two adaptive rotations to be merged into a single online transformation. This separable structure further enables CUDA kernel fusion with adjacent operations, reducing intermediate tensor materialization and global-memory traffic. LiteBoA selectively applies attention-aware second-order compensation to value projection matrices and employs head-shared Hessians to reduce calibration cost while preserving quantization accuracy. Experiments on LLaMA-1/2/3 demonstrate improved quantization accuracy across W4A4KV4 and weight-only settings, with particularly strong performance at 2-bit weights. CSQuant achieves a 2.25 prefill speedup over FP16 under W4A4KV4 quantization with a sequence length of 2048 and a batch size of 4, while our fusion strategy further accelerates existing Hadamard-based rotation methods. CSQuant also substantially reduces peak GPU memory during post-training quantization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient Low-Bit LLM Quantization, Post-Training Quantization, Low-Bit Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GTpwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CSQuant combines Cross-Block Smoothing (CBS) and LiteBoA for low-bit quantization of large language models. CBS replaces the channel permutation in the rotation–permutation–rotation (RPR) pipeline with a cross-block Hadamard transform, allowing the two local rotations to be merged. Fused kernels further reduce online overhead. LiteBoA uses standard GPTQ for Q/K and an attention-aware Hessian shared across heads for V. The paper evaluates LLaMA-1/2/3 under W4A4KV4 and weight-only quantization, reporting perplexity, zero-shot accuracy, inference efficiency, and quantization cost.

Critical strengths
  1. CBS effectively reduces online transformation overhead.

Its factorized structure allows the two local rotations to be merged, while kernel fusion further reduces reads and writes of intermediate tensors. In the single-Transformer-block W4A4KV4 benchmark on an RTX 3090, with a batch size of 4 and a sequence length of 2048, CBS is approximately 20% faster than RPR on both LLaMA-2 7B and 13B. Table 3 also reports lower perplexity for CBS in the evaluated settings. These results support the practical value of the design.

  1. LiteBoA shows potential for reducing quantization cost.

In Table 1, LiteBoA and Full BoA have the same reported perplexity of 5.89, while LiteBoA reduces quantization time from 6260.26 to 1494.93 seconds and peak GPU memory usage from 23.21 to 12.62 GB. This is useful for quantizing models with limited computational resources.

Critical weaknesses
  1. The large 2-bit performance gap relative to DuQuant and RUQuant needs a clearer explanation.

In Table 3, RPR achieves W2A16 perplexities of 12.04 and 8.78 on LLaMA-2 7B and 13B, respectively, whereas the corresponding results for DuQuant and RUQuant in Table 2 are orders of magnitude higher. Appendix B.4 describes the native baseline configurations, but does not clearly explain how they differ from the RPR ablation configuration in Table 3. With LiteBoA enabled, replacing RPR with CBS only reduces perplexity from 10.13 to 9.99 and from 8.11 to 8.07. It is therefore necessary to distinguish the contribution of the proposed modules from the effects of differences in weight reconstruction procedures or other implementation settings when interpreting the much larger gains over these two baselines in the main table. This is my main concern about the attribution of the reported accuracy gains.

  1. The effect of sharing the Value Hessian across heads has not been evaluated in isolation.

Table 1 does compare the attention-aware objectives for Q/K with standard GPTQ. However, it lacks a configuration that uses GPTQ for Q/K while retaining per-head attention-aware Hessians for V. The comparison between Full BoA and LiteBoA changes both the treatment of Q/K and whether the Value Hessian is shared. It therefore does not isolate the effect of head sharing on accuracy and cost. This leaves an important design choice insufficiently validated, especially under the 2-bit settings emphasized in the paper.

Questions for authors
  1. What are the implementation differences between RPR in Table 3 and DuQuant/RUQuant in Table 2?

Please clarify the differences in the weight quantization backend, quantization range selection, and rotation construction. If different weight reconstruction procedures are used, how much of the W2A16 performance gap remains when these settings are matched?

  1. With CBS fixed and Q/K quantized using standard GPTQ, how do per-head and shared Value Hessians compare?

Please compare their perplexity, quantization time, and peak GPU memory usage, and provide results for at least one W2A16 configuration to clarify the effect of head sharing under ultra-low-bit quantization.

  1. Why construct two rotations and then merge them, rather than directly learn the target rotation?

The paper uses a local rotation–cross-block mixing–local rotation pipeline. However, Eq. (8) shows that the final transformation can be written as a single orthogonal transform, T=HB⊗R∗T=H_B\otimes R_*, where R∗=R1R2R_*=R_1R_2. It is unclear to me why two local rotations need to be constructed and then merged, rather than directly learning the target transformation, following the rotation-learning approach of SpinQuant. The paper explains that the two rotations adapt to the distributions before and after mixing, respectively. However, please clarify whether the main advantage of this design over directly learning a rotation is lower calibration cost, more efficient block-wise computation and kernel fusion, or better quantization accuracy. Which advantages arise from the final structured transformation, and which specifically require the two-stage construction?

  1. What practical advantages does CBS offer over FlatQuant, given its more constrained transformation structure?

FlatQuant uses learnable invertible Kronecker factors and fuses the transformations with quantization. In contrast, CBS uses a fixed Hadamard factor and an adaptive orthogonal factor. Since orthogonal transformations are a special case of affine transformations, does the transformation family used by CBS form a subset of that used by FlatQuant? If so, what advantages does CSQuant offer?

5 comments
AuthorsOpenReview ↗

Response to Reviewer GTpw (Part 1/2)

Thanks very much for your helpful and constructive comments.

W1 & Q1: About implementation differences and the W2A16KV16 gain attribution.

A1: In the following, we provide the implementation differences and the conduct additional controlled experiments to isolate the contributions of CBS and weight reconstruction.

(1) Implementation differences. RPR in Table 3 of the main paper is a transformation ablation within our quantization framework, rather than the complete native DuQuant/RUQuant pipeline evaluated in Table 2 of the main paper. Specifically, Table 2 uses the native round-to-nearest (RTN) weight quantization and quantization settings of DuQuant/RUQuant, whereas Table 3 uses GPTQ or LiteBoA with symmetric per-output-channel quantization, no weight grouping, and MMSE-based range selection. Regarding rotation construction, Table 2 retains the respective native transformations of DuQuant and RUQuant, while the RPR ablation in Table 3 uses RUQuant-based local rotations and cross-block permutation.

(2) Comparison under Matched Reconstruction Settings. To isolate these effects, we conduct additional experiments comparing DuQuant, RUQuant (RPR), and CBS under the same weight reconstruction methods, GPTQ and LiteBoA, respectively. Within each reconstruction setting, we match quantization-range selection, weight precision and granularity, calibration samples, and evaluation protocols, only the rotation construction and cross-block transformation differ. As shown in Table 1, under matched reconstruction settings, CBS consistently achieves lower PPL than DuQuant and RUQuant (RPR). These results demonstrate that the W2A16KV16 accuracy improvements arise from both weight reconstruction and CBS transformation.

Table 1: WikiText2 perplexity (PPL ↓\downarrow) of different transformations under GPTQ and LiteBoA with W2A16KV16 quantization.

TransformationReconstructionLLaMA-2 7BLLaMA-2 13B
DuQuantGPTQ13.929.89
RUQuant (RPR)GPTQ12.048.78
CBSGPTQ10.968.51
DuQuantLiteBoA12.008.85
RUQuant (RPR)LiteBoA10.138.11
CBSLiteBoA9.998.07

We have clarified these implementation differences in the caption of Table 3 of the main paper and added the comparison results in Appendix C.6 of the revised paper.

W2 & Q2: About the ablation of Value Hessian sharing across heads.

A2: We conduct experiments to compare per-head and shared Value Hessians on LLaMA2 7B and 13B under W4A4KV4 and W2A16KV16. We keep CBS fixed and use standard GPTQ for Q/K with all other settings unchanged.

Table 2: WikiText2 perplexity (PPL ↓\downarrow), quantization time (s ↓\downarrow), and peak GPU memory (GB ↓\downarrow) with per-head and shared Value Hessians on LLaMA2 models.

ModelSettingV HessianPPL ↓\downarrowTime(s) ↓\downarrowPeak Memory(GB) ↓\downarrow
7BW4A4KV4Per-head5.891593.9915.72
Shared5.891494.9312.62
13BW4A4KV4Per-head5.202894.8320.96
Shared5.192784.0916.06
7BW2A16KV16Per-head9.811630.2514.13
Shared9.991445.5111.99
13BW2A16KV16Per-head7.953125.8020.93
Shared8.072863.0016.03

As shown in Table 2, head sharing reduces quantization time and memory while keeping PPL almost unchanged. Under W2A16KV16, it reduces quantization time by 8.41–11.33% and peak memory by 15.15–23.41%, with PPL increasing by only 0.18 and 0.12 on the 7B and 13B models, respectively. We have included these results in Section 4.3 of the revised paper.

Q3: About two-stage rotation construction versus direct learning.

A3: We compare our two-stage construction with directly learning the local orthogonal factor RR in the same structured family H⊗RH \otimes R using SpinQuant-style optimization, keeping the remaining quantization settings fixed.

Table 3: Comparison of direct learning and two-stage construction under W2A16KV16 on LLaMA-2 7B.

Rotation ConstructionPPL ↓\downarrowTime(s)↓\downarrowPeak Memory(GB)↓\downarrow
Direct learning10.042745.2711.99
Two-stage construction9.991445.5111.99

As shown in Table 3, our two-stage construction reduces quantization time by 47.35% compared with direct learning, while maintaining comparable PPL and peak memory usage. It uses gradient-free, closed-form updates to adapt to activations before and after cross-block mixing, avoiding iterative gradient optimization. The demonstrated advantage is therefore lower offline cost. Block-wise execution and kernel fusion follow from the final structured transformation and are available to both approaches. We have included the comparison and discussion in Appendix C.6 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer GTpw (Part 2/2)

Q4: About practical advantages of CBS over FlatQuant.

A4: We clarify the structural differences between CBS and FlatQuant, and conduct additional experiments to evaluate their practical efficiency and quantization accuracy.

(1) Structural Differences and Design Advantages. With matching Kronecker factor dimensions and channel layout, CBS is a restricted subset of the transformation family used by FlatQuant. Our goal is to combine layer-specific adaptation with efficient deployment. CBS constructs layer-specific rotations independently for each linear layer without gradient-based optimization. Its structured cross-block mixing allows two local rotations to merge exactly, while the fixed Hadamard factor supports fast butterfly computation and kernel fusion. This design preserves layer-specific adaptation while reducing both rotation-construction cost and online overhead.

(2) Experimental Comparison with FlatQuant. To evaluate these advantages, we conduct additional experiments comparing FlatQuant with the complete CSQuant pipeline, including LiteBoA, on LLaMA-2 7B. As shown in Table 4, CSQuant reduces quantization time by 73.44% and peak GPU memory by 59.03% under W4A4KV4, with only a 0.11 increase in PPL. Under W2A16KV16, CSQuant achieves substantially lower PPL while reducing quantization time by 64.47% and peak GPU memory by 42.80%. We further compare W4A4KV4 inference efficiency during prefill and decode. As shown in Table 5, CSQuant consistently achieves higher inference speedups than FlatQuant across all tested prefill and decode settings. These results demonstrate that, despite its more constrained transformation structure, CSQuant offers substantially lower quantization cost and faster inference, while maintaining competitive accuracy under W4A4KV4 and achieving better accuracy under W2A16KV16.

Table 4: WikiText2 perplexity (PPL ↓\downarrow), quantization time (s ↓\downarrow), and peak GPU memory (GB ↓\downarrow) of FlatQuant and CSQuant on LLaMA-2 7B.

SettingMethodPPL ↓\downarrowTime(s) ↓\downarrowPeak Memory(GB) ↓\downarrow
W4A4KV4FlatQuant5.785628.6130.80
CSQuant5.891494.9312.62
W2A16KV16FlatQuant44.104067.9520.96
CSQuant9.991445.5111.99

Table 5: W4A4KV4 inference speedups over FP16 (↑\uparrow) of FlatQuant and CSQuant across prefill and decode stages.

PhaseBatchLengthFlatQuantCSQuant
Prefill410242.21×2.32×
420482.17×2.25×
440962.10×2.18×
Decode1640960.88×1.28×
3240961.60×1.98×
6440962.01×2.43×

Overall, these results show that CSQuant combines layer-specific adaptation with lower quantization cost and faster inference, achieving a favorable accuracy–efficiency trade-off in the tested settings. We have clarified these design advantages and included the additional comparisons in Appendix C.7 of the revised paper.

Thank you for the detailed responses and additional experiments. The matched reconstruction comparisons and the Value-Hessian ablation address my main concerns about gain attribution and validation. The direct-learning comparison also clarifies the meaningful reduction in offline quantization cost. These results strengthen the empirical support for the paper.

My remaining reservation concerns the extent of the methodological contribution. CBS provides an efficient construction within a constrained Kronecker rotation family, while LiteBoA simplifies the existing reconstruction procedure. With LiteBoA fixed, the additional PPL improvement from CBS is modest in the reported W2A16KV16 experiments: 10.13 to 9.99 and 8.11 to 8.07. I consider these useful but incremental advances.

Table 23 also shows that CSQuant is slightly slower than the fusion-enhanced QuaRot and OSTQuant baselines in all four reported TTFT configurations. I acknowledge that the transferable fusion optimizations and lower quantization cost are valuable contributions. However, after considering the full response, I maintain my rating.

Reviewer_vXENOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a post-training quantization framework with low bit-width, CSQuant, aiming to simultaneously address the issues of accuracy in rotation quantization, online inference overhead, and offline quantization costs, consisting of two modules:

  • Cross-Block Smoothing (CBS): Replace the cross-block permutation in the traditional RPR process with normalized cross-block Hadamard transform. Since the cross-block transform and intra-block shared rotation operate on complementary dimensions, they can utilize the Kronecker structure to exchange their order, thereby merging the two consecutive local rotations into a single operation. Fusion CUDA kernel: Integrates cross-block Hadamard with INT4 quantization, packing, SiLU, and gated multiplication to reduce intermediate tensor writes and memory traffic.
  • LiteBoA: Uses standard GPTQ Hessian on Q and K projections, retains attention-aware Hessian only for V projection, and shares Hessian across different attention heads to reduce BoA's quantization time and memory cost.
Critical strengths

CUDA fusion is not merely a conceptual framework; the appendix provides the execution flow and kernel pseudocode. The model scale ranges from 7B to 70B, with comprehensive results reported for nine zero-shot tasks.

Critical weaknesses

The key equation of CBS essentially stems from the Kronecker product rule: I⊗R and H⊗I commute, enabling the merging of two intra-block rotations. The paper needs to more clearly distinguish its differences from existing decompositions such as Hadamard, RUQuant/DuQuant cross-block mixing, and QuaRot kernel optimizations.

Questions for authors

Give more newer models experiments and detail analysis.

2 comments
AuthorsOpenReview ↗

Response to Reviewer vXEN (Part 1/2)

Thanks very much for your valuable, insightful and constructive comments.

W1: Distinction from existing methods.

A1: Thanks very much for your suggestion. We would like to clarify our contribution and its relationship to existing methods in three aspects:

(1) Clarifying our contribution. CBS aims to combine layer-specific adaptation with efficient deployment. It constructs layer-adaptive rotations independently for each linear layer without gradient-based optimization, avoiding the rotation-training overhead of optimization-based fusion methods such as SpinQuant/OSTQuant, while reducing the online overhead of layer-adaptive methods such as DuQuant/RUQuant.

(2) Distinction between transformations. Unlike fixed Hadamard transforms, CBS retains layer-adaptive local rotations. Comparing with the DuQuant/RUQuant permutations, the cross-block mixing of CBS commutes with block-shared rotations, allowing two local rotations to merge exactly at deployment. This preserves adaptive smoothing and cross-block mixing while eliminating one online rotation and the explicit permutation.

(3) Orthogonal kernel optimizations. Our fusion strategy is orthogonal to QuaRot low-bit matrix-multiplication kernels and can be used together with them. It fuses online Hadamard mixing with adjacent quantization/packing or SiLU/gating operations to reduce memory traffic, also benefiting compatible QuaRot paths.

We have added these dissusions in Sections 2 and 3.2 of the revised paper.

W2: End-to-end evaluation under W4A4KV4.

A2: We conduct end-to-end W4A4KV4 inference experiments on LLaMA2-7B in vLLM, comparing time to first token (TTFT, ms) and output throughput (tokens/s) with existing quantization methods and FP16 baseline.

Table 1: TTFT (ms ↓\downarrow) of LLaMA-2 7B under W4A4KV4. Columns denote batch size/input length (tokens).

Method1/5121/20484/5124/2048
FP16114.88438.72424.971671.00
DuQuant51.10175.63161.57666.56
QuaRot42.60161.91147.26621.22
OSTQuant41.64152.13138.49582.98
CSQuant39.08148.12134.29566.79

Table 2: Output throughput (tokens/s ↑\uparrow) of LLaMA-2 7B under W4A4KV4. Input/output lengths are 128/256 tokens, and C denotes concurrency.

MethodC=1C=4C=8C=16C=32
FP1653.99208.08381.08657.65992.02
DuQuant78.30296.05541.00935.421470.01
QuaRot85.35320.36581.901008.991569.74
OSTQuant86.14323.60588.751023.081602.16
CSQuant84.00317.28575.511009.401593.92

As shown in Tables 1 and 2, CSQuant achieves 2.94–3.16× TTFT speedups and 1.51–1.61× output throughput relative to FP16. It delivers the lowest TTFT across all tested settings, with throughput higher than DuQuant and comparable to QuaRot and OSTQuant. These results support its end-to-end efficiency under W4A4KV4. We have included these results and analysis in Section 4.4 and Appendix C.5 of the revised paper.

W3: Zero-shot accuracy at 4-bit precision.

A3: Thank you for this clarification. We have refined the claims in the abstract and introduction of the revised paper to more accurately describe our zero-shot accuracy results under W4A4KV4, W4A16, and W3A16.

W4: Hardware coverage of the system evaluation.

A4: We conduct W4A4KV4 prefill experiments on NVIDIA L20 and A800 GPUs using a single LLaMA-2 7B Transformer block with batch size 4.

Table 3: W4A4KV4 prefill speedups over FP16 (↑\uparrow) on NVIDIA L20. Columns denote sequence lengths (tokens).

Method5121024204840968192
QuaRot1.77×1.60×1.48×1.44×1.42×
DuQuant1.71×1.52×1.40×1.35×1.35×
OSTQuant1.85×1.67×1.53×1.48×1.47×
CSQuant1.93×1.73×1.62×1.56×1.54×

Table 4: W4A4KV4 prefill speedups over FP16 (↑\uparrow) on NVIDIA A800. Columns denote sequence lengths (tokens).

Method5121024204840968192
QuaRot1.07×1.09×1.07×1.07×1.10×
DuQuant1.05×1.07×1.04×1.04×1.04×
OSTQuant1.13×1.16×1.04×1.14×1.16×
CSQuant1.24×1.28×1.25×1.25×1.27×

As shown in Tables 3 and 4, CSQuant consistently achieves greater speedups than strong baselines on both L20 and A800 GPUs, further demonstrating its practical efficiency across different real-world deployment environments. We have included these inference measurements in Appendix C.5 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer vXEN (Part 2/2)

W5: Statistical stability analysis.

A5: We add quantization and latency stability experiments on LLaMA2-7B across 5 different seeds.

(1) Quantization stability. We evaluate W2A16KV16 and W4A4KV4 using five random seeds to sample calibration data, while keeping the calibration size and other settings unchanged. As shown in Table 5, CSQuant exhibits small perplexity variations across different calibration seeds under both quantization settings, demonstrating its robustness to calibration data selection.

Table 5: WikiText2 perplexity (↓\downarrow) of CSQuant across five calibration seeds on LLaMA-2 7B.

SettingSeed=1Seed=2Seed=3Seed=4Seed=5Mean±SD
W2A16KV169.99010.0439.9869.91210.0469.995±0.055
W4A4KV45.9035.9015.8975.8845.8955.896±0.007

(2) Timing stability. We measure W4A4KV4 prefill latency on NVIDIA RTX 3090 using a single LLaMA2-7B Transformer block with batch size 4. As illustrated in Table 6, CSQuant achieves both lower latency and more stable runtime performance across different sequence lengths.

Table 6: W4A4KV4 prefill latency (ms ↓\downarrow, mean ± SD) on NVIDIA RTX 3090. Columns denote sequence lengths (tokens).

Method5121024204840968192
QuaRot7.19±0.1114.02±0.1428.61±0.3861.21±1.07140.10±1.93
DuQuant7.71±0.1115.00±0.1430.47±0.3764.87±1.04151.75±2.62
OSTQuant6.87±0.1013.40±0.1427.18±0.3658.52±1.01134.74±1.78
CSQuant6.50±0.0912.59±0.1225.60±0.2855.12±0.91128.67±1.26

W6: Coverage of model architectures and tasks.

A6: We evaluate Qwen3-8B and Mistral-7B-v0.1 under both W4A4KV4 and W2A16KV16, using WikiText2 perplexity and 5-shot MMLU accuracy to verify the generalizability of our method.

Table 7: WikiText2 perplexity (PPL ↓\downarrow) and 5-shot MMLU accuracy (% ↑\uparrow).

SettingMethodQwen3-8B PPLMMLUMistral-7B-v0.1 PPLMMLU
FP16—9.7174.915.2562.67
W4A4KV4QuaRot11.2667.385.6957.34
OSTQuant10.7568.385.6257.73
CSQuant10.5567.685.5958.75
W2A16KV16QuaRot16.2740.6210.9725.92
OSTQuant23.8334.04268.1125.74
CSQuant14.7946.629.0432.88

As shown in Table 7, CSQuant achieves the lowest PPL across both models under W4A4KV4 and W2A16KV16. For MMLU, CSQuant achieves the highest accuracy on Mistral-7B-v0.1 under W4A4KV4 and on both models under W2A16KV16, demonstrating particularly strong performance at extremely low bit widths. These results further validate the generalizability of CSQuant beyond the LLaMA family and traditional commonsense tasks. We have included these results in Appendix C.4 of the revised paper.

Reviewer_gXaQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CSQuant, a low-bit post-training quantization framework that combines Cross-Block Smoothing and LiteBoA. It uses a cross-block Hadamard transform to merge adaptive rotations and simplifies attention-aware second-order quantization to improve quantization accuracy and inference efficiency.

Critical strengths
  • Innovative method design: Cross-Block Smoothing uses the commutativity of transformations to merge two adaptive rotations, reducing online computation while preserving cross-block outlier redistribution.
  • Practical system optimizations: The authors develop fused CUDA kernels to reduce data movement and extend these optimizations to existing methods such as QuaRot and OSTQuant.
  • Comprehensive experimental results: The experiments cover different model sizes and quantization bit-widths, showing strong accuracy under W2A16 and significantly reduced GPU memory usage during quantization.
Critical weaknesses
  • Unclear performance bottleneck: The paper makes considerable efforts to optimize online rotations, but lacks a detailed performance analysis showing how much online rotations contribute to the overall inference latency and whether they are a major bottleneck.
  • Limited model architectures and quantization settings: All experiments are conducted on the LLaMA family, leaving it unclear whether the method can maintain similar accuracy and efficiency on other popular LLM architectures and quantization settings.
  • Insufficient end-to-end inference evaluation: The W4A4KV4 efficiency evaluation mainly focuses on a single Transformer block and full-model prefill, while the actual serving experiments only consider W4A16, leaving the end-to-end benefits under W4A4KV4 insufficiently validated.
Questions for authors
  1. Could the authors provide a detailed latency breakdown of online rotations, permutation, quantization, and matrix multiplication during prefill and decoding to clarify whether online rotations are a major performance bottleneck worth optimizing?

  2. Could the authors extend the experiments to other model architectures, such as Qwen3-8B or Qwen3-32B, and report quantization accuracy under W4A4KV4 and W4A8KV4 to evaluate the generalizability of the proposed method?

  3. Could the authors provide additional end-to-end W4A4KV4 experiments using vLLM, measuring TTFT with batch sizes of 1 and 4 and input lengths of 512 and 2048 tokens, as well as TPOT and serving throughput with 128 input tokens, 256 output tokens, and concurrency levels of 1, 4, 8, 16, and 32, while comparing against QuaRot and OSTQuant with the same kernel fusion optimizations?

4 comments
AuthorsOpenReview ↗

Response to Reviewer gXaQ (Part 1/2)

Thanks very much for your positive, insightful and constructive comments.

W1 & Q1: Latency breakdown and online transformation overhead.

A1: To quantify the overhead of online rotations, we measure the latency breakdown (in ms) of a single LLaMA-2 7B Transformer block under W4A4KV4. Prefill stage uses a batch size of 4 and a sequence length of 2048, and decode stage uses a batch size of 8 and measures one decoding step with a pre-populated KV cache of length 4096. In the following table, rotation-permutation-rotation (RPR) follows the baseline configuration, CBS (w/o kernel fusion) and CBS denote our method without and with kernel fusion. Other includes all remaining operations, such as attention, normalization, SiLU, and gating. Total denotes the total block latency, including timing instrumentation overhead.

Table 1: Latency breakdown (ms) of a single LLaMA-2 7B Transformer block under W4A4KV4.

StageMethodOnline RotationsPermutationQuantizationLinear GEMMOtherTotal
PrefillRPR1.953.613.758.0413.1330.49
CBS (w/o kernel fusion)2.380.003.768.0613.1427.33
CBS1.620.002.868.0512.7025.24
DecodeRPR0.130.060.500.422.513.61
CBS (w/o kernel fusion)0.120.000.420.412.473.42
CBS0.080.000.310.412.453.24

As shown in Table 1, online rotations and permutation account for 6.4% and 11.8% of RPR prefill latency, respectively. These two transformations account for 18.2% of prefill latency, making them a meaningful target for optimization. Relative to RPR, CBS reduces total latency by 10.4% without kernel fusion and by 17.2% with fusion. During the decode stage, these transformations account for 5.3% of RPR latency, and CBS still achieves a 10.2% latency reduction. Overall, these results show that online transformation overhead is worth optimizing, particularly during prefill. We have added these dissusions in Appendix C.5 of the revised paper.

W2&Q2:Generalization across model architectures and quantization settings.

A2: In terms of quantization accuracy, we evaluate Qwen3-8B, Qwen3-32B, and Mistral-7B-v0.1 under W4A4KV4, W4A8KV4, and W2A16KV16, reporting WikiText2 perplexity (↓\downarrow) in the Table 2. From the table, we can see that CSQuant achieves the lowest (or tied-lowest) perplexity across all three models under W4A4KV4 and W4A8KV4, and strictly achieves the lowest perplexity under W2A16KV16. These demonstrate the generalizability of CSQuant beyond the LLaMA family.

Table 2: WikiText2 perplexity (↓\downarrow) across different models and quantization settings.

SettingMethodQwen3-8BQwen3-32BMistral-7B-v0.1
FP16—9.717.605.25
W4A4KV4QuaRot11.268.645.69
OSTQuant10.758.095.62
CSQuant10.558.095.59
W4A8KV4QuaRot10.057.755.33
OSTQuant10.057.745.34
CSQuant9.927.745.33
W2A16KV16QuaRot16.2710.3910.97
OSTQuant23.83106.56268.11
CSQuant14.7910.379.04

In terms of quantization efficiency, we additionally evaluate single transformer block prefill latency on Qwen3-8B under W4A4KV4, using a batch size of 4 on NVIDIA RTX 3090 GPU. Table 3 reports latency in milliseconds across different input lengths. We can obtain that CSQuant achieves the lowest latency among the compared methods at every tested input length, providing 2.08–2.49× speedups over FP16. These results further verify the inference efficiency of our method on Qwen3-8B beyond the LLaMA family.

Table 3: Prefill latency (ms ↓\downarrow) of a single Qwen3-8B Transformer block under W4A4KV4 on NVIDIA RTX 3090.

Method2565121024204840968192
FP168.7015.2728.9457.82117.98261.54
QuaRot3.896.9213.3827.2458.34134.67
OSTQuant3.676.6112.7725.9055.82129.98
RPR4.267.7014.8830.1464.71153.79
CSQuant3.506.3012.0924.5652.91125.48

We have added these results and discussion in Appendix C.4 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer gXaQ (Part 2/2)

W3 & Q3: End-to-end W4A4KV4 evaluation.

A3: We conduct end-to-end W4A4KV4 inference experiments on LLaMA-2 7B using a modified vLLM implementation. Since vLLM does not natively support the required INT4 × INT4 matrix multiplication path, we modify its source code to enable W4A4KV4 inference following QuaRot. Table 4 reports TTFT with batch sizes of 1 and 4 and input lengths of 512 and 2048 tokens. Tables 5 and 6 report TPOT and serving throughput, respectively, using 128 input tokens and 256 output tokens at concurrency levels of 1, 4, 8, 16, and 32. “+Fusion” denotes the integration of our proposed fused kernels into the corresponding baselines.

Table 4: Time to first token (TTFT, ms ↓\downarrow) of LLaMA-2 7B under W4A4KV4.

Method1/5121/20484/5124/2048
QuaRot42.60161.91147.26621.22
QuaRot+Fusion38.62146.33131.00557.31
OSTQuant41.64152.13138.49582.98
OSTQuant+Fusion37.80145.60131.35557.13
CSQuant39.08148.12134.29566.79

Table 5: Time per output token (TPOT, ms/token ↓\downarrow) of LLaMA-2 7B under W4A4KV4.

MethodC=1C=4C=8C=16C=32
QuaRot11.6312.3513.4915.3319.33
QuaRot+Fusion11.5612.2713.4115.1618.84
OSTQuant11.5312.2313.3515.1518.99
OSTQuant+Fusion11.5612.2913.4415.2318.90
CSQuant11.8112.4913.6715.3819.11

Table 6: Output throughput (tokens/s ↑\uparrow) of LLaMA-2 7B under W4A4KV4.

MethodC=1C=4C=8C=16C=32
QuaRot85.35320.36581.901008.991569.74
QuaRot+Fusion86.07322.89586.861023.921618.80
OSTQuant86.14323.60588.751023.081602.16
OSTQuant+Fusion85.95322.49585.371020.181614.43
CSQuant84.00317.28575.511009.401593.92

As shown in Tables 4, 5 and 6, our kernel fusion reduces TTFT by 9.34–11.04% for QuaRot and 4.29–9.22% for OSTQuant. Compared with the better of the two fusion-optimized baselines at each setting, CSQuant incurs at most 3.39% higher TTFT, 2.16% higher TPOT, and 2.41% lower output throughput. These results demonstrate that CSQuant achieves competitive end-to-end W4A4KV4 inference efficiency, while our proposed kernel fusion also effectively accelerates other methods, particularly during prefill. We have included these results in Section 4.4 and Appendix C.5 of the revised paper.

Thank you for the response and additional experiments. The new results help clarify the cost of online rotations, how well CSQuant generalizes beyond LLaMA, and its end-to-end W4A4KV4 performance. My overall assessment remains positive, and I will maintain my original rating of Weak Accept.

Thank you for your positive assessment and constructive feedback. We appreciate your time and thoughtful evaluation, which have helped us improve the clarity and empirical support of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.