- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
COTQuant combines a cross-modal orthogonal activation transform with quantization-aware weight-channel regrouping for W4A4 PTQ. It evaluates three Qwen VLMs on five benchmarks and reports small average gaps to BF16, including results on Qwen3.5-122B. Real low-bit inference efficiency is measured on Qwen2-VL-7B.
Critical strengths
The method is motivated by multimodal activation outliers and group-wise weight error, with ablations for both components. The paper compares against DuQuant and QuaRot at W4A4 and reports real inference throughput and memory in the appendix.
Critical weaknesses
-
The headline retention averages five task metrics into one number. On Qwen2-VL-7B, for example, COTQuant improves over QuaRot on some tasks but is lower on OCRBench and ScienceQA; the average alone does not show these task-level trade-offs.
-
The 122B result supports an accuracy claim, while the practical inference measurements are reported only for Qwen2-VL-7B. The paper does not establish the calibration/search cost or deployment overhead at the scale emphasized in the abstract.
Questions for authors
- Is the reported 98.03–98.73% BF16 retention computed from an unnormalized arithmetic mean of the five benchmark scores, or by first normalizing each benchmark to its BF16 result? Please clarify the aggregation used for the main claim.
- For Qwen3.5-122B, were COTQuant and DuQuant evaluated with the same calibration set, bit configuration, and inference backend? This comparison drives the claimed scale-up advantage.
4 comments
Rebuttal by Authors
W2. Lack of reporting on calibration/search costs.
We thank the reviewer for raising this concern about calibration costs. To clarify the offline quantization overhead of COTQuant, we report the calibration time required by different PTQ methods on Qwen3.5-122B:
| Method | Calibration Time |
|---|---|
| SmoothQuant | ~10 min |
| OmniQuant | >120 h |
| DuQuant | ~1 h |
| COTQuant (Ours) | ~5 h |
The offline quantization overhead can be substantial for this model because Qwen3.5-122B has a Mixture-of-Experts (MoE) architecture with a large number of linear layers. For example, OmniQuant uses an entire Transformer layer as its optimization unit, minimizes the quantization mean squared error (MSE) of that layer, and jointly updates the per-channel scaling factors of all its linear layers, resulting in particularly high calibration costs. SmoothQuant and DuQuant, in contrast, complete calibration with a single forward pass. COTQuant lies between these approaches: although it also involves optimization, the optimization is performed separately for each linear layer, resulting in smaller computation graphs and substantially lower calibration costs than OmniQuant. Moreover, COTQuant achieves significantly better quantization accuracy than DuQuant on Qwen3.5-122B. We have added this part to Appendix B of the revised paper.
Q2. Concerns regarding the fairness of experiments
We thank the reviewer for the question. We confirm that COTQuant and DuQuant use the same experimental settings on Qwen3.5-122B to ensure a fair comparison.
As described in Appendix A, both methods use the same 128 samples from COCO train2017 for calibration, adopt the same W4A4 quantization configuration, and are evaluated using the same lmms-eval framework. Thus, the calibration data, quantization bit-width, and evaluation protocol are identical for both methods. The performance advantage of COTQuant on this large-scale VLM therefore does not stem from differences in experimental setup, but from its cross-modal orthogonal transformation designed to mitigate multimodal activation outliers.
I thank the authors for the clarifications, which resolve my question on the fairness of the large-scale comparison. However, the headline retention is a ratio of unnormalized averages that still hides task-level regressions against QuaRot, and the rebuttal provides no real inference measurements at the scale emphasized in the abstract. I therefore maintain my original score.
Reply to Official Comment by Reviewer ejWb
We sincerely thank the reviewer for their time and constructive feedback throughout the review process. We also appreciate the reviewer's acknowledgment that our clarifications have resolved the concerns regarding the fairness of the large-scale comparison. Regarding the two remaining concerns, we would like to provide further clarification and additional experimental evidence.
1. Regarding the average performance retention and task-level trade-offs
We understand the reviewer's concern that averaging unnormalized benchmark scores may obscure task-specific performance regressions. To further examine whether our overall performance advantage depends on the use of unnormalized averages, we additionally calculate the BF16-normalized average retention, where each benchmark score is first normalized by its corresponding BF16 score before averaging across the five benchmarks. Under this alternative metric, QuaRot and COTQuant achieve retention rates of 97.73% and 98.04%, respectively, on Qwen2-VL-7B. Thus, COTQuant still maintains an overall advantage even when benchmark-wise normalization is applied.
These results indicate that the overall performance advantage of COTQuant is not solely attributable to the unnormalized averaging scheme. We emphasize that the average retention is intended to summarize the preservation of overall multimodal capabilities rather than imply superiority on every individual task.
2. Regarding real inference efficiency on Qwen3.5-122B
We appreciate the reviewer's concern regarding the lack of real inference measurements at the 122B scale. Our previous submission did not report W4A4 inference efficiency on Qwen3.5-122B primarily due to the hardware limitations of the Kunlun P800 accelerator used in our experiments.
Specifically, the Kunlun P800 does not support native INT4 computation at the hardware level. INT4 quantization is implemented through fake quantization, making it impossible to directly measure the practical inference acceleration of W4A4 quantization on this platform. Furthermore, the P800 currently lacks optimized Fast Hadamard Transform (FHT) kernels, introducing additional computational overhead for methods involving Hadamard transforms.
Nevertheless, to provide additional evidence of practical quantized inference efficiency at the 122B scale, we conduct inference experiments on Qwen3.5-122B using the native INT8 inference kernels provided by vLLM-Kunlun. All methods are evaluated under the same hardware environment and inference settings. Prefill throughput is measured at a sequence length of 2048, while decode throughput is measured with a batch size of 8. The results are presented below:
| Method | Prefill (tok/s) ↑ | Prefill Speedup ↑ | Decode (tok/s) ↑ | Decode Speedup ↑ | Weight (GiB) ↓ | Peak Memory (GiB) ↓ |
|---|---|---|---|---|---|---|
| BF16 | 9992 | 1.00× | 195.9 | 1.00× | 233.0 | 247.6 |
| SmoothQuant | 11763 | 1.18× | 288.1 | 1.47× | 121.0 | 135.2 |
| DuQuant | 10662 | 1.07× | 260.6 | 1.33× | 121.0 | 135.2 |
| COTQuant | 11057 | 1.11× | 228.9 | 1.17× | 121.0 | 135.2 |
The results demonstrate that INT8 quantized inference provides substantial memory savings and practical inference acceleration on Qwen3.5-122B. Specifically, COTQuant achieves prefill and decode throughputs of 11057 and 228.9 tokens/s, corresponding to 1.11× and 1.17× speedups over BF16, respectively. Meanwhile, model-weight memory is reduced from 233.0 GiB to 121.0 GiB, and peak memory usage decreases from 247.6 GiB to 135.2 GiB, corresponding to approximately 1.93× and 1.83× memory compression, respectively.
We hope these additional results and clarifications help address the remaining concerns regarding evaluation metrics and practical inference efficiency at the 122B scale. We sincerely appreciate the reviewer's careful assessment and constructive feedback.
Rebuttal by Authors
W1&Q1. Concern about the evaluation metrics
We thank the reviewer for the valuable comments. We would like to clarify that using the average score across multiple benchmarks to evaluate overall quantization performance is a widely adopted practice in VLM quantization research. For example, MBQ (CVPR 2025) similarly employs the arithmetic mean of multiple multimodal benchmark scores as a primary evaluation metric. Its experimental results also demonstrate that a method achieving better overall performance does not necessarily outperform others on every individual task. Therefore, performance variations across individual benchmarks do not contradict improvements in overall performance.
Across different models, COTQuant demonstrates more consistent advantages in overall quantization performance. On Qwen2-VL-7B, compared with QuaRot, COTQuant improves TextVQA, VizWiz, and MME scores by 0.56, 0.61, and 1.22 points, respectively, increasing the average score from 78.63 to 78.88. On Qwen3-VL-8B, COTQuant outperforms QuaRot across all five benchmarks, improving the average score from 75.50 to 82.78, a gain of 7.28 points. Notably, QuaRot exhibits considerable performance variations across different models, with average performance drops relative to BF16 of 1.83 and 8.66 points on Qwen2-VL-7B and Qwen3-VL-8B, respectively. In contrast, the corresponding drops for COTQuant are only 1.58 and 1.38 points. Furthermore, COTQuant also achieves strong performance on the larger Qwen3.5-122B model. These results demonstrate that our method not only achieves superior average performance but also consistently preserves the overall performance of full-precision models across different model architectures and scales.
On the other hand, we acknowledge that different quantization methods exhibit task-specific performance trade-offs, which cannot be fully captured by a single average score. The five benchmarks used in our evaluation assess different aspects of multimodal capabilities: OCRBench primarily evaluates visual text recognition and understanding; TextVQA assesses visual question answering that requires reasoning over textual information in images; VizWiz evaluates visual understanding and question answering in challenging real-world scenarios; MME provides a comprehensive evaluation of visual perception and cognitive reasoning; and ScienceQA focuses on scientific knowledge understanding and multimodal reasoning. Therefore, we use the average score across these five benchmarks as an overall indicator of the preservation of comprehensive multimodal capabilities, while also reporting individual benchmark results to reveal task-specific performance differences.
Specifically, on Qwen2-VL-7B, COTQuant scores 0.60 and 0.54 points lower than QuaRot on OCRBench and ScienceQA, respectively, indicating that there remains room for improvement in certain visual text understanding and scientific reasoning tasks. This also suggests that different quantization strategies may affect individual model capabilities differently. In future work, we plan to investigate the differential impacts of quantization methods on various model capabilities and explore quantization strategies that preserve performance more evenly across different tasks.
Finally, regarding the reported 98.03%–98.73% BF16 performance retention, we clarify that it is calculated using the arithmetic mean of the raw scores across the five benchmarks, rather than averaging benchmark-wise scores normalized by their corresponding BF16 results. Specifically, we first compute the average scores of the quantized and BF16 models across the five benchmarks, and then calculate the ratio between these two averages:
$ \text{Retention} = \frac{\text{Average}{\text{Quantized}}} {\text{Average}{\text{BF16}}} \times 100%. $
Thus, the reported performance retention represents the overall average performance preserved by the quantized model relative to its BF16 counterpart under a consistent evaluation protocol.