acceptodds
Under review as a conference paper at ICLR 2027

COTQuant: Cross-Modal Orthogonal Transform for Large Vision-Language Model Quantization

Abstract

Post-training quantization (PTQ) is essential for efficient deployment of vision-language models (VLMs), yet low-bit quantization remains challenging due to heterogeneous multimodal activation outliers. We propose COTQuant, a PTQ framework with two complementary components. Cross-Modal Transform (CMT) jointly redistributes text and vision activation outliers through hierarchical orthogonal transformations, while Quantization-Aware Group Search (QAS) reorganizes weight channels to minimize group-wise quantization error. Both transformations leave the model output unchanged and introduce only a small amount of additional inference-time overhead. Across three VLMs and five multimodal benchmarks, COTQuant consistently outperforms existing PTQ methods while retaining 98.03%–98.73% of the corresponding BF16 performance under W4A4 quantization. Notably, we demonstrate effective 4-bit PTQ on Qwen3.5-122B, which is the first 4-bit PTQ attempt on over 100B-scale VLM.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Vision-Language Models, Post-Training Quantization, Weight-Activation Quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ejWbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

COTQuant combines a cross-modal orthogonal activation transform with quantization-aware weight-channel regrouping for W4A4 PTQ. It evaluates three Qwen VLMs on five benchmarks and reports small average gaps to BF16, including results on Qwen3.5-122B. Real low-bit inference efficiency is measured on Qwen2-VL-7B.

Critical strengths

The method is motivated by multimodal activation outliers and group-wise weight error, with ablations for both components. The paper compares against DuQuant and QuaRot at W4A4 and reports real inference throughput and memory in the appendix.

Critical weaknesses
  1. The headline retention averages five task metrics into one number. On Qwen2-VL-7B, for example, COTQuant improves over QuaRot on some tasks but is lower on OCRBench and ScienceQA; the average alone does not show these task-level trade-offs.

  2. The 122B result supports an accuracy claim, while the practical inference measurements are reported only for Qwen2-VL-7B. The paper does not establish the calibration/search cost or deployment overhead at the scale emphasized in the abstract.

Questions for authors
  1. Is the reported 98.03–98.73% BF16 retention computed from an unnormalized arithmetic mean of the five benchmark scores, or by first normalizing each benchmark to its BF16 result? Please clarify the aggregation used for the main claim.
  2. For Qwen3.5-122B, were COTQuant and DuQuant evaluated with the same calibration set, bit configuration, and inference backend? This comparison drives the claimed scale-up advantage.
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

W1&Q1. Concern about the evaluation metrics

We thank the reviewer for the valuable comments. We would like to clarify that using the average score across multiple benchmarks to evaluate overall quantization performance is a widely adopted practice in VLM quantization research. For example, MBQ (CVPR 2025) similarly employs the arithmetic mean of multiple multimodal benchmark scores as a primary evaluation metric. Its experimental results also demonstrate that a method achieving better overall performance does not necessarily outperform others on every individual task. Therefore, performance variations across individual benchmarks do not contradict improvements in overall performance.

Across different models, COTQuant demonstrates more consistent advantages in overall quantization performance. On Qwen2-VL-7B, compared with QuaRot, COTQuant improves TextVQA, VizWiz, and MME scores by 0.56, 0.61, and 1.22 points, respectively, increasing the average score from 78.63 to 78.88. On Qwen3-VL-8B, COTQuant outperforms QuaRot across all five benchmarks, improving the average score from 75.50 to 82.78, a gain of 7.28 points. Notably, QuaRot exhibits considerable performance variations across different models, with average performance drops relative to BF16 of 1.83 and 8.66 points on Qwen2-VL-7B and Qwen3-VL-8B, respectively. In contrast, the corresponding drops for COTQuant are only 1.58 and 1.38 points. Furthermore, COTQuant also achieves strong performance on the larger Qwen3.5-122B model. These results demonstrate that our method not only achieves superior average performance but also consistently preserves the overall performance of full-precision models across different model architectures and scales.

On the other hand, we acknowledge that different quantization methods exhibit task-specific performance trade-offs, which cannot be fully captured by a single average score. The five benchmarks used in our evaluation assess different aspects of multimodal capabilities: OCRBench primarily evaluates visual text recognition and understanding; TextVQA assesses visual question answering that requires reasoning over textual information in images; VizWiz evaluates visual understanding and question answering in challenging real-world scenarios; MME provides a comprehensive evaluation of visual perception and cognitive reasoning; and ScienceQA focuses on scientific knowledge understanding and multimodal reasoning. Therefore, we use the average score across these five benchmarks as an overall indicator of the preservation of comprehensive multimodal capabilities, while also reporting individual benchmark results to reveal task-specific performance differences.

Specifically, on Qwen2-VL-7B, COTQuant scores 0.60 and 0.54 points lower than QuaRot on OCRBench and ScienceQA, respectively, indicating that there remains room for improvement in certain visual text understanding and scientific reasoning tasks. This also suggests that different quantization strategies may affect individual model capabilities differently. In future work, we plan to investigate the differential impacts of quantization methods on various model capabilities and explore quantization strategies that preserve performance more evenly across different tasks.

Finally, regarding the reported 98.03%–98.73% BF16 performance retention, we clarify that it is calculated using the arithmetic mean of the raw scores across the five benchmarks, rather than averaging benchmark-wise scores normalized by their corresponding BF16 results. Specifically, we first compute the average scores of the quantized and BF16 models across the five benchmarks, and then calculate the ratio between these two averages:

$ \text{Retention} = \frac{\text{Average}{\text{Quantized}}} {\text{Average}{\text{BF16}}} \times 100%. $

Thus, the reported performance retention represents the overall average performance preserved by the quantized model relative to its BF16 counterpart under a consistent evaluation protocol.

AuthorsOpenReview ↗

Rebuttal by Authors

W2. Lack of reporting on calibration/search costs.

We thank the reviewer for raising this concern about calibration costs. To clarify the offline quantization overhead of COTQuant, we report the calibration time required by different PTQ methods on Qwen3.5-122B:

MethodCalibration Time
SmoothQuant~10 min
OmniQuant>120 h
DuQuant~1 h
COTQuant (Ours)~5 h

The offline quantization overhead can be substantial for this model because Qwen3.5-122B has a Mixture-of-Experts (MoE) architecture with a large number of linear layers. For example, OmniQuant uses an entire Transformer layer as its optimization unit, minimizes the quantization mean squared error (MSE) of that layer, and jointly updates the per-channel scaling factors of all its linear layers, resulting in particularly high calibration costs. SmoothQuant and DuQuant, in contrast, complete calibration with a single forward pass. COTQuant lies between these approaches: although it also involves optimization, the optimization is performed separately for each linear layer, resulting in smaller computation graphs and substantially lower calibration costs than OmniQuant. Moreover, COTQuant achieves significantly better quantization accuracy than DuQuant on Qwen3.5-122B. We have added this part to Appendix B of the revised paper.


Q2. Concerns regarding the fairness of experiments

We thank the reviewer for the question. We confirm that COTQuant and DuQuant use the same experimental settings on Qwen3.5-122B to ensure a fair comparison.

As described in Appendix A, both methods use the same 128 samples from COCO train2017 for calibration, adopt the same W4A4 quantization configuration, and are evaluated using the same lmms-eval framework. Thus, the calibration data, quantization bit-width, and evaluation protocol are identical for both methods. The performance advantage of COTQuant on this large-scale VLM therefore does not stem from differences in experimental setup, but from its cross-modal orthogonal transformation designed to mitigate multimodal activation outliers.

I thank the authors for the clarifications, which resolve my question on the fairness of the large-scale comparison. However, the headline retention is a ratio of unnormalized averages that still hides task-level regressions against QuaRot, and the rebuttal provides no real inference measurements at the scale emphasized in the abstract. I therefore maintain my original score.

AuthorsOpenReview ↗

Reply to Official Comment by Reviewer ejWb

We sincerely thank the reviewer for their time and constructive feedback throughout the review process. We also appreciate the reviewer's acknowledgment that our clarifications have resolved the concerns regarding the fairness of the large-scale comparison. Regarding the two remaining concerns, we would like to provide further clarification and additional experimental evidence.


1. Regarding the average performance retention and task-level trade-offs

We understand the reviewer's concern that averaging unnormalized benchmark scores may obscure task-specific performance regressions. To further examine whether our overall performance advantage depends on the use of unnormalized averages, we additionally calculate the BF16-normalized average retention, where each benchmark score is first normalized by its corresponding BF16 score before averaging across the five benchmarks. Under this alternative metric, QuaRot and COTQuant achieve retention rates of 97.73% and 98.04%, respectively, on Qwen2-VL-7B. Thus, COTQuant still maintains an overall advantage even when benchmark-wise normalization is applied.

These results indicate that the overall performance advantage of COTQuant is not solely attributable to the unnormalized averaging scheme. We emphasize that the average retention is intended to summarize the preservation of overall multimodal capabilities rather than imply superiority on every individual task.


2. Regarding real inference efficiency on Qwen3.5-122B

We appreciate the reviewer's concern regarding the lack of real inference measurements at the 122B scale. Our previous submission did not report W4A4 inference efficiency on Qwen3.5-122B primarily due to the hardware limitations of the Kunlun P800 accelerator used in our experiments.

Specifically, the Kunlun P800 does not support native INT4 computation at the hardware level. INT4 quantization is implemented through fake quantization, making it impossible to directly measure the practical inference acceleration of W4A4 quantization on this platform. Furthermore, the P800 currently lacks optimized Fast Hadamard Transform (FHT) kernels, introducing additional computational overhead for methods involving Hadamard transforms.

Nevertheless, to provide additional evidence of practical quantized inference efficiency at the 122B scale, we conduct inference experiments on Qwen3.5-122B using the native INT8 inference kernels provided by vLLM-Kunlun. All methods are evaluated under the same hardware environment and inference settings. Prefill throughput is measured at a sequence length of 2048, while decode throughput is measured with a batch size of 8. The results are presented below:

MethodPrefill (tok/s) ↑Prefill Speedup ↑Decode (tok/s) ↑Decode Speedup ↑Weight (GiB) ↓Peak Memory (GiB) ↓
BF1699921.00×195.91.00×233.0247.6
SmoothQuant117631.18×288.11.47×121.0135.2
DuQuant106621.07×260.61.33×121.0135.2
COTQuant110571.11×228.91.17×121.0135.2

The results demonstrate that INT8 quantized inference provides substantial memory savings and practical inference acceleration on Qwen3.5-122B. Specifically, COTQuant achieves prefill and decode throughputs of 11057 and 228.9 tokens/s, corresponding to 1.11× and 1.17× speedups over BF16, respectively. Meanwhile, model-weight memory is reduced from 233.0 GiB to 121.0 GiB, and peak memory usage decreases from 247.6 GiB to 135.2 GiB, corresponding to approximately 1.93× and 1.83× memory compression, respectively.

We hope these additional results and clarifications help address the remaining concerns regarding evaluation metrics and practical inference efficiency at the 122B scale. We sincerely appreciate the reviewer's careful assessment and constructive feedback.

Reviewer_gyxvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes COTQuant, a post-training quantization framework for vision-language models (VLMs). It consists of two components: Cross-Modal Transform (CMT), which uses modality-aware channel scheduling, a shared orthogonal transform, and block-wise Hadamard transformation to address activation outliers across modalities; and Quantization-Aware Group Search (QAS), which optimizes weight channel grouping based on group-wise quantization error. Experiments on three Qwen-family models and five multimodal benchmarks demonstrate the effectiveness of the proposed approach under low-bit quantization settings.

Critical strengths
  1. The paper addresses a practical challenge in low-bit VLM quantization. Combining cross-modal activation transformation with quantization-aware weight grouping provides a coherent framework for improving quantization performance.

  2. The paper evaluates the method across multiple model scales and multimodal benchmarks, with component-wise ablations and inference-efficiency measurements. These experiments provide encouraging evidence of the method's effectiveness.

  3. The proposed framework integrates activation-side transformations with weight-side grouping optimization, addressing two complementary sources of quantization error. This design provides a systematic approach to improving low-bit quantization for VLMs without requiring model retraining.

Critical weaknesses
  1. Limited comparison with recent quantization methods. The main experiments primarily compare COTQuant with established PTQ baselines, while comparisons with recent rotation-based and VLM-specific quantization methods remain limited. Comparisons under comparable quantization settings would better establish the competitiveness of the proposed approach.

  2. Limited evidence of cross-architecture generalization. Although the method is evaluated on models of different sizes, all three backbones belong to the Qwen family. Evaluations on VLMs from other model families would help determine whether the method generalizes beyond the architectures studied.

  3. Insufficient validation of the specific modality-aware channel permutation strategy. The ablation study shows an improvement when the proposed permutation is introduced, but does not compare it with alternative strategies, such as random permutation or modality-agnostic scheduling. Therefore, the contribution of the specific permutation design remains unclear.

Questions for authors
  1. Could the authors compare COTQuant with recent rotation-based quantization methods published in 2026, particularly PeRQ (ICML 2026) and ReSpinQuant (ICML 2026), under comparable model and quantization settings where feasible? Alternatively, could the authors clarify the technical differences between these methods and COTQuant?

  2. Could the authors evaluate the proposed method on VLMs from other model families to assess its cross-architecture generalization?

  3. Could the authors provide additional ablations or theoretical analysis to clarify the contribution of the proposed modality-aware channel permutation compared with alternative permutation strategies?

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

W1&Q1. Limited comparison with recent quantization methods.

We thank the reviewer for the valuable suggestion. To address this concern, we conducted additional W4A4 experiments with PeRQ on Qwen2-VL-7B and Qwen3-VL-8B and evaluated the methods across five multimodal benchmarks. The results are shown below:

ModelMethodBitsOCRBenchTextVQAVizWizMMEScienceQAAverage
Qwen2-VL-7BPeRQW4A476.280.0667.3986.9882.5578.64
COTQuant (Ours)W4A476.680.5167.7186.9082.7078.88
Qwen3-VL-8BPeRQW4A483.879.8768.2385.3890.0381.46
COTQuant (Ours)W4A485.580.4969.8586.0691.9982.78

Under the same W4A4 quantization setting, COTQuant outperforms PeRQ on both models, demonstrating its advantage over this recent rotation-based quantization method. Regarding ReSpinQuant, we examined its paper and publicly available resources but could not find an official code release. Therefore, we do not report reproduced ReSpinQuant results at this stage, to avoid an inaccurate or unfair comparison. We have added these results to Table 1 of the revised paper.

Although PeRQ, ReSpinQuant, and COTQuant all employ orthogonal transformations to mitigate activation outliers under low-bit quantization, they differ in their objectives and designs. PeRQ primarily investigates the outlier-suppression capability of block-wise Hadamard rotations and proposes the MassDiff algorithm, which optimizes channel permutation by balancing activation L1 mass across blocks. In contrast, COTQuant learns an orthogonal matrix QQ to explicitly redistribute outlier energy across blocks in advance, while accounting for the multimodal activation structure. ReSpinQuant primarily investigates the trade-off between accuracy gains from layer-wise rotations and inference efficiency. It learns separate orthogonal transformations for different layers and employs Subspace Residual Rotation Approximation to reduce the alignment overhead between different rotation bases in residual connections, with a stronger emphasis on inference efficiency. By contrast, COTQuant focuses on suppressing multimodal activation outliers. The methods therefore differ in the primary motivation behind their rotation optimization.


W2&Q2. Limited evidence of cross-architecture generalization.

We thank the reviewer for the valuable suggestion. To address the concern that our experiments lack evidence of cross-architecture generalization, we conduct additional experiments on InternVL2. We compare COTQuant with representative PTQ methods, including SmoothQuant, OmniQuant, MBQ, MQuant, DuQuant, QuaRot, and PeRQ, following the same experimental settings as those used for the Qwen models. The results are presented below:

MethodBitsOCRBenchTextVQAVizWizMMEScienceQAAverage ↑
BF16BF1676.276.8160.4485.9796.9779.28
SmoothQuantW4A872.774.5058.6684.7196.4377.40
OmniQuantW4A451.754.4240.7177.9775.9560.15
MBQW4A872.072.9256.6283.9996.4376.39
MQuantW4A472.175.5859.1384.6394.9977.29
DuQuantW4A474.175.0359.9484.6795.9377.93
QuaRotW4A474.773.7558.4384.5894.6577.22
PeRQW4A474.175.6259.8085.4395.6478.12
COTQuantW4A474.175.6560.2685.4396.4878.38
COTQuant(G128)W4A474.475.8762.3185.2696.7878.92

The results demonstrate that COTQuant consistently achieves stronger overall performance than existing W4A4 baselines on InternVL2. Specifically, COTQuant achieves an average score of 78.38, outperforming PeRQ (78.12), the strongest competing W4A4 baseline, by 0.26 points. With the addition of Quantization-Aware Group Search (QAS), COTQuant(G128) further improves the average score to 78.92, exceeding PeRQ by 0.80 points and DuQuant (77.93) by 0.99 points. Moreover, COTQuant(G128) is only 0.36 points below the BF16 baseline (79.28), retaining approximately 99.55% of the full-precision performance.

These additional results further demonstrate that the effectiveness of COTQuant is not limited to the Qwen family. Despite differences in model architectures and activation distributions across VLM families, COTQuant effectively suppresses multimodal activation outliers through Cross-Modal Transform (CMT) and further reduces group-wise weight quantization error through Quantization-Aware Group Search (QAS). Therefore, the results on InternVL2 provide additional empirical evidence supporting the cross-architecture generalization capability of our proposed framework. We have added this part to Appendix D of the revised paper.

AuthorsOpenReview ↗

Rebuttal by Authors

W3&Q3. Insufficient validation of the specific modality-aware channel permutation strategy.

We thank the reviewer for the suggestion. To further validate the effectiveness of the proposed Modality-Aware Channel Scheduling in CMT, we conducted an additional ablation study on Qwen3-VL-8B, comparing our strategy with two alternatives:

  • Random Permutation: Randomly permutes the outlier channels after the orthogonal transform together with all normal channels.
  • Modality-Agnostic Scheduling: Performs channel scheduling without distinguishing text and vision modalities or explicitly modeling their different outlier structures; channel permutation is determined solely by the activation ranges of channels.
  • CMT (Ours): Uses our proposed modality-aware channel scheduling, explicitly distinguishing text-specific, vision-specific, and overlapping outlier channels.

All experiments use W4A4 quantization. The results are presented below:

Permutation StrategyOCRBenchTextVQAVizWizMMEScienceQAAverage ↑
Random Permutation84.279.7069.2585.3891.1381.93 (↓2.23)
Modality-Agnostic Scheduling84.380.2968.3485.1791.1381.85 (↓2.31)
CMT (Ours)85.580.4969.8586.0691.9982.78 (↓1.38)

CMT outperforms both alternative permutation strategies across all five benchmarks. These results further support the importance of modality-aware channel scheduling. In VLMs, the outlier-channel distributions of text and vision activations differ yet partially overlap. Random permutation cannot balance how outlier channels are distributed across blocks, whereas modality-agnostic scheduling can leverage outlier-channel information but does not explicitly account for cross-modal differences and sharing relationships among outlier channels. In contrast, CMT schedules distinct outlier channels across different blocks, avoiding the concentration of multiple outliers within the same block and the resulting imbalance in outlier energy across blocks. By identifying overlapping outlier channels, it also avoids redundant handling of shared channels, allowing the subsequent shared orthogonal transform and block-wise Hadamard transformation to redistribute activation outliers more effectively. We have added these experimental results to the ablation study section of the revised paper.

Reviewer_Z1GAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes COTQuant, a cross-modal orthogonal transformation method for low-bit quantization of vision-language models, addressing the distinct yet partially overlapping distributions of textual and visual outlier channels. Its core module, CMT, explicitly distinguishes text-specific, vision-specific, and shared outlier channels, organizes channel blocks according to their overlap structure, and learns a shared orthogonal rotation within the subspace spanned by the union of these outlier channels to reduce activation peaks in both modalities. Experiments unde W4A4 quantization demonstrate strong performance across multiple models and multimodal tasks. The reported improvements in inference speed and memory efficiency further highlight the method’s practical value.

Critical strengths
  1. The paper starts from an interesting observation: textual and visual outlier channels differ but also partially overlap. Using this structure to guide quantization offers a useful angle on the challenges of low-bit VLMquantization.
  2. The experiments cover models of different scales and several multimodal benchmarks, with strong accuracy results under W4A4 quantization. The speed and memory measurements also provide evidence of the method’s practical benefits.
Critical weaknesses
  1. Comparisons with existing multimodal quantization methods are somewhat limited. I would like to see results for MBQ and MQuant, as well as a clearer discussion of how the method differs from SplitQ in identifying and handling outlier channels. These comparisons would help clarify where the gains come from.
  2. It is still unclear how much of the improvement comes specifically from accounting for channel overlap. The current ablations also involve channel reordering, block partitioning, and rotation learning. A direct comparison between overlap-aware and overlap-agnostic variants, with the other settings held fixed, would strengthen the main claim.
Questions for authors
  1. How much of the improvement comes specifically from modeling channel overlap? Could you compare overlap-aware and overlap-agnostic variants while keeping the block size, rotation dimension, and optimization budget fixed?
  2. Could you report speed and peak memory usage for the best-performing G128 configuration, with a matched comparison against QuaRot, to clarify the trade-off between accuracy and inference cost?
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

W1. Comparision with exiting multimodal quantization methods.

We thank the reviewer for the valuable suggestion. To address this concern, we additionally evaluated MBQ and MQuant on Qwen2-VL-7B and Qwen3-VL-8B under the same experimental setup. The results are presented below:

ModelMethodBitsOCRBenchTextVQAVizWizMMEScienceQAAverage
Qwen2-VL-7BMBQW4A870.774.1259.6786.8280.1274.29
MQuantW4A476.479.0567.2086.8680.9178.08
COTQuant (Ours)W4A476.680.5167.7186.9082.7078.88
Qwen3-VL-8BMBQW4A880.977.0458.8582.0188.8077.52
MQuantW4A482.675.2664.6982.6985.5778.16
COTQuant (Ours)W4A485.580.4969.8586.0691.9982.78

MBQ employs per-channel scaling, which has limited ability to suppress activation outliers and can shift the quantization difficulty from activations to weights. As MBQ failed to produce valid outputs under W4A4, we report its W4A8 results instead. The results show that COTQuant outperforms both MBQ and MQuant across all five benchmarks on both models. We have added these results to Table 1 of the revised paper.

The reviewer also requested a clearer distinction between our method and SplitQ. Although both SplitQ and COTQuant focus on differences in the activation distributions of text and vision modalities, they differ in how they model outlier channels and mitigate quantization error.

First, the methods differ in their modeling of modality-specific outlier channels. SplitQ introduces Modality-specific Outlier Channel Decoupling (MOCD), which partitions channels into three mutually exclusive sets: text-specific, vision-specific, and modality-compatible channels. Its channel selection procedure first identifies vision outlier channels and then selects text outlier channels from the remaining channels. In contrast, our key observation is that text and vision outlier channels are not only modality-specific but also exhibit non-negligible overlap. We therefore identify the outlier sets for the two modalities separately and explicitly obtain their overlapping outlier channels through set intersection. Based on this structure, CMT schedules the different types of outlier channels accordingly.

Second, the methods differ in how they reduce quantization error. SplitQ assigns different channel sets to separate quantization paths and further introduces low-rank weight smoothing and an activation compensation branch through Adaptive Cross-Modal Calibration (ACC) to mitigate cross-modal quantization error. In contrast, COTQuant does not rely on an additional low-rank compensation branch. Instead, it directly reshapes the activation distributions through orthogonal transformations that preserve computational equivalence.


W2&Q1. How much of the improvement comes specifically from accounting for channel overlap

We thank the reviewer for this suggestion. To quantify the specific contribution of modeling overlapping outlier channels across modalities, we conducted an additional ablation study on Qwen3-VL-8B, comparing two CMT variants: Overlap-Aware, which explicitly accounts for overlapping outlier channels, and Overlap-Agnostic, which does not model their overlap. The experiments use W4A4 quantization, with results shown below:

Overlap-AwareOCRBenchTextVQAVizWizMMEScienceQAAverage ↑
No83.880.6468.2886.8290.3881.98 (↓2.18)
Yes85.580.4969.8586.0691.9982.78 (↓1.38)

The results show that explicitly modeling overlapping outlier channels improves the average score from 81.98 to 82.78, a 0.80-point gain, and reduces the performance degradation relative to BF16 by approximately 36.7%. This improvement stems from CMT's explicit modeling of cross-modal outlier structures. The outlier channels of text and vision modalities are not entirely disjoint; instead, they partially overlap. By identifying the shared outlier channels (O=At∩AvO=A_t\cap A_v), CMT avoids handling shared channels redundantly during channel scheduling and organizes the distinct outlier channels of both modalities into a unified, compact subspace. This enables the subsequent shared orthogonal transform to redistribute outlier energy more effectively. We have added these experimental results to the ablation study section of the revised paper.

AuthorsOpenReview ↗

Rebuttal by Authors

Q2. Report speed and peak memory usage for G128 configuration

We thank the reviewer for the suggestion. To further assess the trade-off between quantization accuracy and practical inference efficiency, we conducted additional inference-efficiency experiments for COTQuant(G128) on Qwen2-VL-7B and compared it with QuaRot and the original COTQuant. All methods follow the same evaluation setup as Appendix D of our paper, with prefill throughput measured at a sequence length of 2048.

MethodAverage ↑Prefill Throughput ↑Prefill Speedup ↑Decode Throughput ↑Decode Speedup ↑Weight (MiB) ↓Peak Memory (MiB) ↓Compression ↑
BF1680.464597.61.00×53.81.00×15810.015874.01.00×
QuaRot78.6310030.42.18×155.52.89×6994.97123.92.23×
COTQuant78.888202.51.78×150.72.80×6994.97170.12.21×
COTQuant(G128)79.447700.81.67×148.22.75×7174.07576.72.09×

The results show that COTQuant(G128) achieves the best quantization accuracy, with an average score of 79.44, outperforming QuaRot and the original COTQuant by 0.81 and 0.56 points, respectively. Meanwhile, COTQuant(G128) achieves a peak memory usage of 7576.7 MiB, corresponding to a 2.09× memory compression ratio compared with BF16, thus preserving the substantial memory advantages of low-bit quantization.

Regarding inference speed, COTQuant(G128) achieves prefill and decode throughputs of 7700.8 and 148.2 tokens/s, respectively, slightly lower than those of QuaRot and the original COTQuant. This additional overhead mainly arises from the dequantization of group-wise INT4 computation results, indicating that G128 group-wise quantization introduces a modest inference efficiency penalty in exchange for improved accuracy under the current implementation.

Comments on the paper

2 comments

Dear reviewers,

It seems that the authors have carefully responded to your reviews.

If you haven't already, please take your time to go over them and let authors know what you think, to your earliest convenience.

Best,
AC.

AuthorsOpenReview ↗

Global Response to Reviewers and Area Chairs

Dear Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs,

We sincerely thank all reviewers for their time, effort, and constructive feedback. During the rebuttal, we addressed the major concerns through additional baseline comparisons, ablation studies, cross-architecture validation, and inference efficiency experiments. We summarize the key results below.


1. Current Review Status

Reviewer Z1GA acknowledged that our MQuant comparison and overlap ablation addressed their main concerns. We also corrected the inconsistency in the G128 inference results identified by the reviewer.

Reviewer ejWb acknowledged that our clarification resolved the concern about experimental fairness at the 122B scale. We further addressed the remaining concerns through BF16-normalized evaluation and real INT8 inference measurements.

Reviewer gyxv raised concerns about recent baselines, cross-architecture generalization, and modality-aware channel scheduling. We provided additional experiments addressing all three concerns and have not yet received further feedback.


2. Summary of Key Issues and Additional Results

1. Comparisons with Recent Quantization Methods (Z1GA W1; gyxv W1 & Q1)

To address the lack of baseline comparisons, we added MBQ, MQuant, and PeRQ experiments on Qwen2-VL-7B and Qwen3-VL-8B. COTQuant outperforms these baselines in average performance, exceeding PeRQ by 0.24 and 1.32 points, respectively. We also clarified the technical differences from SplitQ, PeRQ, and ReSpinQuant, and incorporated the new results into the Main Results section.

2. Additional Ablation Studies (Z1GA W2 & Q1; gyxv W3 & Q3)

To isolate the contributions of overlap modeling and modality-aware scheduling, we conducted two additional ablations on Qwen3-VL-8B. Overlap-aware modeling improves the average score from 81.98 to 82.78 (+0.80), while CMT outperforms random permutation and modality-agnostic scheduling by 0.85 and 0.93 points, respectively. These results validate the effectiveness of explicitly modeling and scheduling heterogeneous and overlapping cross-modal outlier channels.

3. Cross-Architecture Generalization (gyxv W2 & Q2)

To evaluate generalization beyond the Qwen family, we conducted additional experiments on InternVL2 against seven PTQ baselines. COTQuant achieves 78.38, outperforming the strongest competing W4A4 baseline, PeRQ, by 0.20 points, while retaining 98.86% of BF16 performance. These results provide further evidence of COTQuant's effectiveness across different VLM families.

4. Evaluation Metrics and Task-Level Trade-offs (ejWb W1 & Q1)

To address concerns about unnormalized averages obscuring task-level regressions, we clarified the retention calculation and acknowledged individual benchmark trade-offs. Under an alternative BF16-normalized average retention metric, COTQuant still outperforms QuaRot on Qwen2-VL-7B (98.04% vs. 97.73%). This confirms that our overall advantage does not solely depend on unnormalized averaging.

5. G128 Inference Efficiency (Z1GA Q2)

To evaluate the practical overhead of G128, we measured throughput and memory usage on Qwen2-VL-7B. COTQuant(G128) achieves 79.44 average accuracy, 1.67× prefill and 2.75× decode speedups, and 2.09× peak-memory compression over BF16. These results demonstrate improved accuracy while preserving substantial inference and memory efficiency benefits.

6. Calibration Cost, Fairness, and 122B Inference (ejWb W2 & Q2)

To address concerns about large-scale deployment, we reported calibration costs, confirmed experimental fairness, and added real inference measurements. On Qwen3.5-122B, COTQuant requires ~5 hours of calibration versus >120 hours for OmniQuant; COTQuant and DuQuant use identical calibration data, W4A4 settings, and evaluation protocols. Due to the Kunlun P800's lack of native INT4 and optimized FHT kernels, we additionally evaluated real INT8 inference, achieving 1.11× prefill and 1.17× decode speedups, with peak memory reduced from 247.6 to 135.2 GiB; these measurements do not represent native W4A4 performance.


We believe these additional results substantially strengthen the experimental evidence supporting COTQuant. We sincerely appreciate the reviewers' constructive feedback and respectfully ask that these updates be considered in the final evaluation.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.