acceptodds
Withdrawn from ICLR 2027

C-PTQ: Unified Channel-wise Sensitivity and Scaling for Post-training Quantization of MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) require huge memory and computational costs, which limit their practical deployment. Post-training quantization (PTQ) techniques offer an efficient solution for model compression and inference acceleration. Yet, the quantized model faces performance degradation due to outlier channels, which are highly sensitive to quantization and substantially impair activation fidelity and task accuracy. To protect these salient channels during quantization, existing PTQ methods leverage modality- or token-level metrics to guide channel-wise scaling (CWS) of LLM decoders. However, these orthogonal measurements fail to capture channel-wise impacts on task-specific loss, and the misalignment between importance and scaling factors ultimately leads to suboptimal performance. To address this issue, we propose C-PTQ, a unified channel-wise PTQ method that harmonizes task-specific loss perturbation and quantization error. Motivated by second-order derivatives, we design a Fisher-weighted objective as a tractable Hessian approximation, seamlessly injecting task sensitivity into the scaling process. Notably, we achieve state-of-the-art performance without auxiliary modules like LoRA, thereby maintaining high efficiency. Experiments on Qwen2.5VL, InternVL2 and LLaVA-OV across 8 benchmarks demonstrate our effectiveness in both weight-only and weight-activation settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Post-training Quantization, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vFrLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose a PTQ framework for MLLMs, named C-PTQ, which incorporates channel-wise Fisher sensitivity into the scaling optimization. Motivated by the mismatch between reconstruction error and task-loss sensitivity, the authors derive a Fisher-weighted reconstruction objective using a diagonal approximation of the loss Hessian. Experiments show its effectiveness.

Critical strengths
  1. Incorporating task loss sensitivity into the quantization objective is a promising direction.
  2. C-PTQ integrates Fisher-weighted reconstruction into the existing scaling search without additional trainable parameters. The computational overhead is also relatively small.
  3. The experiments are extensive and the improvements under W4A6 are encouraging.
Critical weaknesses
  1. The Fisher importance vector is defined on output channels, while the scaling factors are defined on input channels. Although this formulation is mathematically feasible, the paper does not adequately explain why it resolves the dimensional mismatch emphasized in its motivation.
  2. The method is motivated by task-specific loss sensitivity, but its Fisher weights are estimated using a small COCO calibration set and the performance is evaluated on 8 different multimodal tasks. The authors does not clearly explain why the estimated channel importance is aligned with these downstream objectives.
  3. In the ablations, the authors does not sufficiently isolate whether the improvement comes from the Fisher-specific formulation or simply from using squared-gradient-based channel weighting.
Questions for authors
  1. In W1, how does this formulation directly address the channel-dimensional misalignment highlighted in the motivation? Can you clarify the relationship between these two channel spaces?
  2. In W2, how well does this generic calibration-based sensitivity correspond to the different downstream tasks considered in the evaluation? What about the performance comparison between using downstream datasets for calibration or using a general calibration set on the corresponding task?
  3. In W3, can you compare against alternative weighting strategies, such as randomly permuted Fisher scores or other channel-wise gradient statistics?
Reviewer_S3tiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

C-PTQ presents a post-training quantization method for multimodal large language models. It uses Fisher-based weights, estimated from squared task-loss gradients, to weight the differences between full-precision and quantized outputs. These weights guide the search for channel scaling factors. The experiments cover several model families, eight benchmarks, and both weight-only and weight–activation quantization.

Critical strengths
  1. A simple approach with limited calibration cost. The method brings task sensitivity into the scaling search without adding learned modules or requiring iterative parameter training. This could be useful when both accuracy and calibration time matter.

  2. Clear gains in a difficult quantization setting. On Qwen2.5VL-7B under W4A6, the reported average score increases from the strongest listed baseline’s 69.3 to 72.4. This is stronger evidence of practical value than the small gains reported in some weight-only settings.

Critical weaknesses
  1. The paper does not establish what is new relative to BRECQ. BRECQ already uses diagonal Fisher information and squared gradients to weight intermediate-output reconstruction errors. Yet this paper neither cites BRECQ nor explains the relationship between the two methods, while presenting its Fisher-weighted objective as a main contribution. This omission leaves a central novelty claim unsupported. The authors need to explain the differences from BRECQ, identify the specific methodological innovations, and provide their theoretical basis.

  2. The experiments omit the comparisons most relevant to the claimed contribution. There is no comparison between BRECQ-style element-wise Fisher weighting and the proposed method within the same framework. Later Fisher/BRECQ-related quantization methods, such as FIMA-Q, are also missing from the discussion and comparisons.

Questions for authors

Please address the points raised in Critical Weaknesses 1 and 2.

Reviewer_bSnJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes C-PTQ, which uses channel-averaged squared gradients to weight reconstruction errors when selecting quantization scales for MLLMs. The method is simple and evaluates both weight-only and weight–activation quantization across three model families and eight benchmarks.

Critical strengths

The main strengths are the broad evaluation and low calibration overhead. The W4A6 results are particularly encouraging: on Qwen2.5VL-7B, C-PTQ achieves an average score of 72.4 versus 69.3 for the strongest reported baseline. The calibration-data and diagonal-versus-dense Fisher ablations also provide useful support for the practical design.

Critical weaknesses

The novelty needs clearer positioning. Fisher-weighted output reconstruction already appears in BRECQ, §3.3, and PTQ4ViT, §3.3 uses squared-gradient weighting to search quantization step sizes. C-PTQ’s channel aggregation and integration with AWQ-style rescaling are distinct choices, but their contribution should be isolated through a comparison against elementwise squared-gradient weighting under the same search setup.

The claimed channel alignment is unclear. Algorithm 1 defines sensitivity weights over output channels, while the scaling factors operate on input channels. This does not make the objective invalid, but it does require a more precise explanation of what is being aligned. Similarly, the pooled squared SFT gradients are a curvature surrogate; the dense-Fisher comparison alone does not establish that they accurately approximate the task-loss Hessian.

An experimental inconsistency needs resolution. For LLaVA-OV-7B under W4A6, Figure 3 reports MMMU/AI2D scores of 44.7/77.9, whereas Table 2 reports 40.6/75.7. Please explain whether the settings differ or correct the results, since this figure supports the choice of Fisher weighting.

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.