- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors propose a PTQ framework for MLLMs, named C-PTQ, which incorporates channel-wise Fisher sensitivity into the scaling optimization. Motivated by the mismatch between reconstruction error and task-loss sensitivity, the authors derive a Fisher-weighted reconstruction objective using a diagonal approximation of the loss Hessian. Experiments show its effectiveness.
Critical strengths
- Incorporating task loss sensitivity into the quantization objective is a promising direction.
- C-PTQ integrates Fisher-weighted reconstruction into the existing scaling search without additional trainable parameters. The computational overhead is also relatively small.
- The experiments are extensive and the improvements under W4A6 are encouraging.
Critical weaknesses
- The Fisher importance vector is defined on output channels, while the scaling factors are defined on input channels. Although this formulation is mathematically feasible, the paper does not adequately explain why it resolves the dimensional mismatch emphasized in its motivation.
- The method is motivated by task-specific loss sensitivity, but its Fisher weights are estimated using a small COCO calibration set and the performance is evaluated on 8 different multimodal tasks. The authors does not clearly explain why the estimated channel importance is aligned with these downstream objectives.
- In the ablations, the authors does not sufficiently isolate whether the improvement comes from the Fisher-specific formulation or simply from using squared-gradient-based channel weighting.
Questions for authors
- In W1, how does this formulation directly address the channel-dimensional misalignment highlighted in the motivation? Can you clarify the relationship between these two channel spaces?
- In W2, how well does this generic calibration-based sensitivity correspond to the different downstream tasks considered in the evaluation? What about the performance comparison between using downstream datasets for calibration or using a general calibration set on the corresponding task?
- In W3, can you compare against alternative weighting strategies, such as randomly permuted Fisher scores or other channel-wise gradient statistics?