- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the limitations of existing post-training quantization methods for multimodal large language models, which primarily rely on modality-level statistics and overlook variations in activation distributions across tokens within the same modality. The authors propose Token Routing Attention Quantization (TRAQ), a framework that uses modality-specific smoothing scales as anchors and dynamically constructs token-specific scales through activation-magnitude-based compatibility scores. TRAQ further introduces an error-driven calibration strategy that adaptively balances reconstruction losses between text and vision tokens at each layer to optimize the scale anchors. Experiments across multiple multimodal models, benchmarks, and quantization settings demonstrate that TRAQ improves low-bit quantization performance while maintaining inference efficiency comparable to existing methods.
Critical strengths
- The paper identifies an important limitation of existing modality-aware PTQ methods: tokens within the same modality may exhibit substantially different activation characteristics but are typically assigned the same smoothing transformation. TRAQ addresses this granularity gap by introducing token-specific scale routing over modality-informed anchors. This is a conceptually meaningful extension beyond modality-level adaptation and token-importance-based loss weighting, offering a promising direction for fine-grained multimodal quantization.
- TRAQ demonstrates strong performance across multiple model architectures, parameter scales, quantization configurations, and multimodal benchmarks. In particular, it achieves the best results on all four benchmarks for both main models under W4A8 quantization. The ablation studies further show that token-adaptive routing and error-driven calibration independently contribute to performance improvements. These results provide substantial empirical support for the practical value of the proposed approach.
Critical weaknesses
- This paper suffers from insufficient justification and validation of the core token-routing mechanism. TRAQ determines token-anchor compatibility using the magnitude of normalized activations (Eq. 10), but smaller normalized activation magnitudes do not necessarily imply lower quantization reconstruction errors, particularly after rescaling. The strong correlation shown in Figure 3 is largely expected from the routing formulation itself and does not establish that the selected scales are optimal or even consistently beneficial. Moreover, the paper lacks comparisons with simpler token-adaptive scaling strategies or error-based routing alternatives. Since token-level scale routing is the central methodological contribution, the absence of direct evidence for the effectiveness and necessity of this particular routing mechanism substantially weakens the novelty and technical justification of the approach.
- There are unresolved inconsistencies between the quantization formulation and empirical results. Unlike conventional SmoothQuant, TRAQ's actual inference formulation (Eq. 13) applies token-specific scaling only to activations while keeping quantized weights fixed. When activation quantization is nearly lossless, the scaling and inverse scaling should approximately cancel, making the substantial improvements reported under W4A16 difficult to explain without further implementation details. Additionally, Table 1 reports an unusually low FP16 OCRBench score for InternVL2-8B (0.621), substantially below its W4A8 result (0.777), raising concerns about evaluation consistency or reporting accuracy. These issues are particularly consequential because they affect the interpretation and reliability of the empirical improvements, which constitute the paper's strongest argument for acceptance.
Questions for authors
- In Eq. 13, token-specific scaling is applied to activations while the quantized weights remain fixed. If A16 represents FP16 activations, the scaling and inverse scaling should approximately cancel. How can the authors explain the substantial improvements under W4A16? Please clarify the activation quantization scheme and which quantization parameters are optimized during calibration.
- The proposed compatibility score shown in Eq. 10 is based on normalized activation magnitudes, which do not necessarily correspond to actual quantization reconstruction errors. Can the authors provide direct evidence that this routing criterion selects more suitable scales, preferably through comparisons with simpler token-adaptive scaling strategies or reconstruction-error-based routing alternatives?
- Table 1 reports an unusually low FP16 OCRBench score for InternVL2-8B (0.621), substantially below the corresponding W4A8 result (0.777) and the FP16 results of other InternVL2 variants. Can the authors verify these numbers and confirm that the full-precision and quantized models were evaluated using identical protocols? If there is a reporting or evaluation error, how does correcting it affect the main conclusions?
Dear Reviewer YoJT,
Please respond to the author soon and engage in the discussion.
Best,
AC