acceptodds
Under review as a conference paper at ICLR 2027

Beyond Modality Importance: Token Routing Attention for Multimodal LLM Quantization

Abstract

Post-training quantization (PTQ) has become a key technique for reducing the storage and inference costs of multimodal large language models (MLLMs). Existing modality-aware PTQ methods learn independent activation smoothing scales for text and vision modalities, yet still rely on modality labels as the unit of quantization decisions, overlooking fine-grained variations in activation magnitudes across tokens within the same modality. In this paper, we propose Token Routing Attention Quantization (TRAQ), a PTQ framework that treats modality-specific activation smoothing scales as anchors and performs token-level scale routing through compatibility-based attention. Our core insight is that the optimal activation smoothing scale depends not only on global modality activation distributions but also on token-specific activation magnitudes. To this end, TRAQ treats modality-specific activation smoothing scales as prior anchors and employs a token-level attention mechanism that evaluates the compatibility between each token and each anchor, thereby assigning each token an activation smoothing scale aligned with its own activation magnitudes. On this basis, TRAQ computes layer-wise text and vision reconstruction errors and converts them into dynamic weights for the modality-specific reconstruction losses. The resulting weighted objective assigns greater emphasis to the modality with larger quantization error and is minimized to update the scale anchors. Extensive experiments show that TRAQ consistently outperforms competing PTQ methods, with gains of up to 7.70 points, while introducing no additional learnable modules and maintaining comparable inference efficiency to existing modality-aware PTQ methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Post-Training Quantization, Token Routing Attention, Token-Adaptive Quantization, Activation Smoothing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hmTEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the limitations of existing post-training quantization methods for multimodal large language models, which primarily rely on modality-level statistics and overlook variations in activation distributions across tokens within the same modality. The authors propose Token Routing Attention Quantization (TRAQ), a framework that uses modality-specific smoothing scales as anchors and dynamically constructs token-specific scales through activation-magnitude-based compatibility scores. TRAQ further introduces an error-driven calibration strategy that adaptively balances reconstruction losses between text and vision tokens at each layer to optimize the scale anchors. Experiments across multiple multimodal models, benchmarks, and quantization settings demonstrate that TRAQ improves low-bit quantization performance while maintaining inference efficiency comparable to existing methods.

Critical strengths
  1. The paper identifies an important limitation of existing modality-aware PTQ methods: tokens within the same modality may exhibit substantially different activation characteristics but are typically assigned the same smoothing transformation. TRAQ addresses this granularity gap by introducing token-specific scale routing over modality-informed anchors. This is a conceptually meaningful extension beyond modality-level adaptation and token-importance-based loss weighting, offering a promising direction for fine-grained multimodal quantization.
  2. TRAQ demonstrates strong performance across multiple model architectures, parameter scales, quantization configurations, and multimodal benchmarks. In particular, it achieves the best results on all four benchmarks for both main models under W4A8 quantization. The ablation studies further show that token-adaptive routing and error-driven calibration independently contribute to performance improvements. These results provide substantial empirical support for the practical value of the proposed approach.
Critical weaknesses
  1. This paper suffers from insufficient justification and validation of the core token-routing mechanism. TRAQ determines token-anchor compatibility using the magnitude of normalized activations (Eq. 10), but smaller normalized activation magnitudes do not necessarily imply lower quantization reconstruction errors, particularly after rescaling. The strong correlation shown in Figure 3 is largely expected from the routing formulation itself and does not establish that the selected scales are optimal or even consistently beneficial. Moreover, the paper lacks comparisons with simpler token-adaptive scaling strategies or error-based routing alternatives. Since token-level scale routing is the central methodological contribution, the absence of direct evidence for the effectiveness and necessity of this particular routing mechanism substantially weakens the novelty and technical justification of the approach.
  2. There are unresolved inconsistencies between the quantization formulation and empirical results. Unlike conventional SmoothQuant, TRAQ's actual inference formulation (Eq. 13) applies token-specific scaling only to activations while keeping quantized weights fixed. When activation quantization is nearly lossless, the scaling and inverse scaling should approximately cancel, making the substantial improvements reported under W4A16 difficult to explain without further implementation details. Additionally, Table 1 reports an unusually low FP16 OCRBench score for InternVL2-8B (0.621), substantially below its W4A8 result (0.777), raising concerns about evaluation consistency or reporting accuracy. These issues are particularly consequential because they affect the interpretation and reliability of the empirical improvements, which constitute the paper's strongest argument for acceptance.
Questions for authors
  1. In Eq. 13, token-specific scaling is applied to activations while the quantized weights remain fixed. If A16 represents FP16 activations, the scaling and inverse scaling should approximately cancel. How can the authors explain the substantial improvements under W4A16? Please clarify the activation quantization scheme and which quantization parameters are optimized during calibration.
  2. The proposed compatibility score shown in Eq. 10 is based on normalized activation magnitudes, which do not necessarily correspond to actual quantization reconstruction errors. Can the authors provide direct evidence that this routing criterion selects more suitable scales, preferably through comparisons with simpler token-adaptive scaling strategies or reconstruction-error-based routing alternatives?
  3. Table 1 reports an unusually low FP16 OCRBench score for InternVL2-8B (0.621), substantially below the corresponding W4A8 result (0.777) and the FP16 results of other InternVL2 variants. Can the authors verify these numbers and confirm that the full-precision and quantized models were evaluated using identical protocols? If there is a reporting or evaluation error, how does correcting it affect the main conclusions?
1 comment

Dear Reviewer YoJT,

Please respond to the author soon and engage in the discussion.

Best,

AC

Reviewer_ZotPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Token Routing Attention Quantization (TRAQ) for post-training quantization of multimodal large language models. Instead of assigning smoothing scales solely according to modality labels, TRAQ constructs token-specific scales by combining text and vision scale anchors using activation-dependent compatibility scores. It additionally weights modality-specific reconstruction losses according to their current errors during layer-wise calibration. The paper evaluates several weight–activation precision configurations across Qwen2.5-VL, InternVL2, and LLaVA-OneVision models, reporting improvements over competing PTQ methods, component ablations, and inference efficiency measurements.

Critical strengths

1.Relevant problem and a compact approach. The paper addresses a meaningful limitation of modality-level quantization: tokens within the same modality can have different activation characteristics. Interpolating between a small number of scale anchors is a potentially useful way to introduce token-level adaptation without adding expert networks.

2.Broad empirical coverage and useful component ablations. The evaluation includes multiple model families and scales, with additional results extending from 1B to 26B parameters in Appendix A. Table 2 separately evaluates routing and error-driven calibration, providing initial evidence that both contribute. Some reported gains are substantial: for example, Qwen2.5-VL-7B W4A8 OCR performance increases from the strongest listed baseline’s 0.754 to 0.802. If verified under matched implementations and evaluation settings, these improvements would be practically relevant.

Critical weaknesses
  1. The source of the A16 gains is unclear. If A16 retains floating-point activations, the scaling and restoration in Equation (13) approximately cancel for identical quantized weights. Yet the headline 7.70-point gain occurs under W4A16. This gain could arise from calibrated weight quantization parameters or other components rather than routing itself. Please clarify the A16 computation and optimized parameters; a routing ablation with identical quantized weights would help identify the contribution.

  2. The routing rationale and deployment details need stronger support. Smaller normalized activation magnitudes do not necessarily imply lower reconstruction error after scale restoration. Appendix D acknowledges that the compatibility score is a proxy, but Figures 2–3 do not directly validate its relationship to reconstruction error. Appendix I provides a plausible fused-dequantization scheme, yet the concrete kernels, arithmetic datatypes, and incremental rescaling cost underlying Table 3 remain unspecified, limiting reproducibility and assessment of routing overhead.

Questions for authors
  1. What computation and calibration parameters are used under A16? Since scaling approximately cancels for identical quantized weights when activations remain floating-point, what accounts for the reported gains?

  2. Which kernels and arithmetic datatypes produced Table 3? Were timing boundaries matched across methods, and which stages were included?

  3. Beyond Table 2’s task-level ablations, was the compatibility score’s relationship to output reconstruction error after scale restoration evaluated?

  4. Is the InternVL2-8B FP16 OCR score of 0.621 correct? Were its evaluation settings matched to the W4A16 methods scoring 0.715–0.781, and what might explain this difference?

Reviewer_YoJTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TRAQ learns channel scales for text and vision, then mixes the two sets of scales for each token using activation-based scores. Reconstruction errors determine the modality weights in calibration. At inference, activations are scaled, quantized, rescaled, and multiplied by one shared, fixed quantized weight matrix. Experiments cover three multimodal model families and several precisions.

Critical strengths
  1. Moving from modality-level to dynamic token-wise scales offers some methodological novelty. Prior work includes modality-specific scales in MASQuant and token-dependent error correction in QuantExperts. TRAQ instead mixes two modality-scale anchors for each token, allowing more flexible scale selection through a compact representation without adding multiple low-rank experts.
  2. W4A8 results and model coverage are encouraging. Compared with QuantExperts, Qwen2.5-VL-7B OCRBench improves from 75.4% to 80.2%, and InternVL2-8B MMMU from 45.17% to 47.44% in Table 1. Appendix A also tests InternVL2-1B/2B/26B and LLaVA-OneVision-7B.
  3. Both component ablations and actual timings are reported. Table 2 compares unified scales, modality anchors, routing alone, error-driven calibration alone, and the full method across two models, two precisions, and four tasks. Results support complementary effects. Table 3 also reports end-to-end latency and memory. The remaining concerns focus on the routing rule's rationale and the underlying arithmetic.
Critical weaknesses
  1. The source of the main A16 gain is unclear. Equation (2) quantizes scaled weights W⊙s. Equation (13) instead uses fixed QW(W) and restores s after activation quantization. If A16 retains floating-point activations, scaling and rescaling cancel for fixed weights, so routing should not change the output. Yet the largest gain is 7.70 points on Qwen W4A16 OCRBench, at 81.2% versus 73.5%. Appendix H allows updates to unspecified quantization parameters, but does not define A16, explain how the weight grid is calibrated, or identify which updates produce the gain. It is therefore unclear how much improvement comes from routing rather than other calibration changes.
  2. A better score need not mean lower quantization error. Equation (10) uses mean normalized activation magnitude. Under common dynamic per-token quantization, multiplying an anchor by c divides the cost by c, while the decoded activation can remain unchanged. The score can therefore improve without any change in actual error. Appendix D calls it a proxy, and Figure 3 shows dependence on activation magnitude. Agreement with true errors and comparisons with simple interpolation rules are still needed to justify this score.
  3. Speed is measured, but efficient low-bit implementation still needs clarification. Appendix I folds token/channel scales into dequantization and passes floating-point tiles to LINEARTILE. This avoids an extra global floating-point tensor, but leaves the MMA precision, quantization granularity, and agreement with calibration unspecified. The measured 14.2 ms prefill latency in Table 3 does not establish whether this dynamic computation retains the efficiency of low-bit matrix multiplication. Also, InternVL2-8B FP16 OCRBench is 62.1%, while all W4A16 methods reach 71.5%–78.1%. Checkpoints, image preprocessing, and evaluation settings need checking (Table 1).
Questions for authors
  1. What does routing change at A16? Please define A16, weight and activation granularities, and the variables updated during calibration. Please also explain the difference between Eqs. (2) and (13). Can routing be toggled with the same calibrated weights, then dynamic loss weighting and weight-quantizer updates checked separately to locate the gain?
  2. Does the score rank actual errors correctly? For the same tokens, can the ranking in Eq. (10) be compared with candidate-anchor reconstruction errors? Please compare against fixed averaged anchors and a score that removes overall scale effects under the same budget. This would show whether the current routing rule is needed.
  3. Do accuracy and speed tests use the same arithmetic? Please specify the actual kernel, MMA precision, and quantization settings in Table 3, and compare deployed outputs with Eq. (13). Please also check InternVL2-8B FP16 and quantized OCRBench scores in one evaluation pipeline. These details directly affect the main conclusions.
4 comments

Dear Reviewer,

As the rebuttal discussion period is approaching its end, we would sincerely appreciate it if you could take a moment to review our detailed rebuttal responses and the updated manuscript.

We have carefully addressed your concerns, conducted additional experiments, and made corresponding revisions to the manuscript. In particular, we have further clarified the methodological formulation, provided additional experimental evidence, and refined the scope of our inference-efficiency claims. All modified and newly added content is highlighted in blue for ease of reference.

We believe these additional clarifications and experimental results substantially address the concerns raised in your review. We would therefore be very grateful if you could reconsider your original rating in light of our responses and the revised manuscript.

If any questions or concerns remain, we would be happy to provide further clarification before the discussion period closes.

Thanks.

Dear Reviewer YoJT,

Please respond to the author soon and engage in the discussion.

Best,

AC

Thank you to the author for the detailed reply and supplementary experiments. The calculation method of A16, the validity of the Routing Score, and the issues with the experimental data were explained well. However, I still have the following concerns:

  1. The practical value of W4A16: The practical value of W4A16: This approach requires storing two sets of INT4 weights and performing two linear operations, bringing the weight storage overhead close to that of W8A16. Given this, why not simply use W8A16? The author needs to explain the advantages of dual W4A16 in terms of accuracy, memory, and inference efficiency compared to a single W8A16.

  2. Differences Between the Two Quantization Mechanisms: W4A8 employs "Token-adaptive Activation Quantization," whereas W4A16 utilizes an output fusion strategy based on dual weight grids. Given the significant differences in their computational mechanisms and deployment costs, their respective scopes of application should be further clarified.

  3. Kernel Implementation for W4A8: The smoothing scale in Eq. 13 has a granularity of (Token × Input Channel), which cannot be directly factored out of the K-dimension accumulation within a GEMM operation. While the authors claim to use INT8 MMA (m16n8k32), they have not yet explained how the scales for different channels are handled within the K-dimension. It is recommended to clarify the actual quantization granularity and the kernel computation method, and to verify that the deployment output is numerically consistent with Eq. 13.

Overall, while the additional experiments strengthen the paper’s persuasiveness, questions remain regarding resource fairness and the practical computational implementation of W4A8. Consequently, I am maintaining my current score for the time being.

Dear Reviewer,

Thank you for your further comments. We appreciate your careful examination of the practical scope of W4A16, the differences between the two quantization mechanisms, and the W4A8 kernel implementation. We would like to clarify these points as follows.

1. Practical Value and Role of W4A16.

We would like to clarify that W4A16 is included primarily to examine whether TRAQ's token-adaptive routing principle can also be extended to weight-only quantization, rather than to propose dual-grid W4A16 as a more efficient alternative to W8A16. Our primary focus is weight-activation joint quantization, particularly challenging low-bit settings such as W4A8, where both weight and activation quantization errors need to be addressed. Under W4A16, we adopt two modality-specific weight grids to preserve modality-dependent quantization representations and enable token-dependent routing. We acknowledge that this design introduces additional computation and does not provide a storage advantage over a single W8A16 weight grid. We therefore do not claim that dual-grid W4A16 is superior to W8A16 in terms of deployment efficiency. Instead, W4A16 serves as an additional validation of TRAQ's applicability to weight-only quantization.

2. Differences and Applicability of the Two Quantization Mechanisms.

We have further clarified that the two configurations use different implementations of the same token-adaptive routing principle. Under weight-activation quantization, TRAQ aggregates modality-specific activation smoothing scales and performs a single linear transformation using shared quantized weights. Under W4A16, activations remain in FP16, and TRAQ instead employs two modality-specific quantized weight grids with token-dependent output fusion based on weight-quantization errors. Thus, the activation-scale routing formulation is applicable to activation-quantized settings more generally, whereas the A16 formulation is a separate extension for weight-only quantization. We have explicitly clarified these scopes in the Abstract, Section 3.3, Conclusion, and Appendices H and I.

3. W4A8 Kernel Implementation.

We agree that channel-dependent effective activation scales cannot be applied after a complete K-dimension accumulation. We have therefore clarified the tiled implementation described in Appendix I.

For each token tile II and input-channel tile JJ, the kernel reconstructs the corresponding scale tile SI,J=∑kαI,ksk,J∗S_{I,J}=\sum_k\alpha_{I,k}s^*_{k,J}, computes the normalized activation tile ZI,J=HI,J⊘SI,JZ_{I,J}=H_{I,J}\oslash S_{I,J}, and quantizes it to obtain the integer activation codes and their quantization scales.

The effective dequantization scale is then computed element-wise as Δeff,I,JA=ΔI,JA⊙SI,J\Delta^A_{\mathrm{eff},I,J}=\Delta^A_{I,J}\odot S_{I,J}.

The dequantized activation tile is formed as AregI,J=cast⁡(qAI,J)⊙Δeff,I,JAA^{\mathrm{reg}}{I,J}=\operatorname{cast}(q^A{I,J})\odot\Delta^A_{\mathrm{eff},I,J}, and is kept in registers or other on-chip storage before being consumed by the corresponding weight tile in the linear computation.

Thus, the channel-dependent scale is applied before the matrix-product reduction, rather than after the complete K-dimension accumulation. This avoids materializing an additional full-precision activation tensor while preserving the computation in Eq. (13).

We have further clarified the numerical consistency between this tiled computation and Eq. (13) . We will also ensure that the description of the MMA datatype in the efficiency analysis is consistent with the actual linear backend.

We sincerely appreciate your comments, which have helped us clarify both the intended scope of TRAQ and the practical implementation of its activation-quantized path. All corresponding modifications are highlighted in blue in the revised manuscript.

Thank you again for your careful evaluation and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.