acceptodds
Under review as a conference paper at ICLR 2027

TRPQuant: Token Rearrangement and Channel Pruning for Quantizing Large Vision-Language Models

Abstract

High memory and computation costs prevent the practical edge deployment of vision-language models (VLMs). Post-training quantization (PTQ) offers a promising solution without expensive retraining. Given the visual and textual inputs in VLMs, existing methods typically adopt modality-level token partitioning and processing. However, they treat tokens uniformly within each modality, overlooking intra-modality activation distribution differences. This coarse-grained strategy leads to degraded accuracy and redundant computation under low-bit quantization. We propose TRPQuant, a new framework that addresses these limitations through Token Rearrangement and Channel Pruning, enabling fine-grained token grouping and computation reduction. Specifically, TRPQuant classifies tokens into Dense, Intermediate, and Sparse groups based on their activation patterns and applies tailored quantization strategies accordingly: dense tokens preserve full channels and go through the standard branch, sparse tokens are routed to the sparse branch with channel pruning to reduce computation, and intermediate tokens combine both mechanisms via dual-branch processing. Extensive experiments on mainstream VLMs demonstrate that TRPQuant achieves strong quantization accuracy. Notably, TRPQuant preserves 94.1% of FP16 performance under the challenging W3A3 setting (69.9 vs. 74.3) on Qwen2.5-VL-7B, pushing the frontier for ultra-low-bit quantization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Post-Training Quantization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sEjxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses accuracy degradation and computational redundancy in low-bit post-training quantization of VLMs, arising from heterogeneous activation distributions. The authors propose TRPQuant, a framework that moves beyond conventional modality-level token partitioning by dynamically categorizing tokens into Dense, Intermediate, and Sparse groups based on their channel-wise activation energy distributions. Different quantization strategies are applied to each group: dense tokens retain full-channel computation, sparse tokens undergo channel pruning to reduce computational costs, and intermediate tokens are processed through both branches. The framework further employs branch-specific quantization transformations to improve low-bit accuracy. Extensive experiments on Qwen2.5-VL, Qwen3-VL, and LLaVA demonstrate strong performance under W4A4 and W3A3 quantization, substantially outperforming existing baselines in ultra-low-bit settings while reducing computational overhead.

Critical strengths
  1. The paper identifies meaningful intra-modality heterogeneity in VLM activation distributions and proposes a principled approach to grouping tokens based on their channel-wise energy concentration. By combining token-specific routing with branch-specific quantization and selective channel pruning, TRPQuant addresses both quantization accuracy and computational redundancy within a unified framework. This is a key strength because it provides a clear motivation for moving beyond conventional modality-aware quantization, supported by activation analyses and component ablations.
  2. TRPQuant demonstrates substantial improvements over the evaluated baselines, particularly under W3A3 quantization, across six VLMs from three model families. The ablation studies further support the effectiveness of token rearrangement and show that channel pruning reduces computational costs with minimal accuracy degradation. Importantly, the paper also reports actual GPU inference measurements rather than relying solely on theoretical Bit-FLOPs. These results provide compelling evidence of the framework's practical potential and constitute a strong empirical argument for acceptance.
Critical weaknesses
  1. Although the paper discusses several closely related approaches, including QIG, SplitQ, and Quant Experts, these methods are absent from the experimental comparisons. This is particularly concerning because some of them also exploit token-level characteristics or adaptive quantization strategies, making them directly relevant to TRPQuant's claimed contributions. Without comparisons against these methods, it is difficult to establish whether the proposed framework represents a meaningful advance over the current state of the art, despite its strong results against the selected baselines. This limitation directly affects the assessment of both novelty and empirical significance.
  2. Although the paper reports substantial theoretical Bit-FLOPs reductions and GPU acceleration, the measured model-level prefill speedup over FlatQuant is relatively modest. For example, on Qwen2.5-VL-7B, TRPQuant improves prefill latency from 84.23 ms (FlatQuant) to 73.72 ms, corresponding to only a 1.14× speedup, while slightly increasing GPU memory consumption. Furthermore, although the paper includes single-kernel benchmarks, its model-level latency evaluation is limited to W4A4 prefill on an RTX 4090, without examining autoregressive decoding, varying batch sizes, or resource-constrained hardware. Since computational efficiency and edge deployment are central motivations of the paper, these limitations weaken the practical significance of the proposed channel pruning and token routing mechanisms.
Questions for authors
  1. The paper discusses recent VLM quantization methods such as QIG, SplitQ, and Quant Experts but does not include them in the experiments. Could the authors provide direct comparisons with these methods under consistent quantization settings, particularly W3A3? If such comparisons are infeasible, please clarify the reasons and explain what advantages TRPQuant offers over these closely related approaches.
  2. While TRPQuant substantially reduces theoretical Bit-FLOPs, its measured prefill speedup over FlatQuant is relatively modest (1.14×). Could the authors provide a breakdown of the computational overhead introduced by token classification, rearrangement, and dual-branch processing? Additionally, how does TRPQuant perform during autoregressive decoding, where the small number of tokens processed per step may limit the benefits of token rearrangement and channel pruning?
  3. TRPQuant relies on a fixed layer-wise active-channel set identified from calibration data. While the paper demonstrates stability across different calibration set sizes, it remains unclear whether this set generalizes to unseen activation distributions. Could the authors evaluate the method under calibration–evaluation domain shifts or analyze how consistently the identified active channels capture the dominant activation energy of unseen sparse tokens? This would help establish whether the proposed token routing and pruning strategy is robust beyond the calibration distribution.
Reviewer_Bs71OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRPQuant is a post-training quantization method for the language backbone of large vision-language models. It first identifies a fixed set of active channels from calibration data, then classifies tokens as Dense, Intermediate, or Sparse according to the fraction of their energy within that set. Dense tokens use the full-channel branch, Sparse tokens have inactive channels pruned, and Intermediate tokens have their active and residual components computed separately. Both branches use learnable transformations to improve quantization. The paper evaluates low-bit accuracy on six models and reports component ablations, threshold sensitivity, and four-bit kernel and prefill efficiency.

Critical strengths
  1. The routing criterion is well connected to the computation paths. Modality labels do not fully describe token activation distributions. Routing by the actual energy fraction accommodates sparse text tokens and relatively dense visual tokens. The Intermediate path retains both components, avoiding the need to prune all tokens indiscriminately.
  2. The accuracy gains are substantial and accompanied by evidence of reduced computation and measured speedups. On Qwen2.5-VL-7B at W3A3, the six-task average improves from 62.3 with FlatQuant to 69.9. In Table 6, pruning reduces the three-task average only from 68.0 to 67.7 while lowering theoretical computation. Table 9 also reports a four-bit prefill speedup over FlatQuant. These results give the method practical promise.
Critical weaknesses
  1. Energy concentration alone does not adequately explain pruning reliability in the quantized pipeline. Eq. (8) is a valid absolute-error bound for one unquantized layer, but it still depends on the operator norm of the discarded weight submatrix. A high retained energy fraction does not guarantee a small relative output error. For example, with x=(1,1,0.1)x=(1,1,0.1) and W=(1,−1,1)⊤W=(1,-1,1)^\top, the first two coordinates contain about 99.5%99.5\% of the energy, yet pruning changes the output from 0.10.1 to 00. This illustrates a limitation of the criterion, not an observed model failure. Likewise, the Jaccard stability in Table 5 across calibration sets from the same source does not establish post-quantization layerwise output accuracy or coverage across tasks. Direct diagnostics relating energy, routing, and error would better support the claim that low-energy channels can be safely removed.
  2. The contribution of routing and its resource benefits need clearer attribution. Both branches use FlatQuant's learnable transformations [1]. Table 6 includes a useful two-transformation modality control, but does not fully specify how FlatQ+TR handles the three token groups without pruning. It is therefore difficult to check whether branch capacity and calibration budgets are matched. The efficiency evidence also has a specific scope: W4A4 prefill time falls from 84.23 ms to 73.72 ms in Table 9, while memory rises from 4.96 GB to 5.10 GB; Table 8 measures W4A16. These results do not directly establish W3A3 or decoding speedups. Complete ablation definitions and cost accounting would help confirm the gains attributable to routing and pruning. Given the accuracy results, I would consider raising my rating to 3 if these concerns are convincingly addressed. [1] FlatQuant: Flatness Matters for LLM Quantization, International Conference on Machine Learning (ICML), 2025.
Questions for authors
  1. Pruning error: On existing calibration or evaluation samples, how does the retained energy fraction relate to relative output error? Are the most affected layers associated with large discarded-weight norms or cancellation between channel contributions?
  2. Fixed channel set: When activations come from already quantized preceding layers, how do routing proportions and channel-set coverage change? Sparse tokens whose energy lies mostly outside the set may be routed to Dense, primarily affecting efficiency. Does this proportion vary across tasks?
  3. Matched ablations: What exactly is computed for each token group in FlatQ+TR without pruning? Are transformation dimensions, weight storage, and calibration budgets matched to FlatQ+Modal? Please provide the transformation-optimization and quantization settings.
  4. Costs and selection: Which transformation, rearrangement, and dual-branch costs are included in Bit-FLOPs and timing? Which data split was used to select the thresholds in Table 7, and was the active set rebuilt when they changed? If three-bit or decoding timings are unavailable, can the speedup claims be explicitly limited to the measured settings?
Reviewer_cdYkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work identifies differences in the activation patterns of different tokens within a modality, and proposes TRPQuant based on this. This method identifies a fixed set of active channels and dynamically categorizes them into three types—dense, intermediate, and sparse—based on the energy concentration of tokens in these channels. Dense tokens retain all channels and enter the standard branch; sparse tokens undergo channel pruning and enter the sparse branch; and the active channel portion of intermediate tokens enters the sparse branch, while the remaining channel portion enters the standard branch. Quantization is performed using different transformations on these two branches. Experiments on six models in the Qwen2.5-VL, Qwen3-VL, and LLaVA-v1.5 series demonstrate that this method improves low-bit quantization accuracy. On the Qwen2.5-VL-7B model, the W4A4 implementation achieves a 2.60x speedup in pre-filling compared to the FP16 baseline on an RTX 4090, while reducing GPU memory usage by approximately 63%.

Critical strengths
  1. This method is designed based on the clear insight that tokens within the same modality exhibit significant differences in activation patterns. Tokens are grouped according to channel energy concentration and assigned to appropriate quantization paths. A fixed set of activated channels allows dynamically grouped tokens to share a consistent sparse subspace, thus linking this grouping to a practical reduction in computational cost. Component ablation experiments also clearly demonstrate the effectiveness of token rearrangement and channel pruning.
  2. Experimental results show that this method offers advantages in both accuracy and efficiency across six models from three VLM families. On Qwen2.5-VL-7B, TRPQuant's average W3A3 score is 7.6 points higher than FlatQuant's. In addition, its W4A4 implementation achieves a 2.60× prefill speedup over FP16 and lower prefill latency than FlatQuant (73.72 vs. 84.23 ms), while maintaining an average score of 73.7.
Critical weaknesses
  1. Insufficient disclosure of details can hinder the reproducibility of results. Providing more details, such as optimization schedule, quantization granularity, and how the transformation is calibrated, will help improve the reproducibility of results. In particular, it is important to clarify which settings follow FlatQuant and which settings have been specifically modified.
  2. The analysis of the methodology design can be further strengthened, such as analyzing the proportion of active channels and the source of calibration data, which will help to more comprehensively understand the stability of the selected channels and token groups.
Questions for authors
  1. Please clarify the specific quantization range of TRPQuant in LLM. Specifically, does it cover all linear projections in the attention module and the feedforward neural network (FFN) module?
  2. Please report the decoding throughput or decoding time per output token with the same hardware, batch size, and context length settings. Additionally, please explain whether the additional efficiency gains from token rearrangement and channel pruning persist during autoregressive decoding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.