- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PackAdapt, a learning-based visual token compression framework for MLLMs. The method aims to mitigate information loss caused by conventional token pruning by incorporating information from discarded tokens into retained tokens under a fixed token budget. Specifically, PackAdapt introduces two components: Information Folding, which combines semantic similarity with learnable routing to aggregate information from pruned tokens, and Representation Adaptation, which employs a lightweight residual adapter to refine the aggregated token representations. The token selector, routing module, and representation adapter are jointly optimized using the language modeling objective while keeping the backbone MLLM frozen. Experiments across multiple MLLM backbones and image/video understanding benchmarks demonstrate improvements over existing token pruning methods, particularly under aggressive compression ratios.
Critical strengths
The paper addresses a relevant limitation of existing visual token pruning methods, where potentially useful information is discarded during token selection. The proposed framework is clearly presented and combines information aggregation and representation adaptation in a unified optimization framework.
Critical weaknesses
PackAdapt requires jointly training the token selector, routing projections, temperature parameter, and representation adapter for each target MLLM. Although the backbone remains frozen, such model-specific training introduces additional computational and deployment costs, potentially limiting its practicality when applied to a diverse range of MLLMs. Furthermore, the improvements are most pronounced at the aggressive 10% token retention ratio, where the absolute performance degradation relative to the uncompressed model remains substantial. For example, PackAdapt retains only 82.16% of the original performance on LLaVA-OneVision-1.5-8B at this retention ratio. Such accuracy degradation may be unacceptable for many practical applications. At more moderate compression ratios, where accuracy preservation is more important, the improvements over existing methods are relatively modest. These observations raise concerns about the practical benefits of the proposed framework relative to its additional training requirements.
Questions for authors
-
Why is VisionSelector chosen as the token selection mechanism? The proposed framework directly adopts VisionSelector for token selection, but the paper does not sufficiently justify this particular choice. Is PackAdapt specifically designed around VisionSelector, or can it be integrated with other token selection methods?
-
Can the trained PackAdapt components be transferred across different MLLM backbones without retraining? The current experiments appear to require separate training of the selector, routing projections, temperature parameter, and representation adapter for each target model. Can these trained components be directly transferred to another MLLM without additional training? Given the practical importance of supporting multiple target models, cross-model transfer experiments would substantially strengthen the applicability and generalizability claims of the paper.
-
What explains the discrepancies between the reported VisionSelector results and those in the original VisionSelector paper? The VisionSelector results in Tables 1 and 2 differ from those reported in the original VisionSelector paper. For instance, in Table 2, the ChartQA scores of VisionSelector at 20% and 10% token retention are 73.36 and 61.00, respectively, whereas the original VisionSelector paper reports 74.72 and 62.12 under the corresponding settings. Could the authors clarify the reasons for these discrepancies?
We thank the reviewer for the constructive comments and helpful suggestions.
W1Q1. Contributions of Information Folding and Representation Adaptation. We first clarify that Selection + Adaptation was not conceived as a necessary standalone component of our framework. As motivated in the Introduction, Representation Adaptation was specifically designed to reconcile representation shifts induced by folding pruned information into retained tokens prior to feeding them to the frozen LLM. Applying the adapter directly to unaltered selected tokens deviates from its intended role in PackAdapt. Nonetheless, to directly address the reviewer's concern and isolate whether performance gains stem primarily from the adapter alone, we evaluated this configuration as an additional baseline. The comprehensive ablation results are summarized below (with the newly added configuration requested by the reviewer in bold):
These results demonstrate the complementary synergy of the two modules. While Selection + Adaptation improves upon Selection Only, confirming that representation adaptation itself is beneficial, Full PackAdapt consistently outperforms this baseline across all five benchmarks. This demonstrates that its gains cannot be attributed to the adapter alone, and that incorporating information from pruned tokens provides distinct added value beyond merely adapting the original retained features.
Notably, although Folding Only degrades ChartQA performance, coupling Information Folding with Representation Adaptation substantially boosts accuracy. This directly substantiates our motivation that aggregating discarded visual information alone is insufficient, as the resulting representations require subsequent adaptation to align with downstream predictions. Furthermore, Full PackAdapt consistently surpasses Cosine Routing + Adaptation, validating the empirical advantage of our learnable, task-aware routing strategy over static heuristic routing.
W2Q2. Training–Inference Discrepancy. Our training strategy builds upon the differentiable Top-K paradigm established in prior work (e.g., VisionSelector), employing continuous selection weights as a differentiable surrogate for discrete Top-K selection. Specifically, hard Top-K partitions the input visual sequence into a retained set S and a pruned set D, whereas the differentiable Top-K formulation yields continuous gating scores mi∈[0,1]. During training, the contribution of each pruned token in Information Folding is weighted by (1−mi), while the aggregated retained features are modulated by mj before being fed into the frozen LLM. Concurrently, an auxiliary binary cross-entropy (BCE) consistency loss explicitly aligns these continuous gates with the hard Top-K decisions—driving retained-token gates toward 1 and pruned-token gates toward 0.
This continuous relaxation allows the end-to-end language modeling objective to jointly optimize token selection, information routing, and representation adaptation. Under this formulation, pruned tokens route their complementary features through the folding pathway, while their standalone representations are progressively suppressed by the continuous gating mechanism. The surrogate relaxation thus effectively bridges discrete combinatorial selection and gradient-based optimization.
At inference time, this continuous relaxation is eliminated and replaced with standard hard Top-K selection. Information Folding and Representation Adaptation operate strictly on the selected indices, and pruned visual tokens are physically removed. Consequently, exactly K adapted tokens enter the LLM, strictly adhering to the target inference compute budget. All reported efficiency benchmarks and empirical results are measured under this deterministic hard K-token inference regime without discrepancy.
We have substantially expanded Section 3.4 in the revised manuscript to elucidate the differentiable selection mechanics, the role of the BCE consistency objective, and the exact transition from continuous training to hard discrete inference. All relevant updates are highlighted in blue.