acceptodds
Withdrawn from ICLR 2027

Pack, Don't Drop: Breaking the Selection Bottleneck in Visual Token Pruning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual token pruning reduces the inference cost of multimodal large language models (MLLMs), yet under strict token budgets, selection alone inevitably discards complementary visual evidence. Existing aggregation methods reuse information from pruned tokens within compressed sequences, providing a complementary direction beyond selection. Under a fixed token budget, effective information reuse therefore requires both appropriate routing of pruned content and suitable representations at the retained positions. To this end, we propose PackAdapt, which enhances a given set of retained positions without changing the final token budget. Its Information Folding module learns task-relevant routing from pruned tokens to retained tokens on top of semantic similarity, while its Representation Adaptation module further transforms the folded features through a lightweight residual mapping. With the base MLLM frozen, the selector, routing module, and representation adapter are jointly trained with the language-modeling objective, so supervision from the final prediction reaches all three components. Extensive experiments across multiple multimodal backbones, image-understanding benchmarks, and video-understanding benchmarks demonstrate that PackAdapt consistently improves compressed-model performance under different token budgets, with particularly strong gains under aggressive visual compression.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Token Pruning, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o1nKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PackAdapt, a learning-based visual token compression framework for MLLMs. The method aims to mitigate information loss caused by conventional token pruning by incorporating information from discarded tokens into retained tokens under a fixed token budget. Specifically, PackAdapt introduces two components: Information Folding, which combines semantic similarity with learnable routing to aggregate information from pruned tokens, and Representation Adaptation, which employs a lightweight residual adapter to refine the aggregated token representations. The token selector, routing module, and representation adapter are jointly optimized using the language modeling objective while keeping the backbone MLLM frozen. Experiments across multiple MLLM backbones and image/video understanding benchmarks demonstrate improvements over existing token pruning methods, particularly under aggressive compression ratios.

Critical strengths

The paper addresses a relevant limitation of existing visual token pruning methods, where potentially useful information is discarded during token selection. The proposed framework is clearly presented and combines information aggregation and representation adaptation in a unified optimization framework.

Critical weaknesses

PackAdapt requires jointly training the token selector, routing projections, temperature parameter, and representation adapter for each target MLLM. Although the backbone remains frozen, such model-specific training introduces additional computational and deployment costs, potentially limiting its practicality when applied to a diverse range of MLLMs. Furthermore, the improvements are most pronounced at the aggressive 10% token retention ratio, where the absolute performance degradation relative to the uncompressed model remains substantial. For example, PackAdapt retains only 82.16% of the original performance on LLaVA-OneVision-1.5-8B at this retention ratio. Such accuracy degradation may be unacceptable for many practical applications. At more moderate compression ratios, where accuracy preservation is more important, the improvements over existing methods are relatively modest. These observations raise concerns about the practical benefits of the proposed framework relative to its additional training requirements.

Questions for authors
  1. Why is VisionSelector chosen as the token selection mechanism? The proposed framework directly adopts VisionSelector for token selection, but the paper does not sufficiently justify this particular choice. Is PackAdapt specifically designed around VisionSelector, or can it be integrated with other token selection methods?

  2. Can the trained PackAdapt components be transferred across different MLLM backbones without retraining? The current experiments appear to require separate training of the selector, routing projections, temperature parameter, and representation adapter for each target model. Can these trained components be directly transferred to another MLLM without additional training? Given the practical importance of supporting multiple target models, cross-model transfer experiments would substantially strengthen the applicability and generalizability claims of the paper.

  3. What explains the discrepancies between the reported VisionSelector results and those in the original VisionSelector paper? The VisionSelector results in Tables 1 and 2 differ from those reported in the original VisionSelector paper. For instance, in Table 2, the ChartQA scores of VisionSelector at 20% and 10% token retention are 73.36 and 61.00, respectively, whereas the original VisionSelector paper reports 74.72 and 62.12 under the corresponding settings. Could the authors clarify the reasons for these discrepancies?

Reviewer_2PYMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PackAdapt extends VisionSelector with learnable routing that aggregates discarded visual tokens into retained positions, followed by a residual adapter. It freezes the base multimodal model and preserves the final token budget. Experiments cover three backbones, eight image benchmarks, and two video benchmarks.

Critical strengths
  1. Consistent gains across backbones. At 10% retention, normalized average performance improves over the strongest baseline by 3.26 percentage points on Qwen2.5-VL and 1.61 points on LLaVA-OneVision.
  2. Practical design. The method freezes the backbone, preserves the final token count, and reduces measured inference latency.
Critical weaknesses
  1. The contribution of Information Folding is not isolated. In Figure 3, Folding Only reduces ChartQA performance relative to Selection Only, while adding Adaptation substantially improves it. The missing Selection + Adaptation baseline prevents attributing these gains to recovering discarded information. Comparable performance from an adapter alone would substantially weaken the central contribution.
  2. Training may bypass the compression bottleneck. Section 3.4 passes discarded tokens directly into the language model as mixim_i x_i during training, then removes them at inference. Feature scaling alone does not exclude these positions from attention. The paper neither clarifies whether attention masking blocks this path nor quantifies the training–inference gap, leaving the effectiveness of the surrogate insufficiently validated.
Questions for authors
  1. Under identical training settings and matched trainable parameter counts, how does Selection + Adaptation with α=0\alpha = 0 perform? Does the full method retain a consistent advantage?
  2. Are discarded positions masked out of attention during training? If not, do the gains persist when training passes only the KK retained tokens to the language model?
1 comment

We thank the reviewer for the constructive comments and helpful suggestions.

W1Q1. Contributions of Information Folding and Representation Adaptation. We first clarify that Selection + Adaptation was not conceived as a necessary standalone component of our framework. As motivated in the Introduction, Representation Adaptation was specifically designed to reconcile representation shifts induced by folding pruned information into retained tokens prior to feeding them to the frozen LLM. Applying the adapter directly to unaltered selected tokens deviates from its intended role in PackAdapt. Nonetheless, to directly address the reviewer's concern and isolate whether performance gains stem primarily from the adapter alone, we evaluated this configuration as an additional baseline. The comprehensive ablation results are summarized below (with the newly added configuration requested by the reviewer in bold):

MethodOCRBenchDocVQAChartQASQAPOPE-F1
Selection Only62772.0161.0482.6577.47
Folding Only63074.0158.4883.7979.47
Selection + Adaptation63272.9663.2083.1780.25
Cosine Routing + Adaptation62073.5167.6883.7982.64
Full PackAdapt64376.7968.8885.0383.75

These results demonstrate the complementary synergy of the two modules. While Selection + Adaptation improves upon Selection Only, confirming that representation adaptation itself is beneficial, Full PackAdapt consistently outperforms this baseline across all five benchmarks. This demonstrates that its gains cannot be attributed to the adapter alone, and that incorporating information from pruned tokens provides distinct added value beyond merely adapting the original retained features.

Notably, although Folding Only degrades ChartQA performance, coupling Information Folding with Representation Adaptation substantially boosts accuracy. This directly substantiates our motivation that aggregating discarded visual information alone is insufficient, as the resulting representations require subsequent adaptation to align with downstream predictions. Furthermore, Full PackAdapt consistently surpasses Cosine Routing + Adaptation, validating the empirical advantage of our learnable, task-aware routing strategy over static heuristic routing.

W2Q2. Training–Inference Discrepancy. Our training strategy builds upon the differentiable Top-KK paradigm established in prior work (e.g., VisionSelector), employing continuous selection weights as a differentiable surrogate for discrete Top-KK selection. Specifically, hard Top-KK partitions the input visual sequence into a retained set S\mathcal{S} and a pruned set D\mathcal{D}, whereas the differentiable Top-KK formulation yields continuous gating scores mi∈[0,1]m_i \in [0, 1]. During training, the contribution of each pruned token in Information Folding is weighted by (1−mi)(1 - m_i), while the aggregated retained features are modulated by mjm_j before being fed into the frozen LLM. Concurrently, an auxiliary binary cross-entropy (BCE) consistency loss explicitly aligns these continuous gates with the hard Top-KK decisions—driving retained-token gates toward 1 and pruned-token gates toward 0.

This continuous relaxation allows the end-to-end language modeling objective to jointly optimize token selection, information routing, and representation adaptation. Under this formulation, pruned tokens route their complementary features through the folding pathway, while their standalone representations are progressively suppressed by the continuous gating mechanism. The surrogate relaxation thus effectively bridges discrete combinatorial selection and gradient-based optimization.

At inference time, this continuous relaxation is eliminated and replaced with standard hard Top-KK selection. Information Folding and Representation Adaptation operate strictly on the selected indices, and pruned visual tokens are physically removed. Consequently, exactly KK adapted tokens enter the LLM, strictly adhering to the target inference compute budget. All reported efficiency benchmarks and empirical results are measured under this deterministic hard KK-token inference regime without discrepancy.

We have substantially expanded Section 3.4 in the revised manuscript to elucidate the differentiable selection mechanics, the role of the BCE consistency objective, and the exact transition from continuous training to hard discrete inference. All relevant updates are highlighted in blue.

Reviewer_BX1sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PackAdapt, which augments learned visual token selection with information folding and representation adaptation. Pruned features are routed into retained tokens using cosine similarity and a learned correction, then refined by a residual adapter. The base MLLM remains frozen. Experiments across three backbones and image and video benchmarks report improved performance at fixed token budgets.

Critical strengths
  1. The method addresses a practical limitation of selection-only compression: useful information in discarded tokens. It preserves the final token count and requires no updates to the base MLLM.
  2. Evaluation covers multiple backbones, eight image benchmarks, and two video benchmarks. At 10% retention on Qwen2.5-VL, PackAdapt improves normalized average performance from VisionSelector’s 85.55% to 89.16%, supporting its practical value under aggressive compression.
Critical weaknesses
  1. The central contribution is not sufficiently isolated. Figure 3 neglects a selection-plus-adapter baseline without folding. Consequently, the results cannot determine how much improvement requires recovering discarded information rather than simply adapting retained features.
  2. Comparisons with closely related merging methods are incomplete. The paper discusses LLaVA-PruMerge and VISA as approaches that incorporate removed information into retained representations, but does not evaluate them. VisionZip is included, yet its separate contextual-token allocation differs from the central design being proposed.
  3. The training–inference mismatch is insufficiently examined. Training retains all visual positions with continuous feature weights, whereas inference physically removes pruned positions and changes the folding weights. Feature scaling alone is not equivalent to removing tokens from attention. The paper does not quantify this discrepancy.
  4. Efficiency and uncertainty reporting are incomplete. Table 4 neglects VisionSelector. It also lacks module-level timings and memory scaling with input length. Results average three seeds but provide no variability estimates, making smaller improvements difficult to assess.
  5. Generalization across model generations and scales remains unverified. Evaluation is restricted to approximately 7–8B backbones and does not include newer models such as Qwen3-VL. Since the method operates directly on visual interface representations, its effectiveness may depend on the backbone’s visual encoding and multimodal integration design.
Questions for authors
  1. How does selection plus the same representation adapter, without information folding, compare with full PackAdapt under identical training conditions?
  2. What is the additional latency and peak memory relative to VisionSelector, and how does this overhead scale with visual sequence length?
  3. How does PackAdapt compare with LLaVA-PruMerge and VISA at matched token budgets? Please clarify training-budget differences.
  4. Could you provide variability estimates across the three seeds, particularly for the smaller improvements on LLaVA-OneVision and video benchmarks?
  5. Do the gains persist on a newer backbone such as Qwen3-VL-8B? This would help establish whether the benefits extend beyond the evaluated model generations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.