acceptodds
Under review as a conference paper at ICLR 2027

One Objective, One Stage: Unifying Relevance, Coverage, and Redundancy in Visual Token Pruning

Abstract

Multimodal large language models (MLLMs) process large numbers of visual tokens alongside text, incurring substantial computation and inference latency. Existing training-free visual token pruning methods typically rely on single heuristics such as relevance, coverage, or redundancy. Although some approaches mitigate the limitations of single criteria through multi-stage or multi-module combinations, they still lack a principled, unified measure of the retained set value. Therefore, we propose **UniPrune**, a unified visual token pruning method that operates under a single objective in a single stage. UniPrune first utilizes cross-modal relevance operators to measure how visual tokens support text semantics. Subsequently, it uses Log-Sum-Exp (LSE) aggregation to combine the complementary support provided by different visual tokens, while its logarithmic form reduces the marginal benefit of redundant support. This objective jointly and explicitly captures relevance, semantic coverage, and redundancy suppression. We prove that this objective is monotone submodular, allowing the classical greedy algorithm to provably achieve an approximately optimal solution with low complexity. Across multiple benchmarks on three MLLM base models, UniPrune consistently achieves the highest average performance retention among all compared methods, including under the most aggressive 95% visual token pruning. UniPrune is independent of the vision encoder, requires minimal architectural changes, and remains compatible with acceleration methods such as FlashAttention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Visual Token Pruning, Training-Free Inference Acceleration, Submodular Optimization.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cKX8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniPrune, a training-free method for pruning visual tokens in multimodal large language models. After an early LLM layer, UniPrune constructs a positive support matrix from visual–textual relevance scores and defines the value of a retained token set through a Log-Sum-Exp objective. Tokens are selected greedily by repeatedly updating their marginal gains. The authors interpret this formulation as jointly modeling relevance, textual semantic coverage, and redundancy suppression. They establish that the regularized objective is monotone submodular, enabling the classical greedy approximation guarantee under a cardinality constraint. The performance evaluation covers three backbone models, nine image benchmarks, and six video benchmarks, with additional efficiency measurements on LLaVA-NeXT-7B. UniPrune achieves the highest average performance retention among the compared methods across the main evaluated settings and reports approximately 1.76× end-to-end acceleration. The supplementary material includes analyses of temperature, pruning position, token budget, and fine-grained capabilities.

Critical strengths
  1. A compact formulation with clear mathematical properties. For a fixed support matrix, the regularized objective has a straightforward monotone-submodular structure and a closed-form marginal gain. The method requires no additional training, explicit attention maps, or token reconstruction, which makes it practically attractive to integrate.

  2. Broad empirical coverage and competitive average performance. Tables 1–3 evaluate multiple backbones across image and video benchmarks, with consistent average improvements under tight token budgets. The supplementary budget, temperature, position, and capability analyses provide useful additional evidence. The efficiency evaluation also includes pruning overhead, making the reported acceleration more informative than measurements that exclude selection cost.

Critical weaknesses
  1. The motivation does not sufficiently establish the advance over existing set-based methods. MMTok and CoIn already jointly consider complementary selection criteria, and MMTok also provides a submodular formulation with a greedy approximation guarantee. The paper needs to demonstrate more clearly why its LSE objective provides a substantive advantage beyond simplifying existing formulations. Moreover, equally weighting text tokens does not necessarily correspond to covering distinct, task-relevant semantics.

  2. The experimental evidence does not fully establish robust accuracy–efficiency superiority. Average improvements conceal substantial task-specific weaknesses, including lower POPE performance than CoIn/MMTok and severe TextVQA degradation under aggressive pruning. Some gains are small and lack uncertainty estimates. Table 6 also combines latency from LLaVA-NeXT with average accuracy from other backbones, limiting conclusions about the overall tradeoff. Matched accuracy–latency evaluations and scaling measurements are needed, particularly because selection costs O(KNM)O(KNM), which becomes quadratic in visual token count at a fixed retention ratio.

Questions for authors
  1. Implementation consistency in Table D6: Why is the text-independent subtraction in “Negative redundancy” not canceled by the subsequent row-wise z-score normalization? Please provide the exact operation order and compare its normalized score matrix and selected indices with those of default cosine, including the numerical precision used.

  2. Interpretation of Frozen-Marginal Top-K: What is the distribution of initial marginal gains under the actual temperature and ε settings? How frequent are exact or near ties, and how sensitive are the results to token order, numerical precision, and tie-breaking? How does this variant compare with random selection and raw-cosine Top-K?

  3. Evidence for the core mechanism: Under matched pruning positions, normalization, and temperature settings, do the benefits of LSE over Hard-Max also hold on Qwen and InternVL? How sensitive is selection to prompt templates, function words, and repeated text tokens?

  4. Robustness of the performance gains: Can the authors provide paired confidence intervals against the strongest baselines and explain the weaker results on POPE, TextVQA, and motion-related tasks? Does the contribution of Video-MMMU to the video average indicate a task-specific preference?

  5. Matched efficiency evaluation: Can the authors report accuracy–end-to-end latency curves under the same backbone, input lengths, and token budgets, together with timing variability? How does selection overhead scale with visual token count, text length, and video frame count?

  6. Reproducibility details: Please specify which text tokens participate in scoring, the order of normalization and temperature computation, the numerically stable implementation of exponential support and marginal gains, and the textual representations used for per-frame video selection.

1 comment

Thank you for the detailed response. The matched LSE versus Hard-Max comparisons, bootstrap interval, and timing breakdown address most of my empirical concerns. The side-by-side objectives also clarify UniPrune’s differences from MMTok and CoIn. My remaining concern is whether equal-weight aggregation over text tokens reliably captures distinct semantic aspects, especially when prompts contain repeated or low-information tokens. I would keep my current rating while noting that the rebuttal has increased my confidence in the empirical results.

Reviewer_ke6QOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

UniPrune performs one visual-token selection after an early LLM layer. It converts visual–text similarities into positive support and greedily maximizes a regularized sum of logarithmic support across text tokens. The paper establishes monotone submodularity and evaluates nine image and six video benchmarks. Reported average retention reaches 95.2%/91.0% with 64/32 LLaVA tokens, alongside a 1.76× end-to-end speedup on LLaVA-NeXT.

Critical strengths
  1. The objective and incremental marginal-gain rule are compact and easy to follow. Avoiding attention maps and an explicit visual–visual matrix is a useful implementation choice.
  2. The evaluation includes recent, closely related baselines, multiple architectures, and five LLaVA budgets. Average retention consistently improves over the reported competitors, with particularly strong video action-count and object-existence results.
  3. The structural, pruning-position, operator, and temperature analyses provide useful evidence. I appreciate the acknowledgement that dynamic temperature is a configuration convenience rather than a consistently superior innovation.
  4. The regularized objective has a valid submodularity proof, and selection overhead is included in the reported effective-prefill timing.
Critical weaknesses
  1. Does the objective preserve complementary evidence? Appendix A.5 applies row-wise z-scores, removing each visual token's mean alignment, while the ablations mainly establish component-level accuracy benefits. A small annotated-region comparison of the full objective, Hard-Max, and unnormalized cosine would help connect the gains to necessary evidence coverage and redundancy.

  2. The performance–efficiency comparison combines different workloads. Table 6 pairs LLaVA-NeXT/POPE latency with average retention from three other model suites. Could the authors provide accuracy–latency curves on one shared backbone over a few budgets, with equal validation tuning opportunities and repeated timing including all selection costs, especially given the approximately 4.8% end-to-end margin over MMTok?

  3. How does the negative-redundancy term survive normalization? Table D6 subtracts a row-constant scalar before the shared row-wise z-score, so the term should cancel algebraically, yet the results differ from default cosine. The actual formula, preprocessing order, and numerical handling need clarification before this ablation can support a redundancy claim.

  4. The guarantee needs a clearer practical interpretation. The classical bound [1] is valid for Eq. (7), but its additive baseline is approximately 18.42 M tau at epsilon = 1e-8; the same multiplicative guarantee does not transfer to raw LSE support or pruning loss. For fixed epsilon, the regularized strongest-match term also tends to max(0, r_max) as tau approaches zero, which should be explicit in the Hard-Max interpretation.

[1] G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. “An Analysis of Approximations for Maximizing Submodular Set Functions—I.” Mathematical Programming 14:265–294, 1978. Publisher.

[2] Sixun Dong, Juhua Hu, Mian Zhang, Ming Yin, Yanjie Fu, and Qi Qian. “MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs.” ICLR, 2026. Paper.

[3] Chenxi Du, Yongheng Deng, Jiani Liu, Yujia Zhang, Xi Chen, and Ju Ren. “CoIn: Coverage and Informativeness-Guided Token Reduction for Efficient Large Multimodal Models.” CVPR, pp. 10492–10501, 2026. Official paper.

[4] Qingchan Zhu, Weihang You, Hanqi Jiang, Changdi Yang, Tianming Liu, and Geng Yuan. “Who Speaks for the Pruned? Visual Token Pruning as Coverage Optimization.” arXiv:2609.03158, 2026; authors report acceptance to EMNLP 2026 main. Paper.

Questions for authors

please refer to the weaknesses above.

Reviewer_UQnmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UniPrune, a training-free visual token pruning method that unifies cross-modal relevance, semantic coverage, and redundancy suppression through a single Log-Sum-Exp (LSE) objective. By formulating token selection as a monotone submodular optimization problem, UniPrune enables efficient greedy selection with theoretical approximation guarantees. Experiments across three MLLM backbones and 15 image and video benchmarks demonstrate competitive performance retention under aggressive pruning ratios, while achieving inference acceleration without additional training or substantial architectural modifications.

Critical strengths

1. Unified formulation with theoretical guarantees. UniPrune integrates relevance, semantic coverage, and redundancy suppression into a single objective, providing a simple and interpretable framework with theoretical guarantees based on submodular optimization.

2. Comprehensive experiments and strong performance. Extensive experiments across multiple MLLMs and image/video benchmarks demonstrate consistent performance improvements, particularly under aggressive token pruning. Comparisons with recent baselines and ablation studies further support its effectiveness.

Critical weaknesses
  1. The proposed LSE objective builds on well-established submodular optimization principles, and its theoretical guarantees follow from classical results. The methodological differences from existing set-based pruning methods, such as CoIn and MMTok, are not sufficiently substantial or clearly demonstrated.

  2. The row-wise z-score normalization may cause the initial marginal gains of different visual tokens to become nearly identical when the regularization term is negligible. This raises concerns about the sensitivity of the initial selection to numerical precision and tie-breaking, which is not adequately investigated.

  3. Although the proposed objective theoretically integrates relevance, coverage, and redundancy suppression, the paper lacks direct empirical evidence supporting these properties. The authors could quantitatively evaluate the cross-modal relevance, semantic coverage, and pairwise redundancy of retained visual tokens, and compare these metrics with existing methods under the same token budgets. Such analyses would help demonstrate whether the performance improvements indeed result from the claimed unified optimization mechanism.

  4. The video experiments uniformly sample eight frames and perform token pruning independently for each frame, without explicitly considering cross-frame redundancy. The authors could investigate the impact of different frame counts and compare per-frame selection with global token selection across frames to better demonstrate the method's effectiveness on video understanding tasks.

Questions for authors

See Weakness

1 comment
Reviewer_UQnmOpenReview ↗

Response to rebuttal

Overall, the rebuttal has clarified several technical details and strengthened the empirical validation. Nevertheless, my primary concern regarding the methodological novelty remains. I therefore maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.