acceptodds
Withdrawn from ICLR 2027

ConTra: Visual Token Pruning via Contrast Invariance in MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) incur substantial inference costs from the sheer volume of visual tokens, especially in text-rich scenarios where textual and visual semantics are deeply entangled within visual tokens. Existing pruning methods rank tokens by holistic visual salience and thus frequently discard text-critical tokens as redundant background. We address this limitation with the Contrast Invariance Principle—textual semantics remain stable under image-level contrast perturbations, whereas background semantics drift substantially. Because glyphs are only the extremal case of ordinal structure in images, the same principle transfers safely to natural, largely text-free scenes. Building on this principle, we propose ConTra, a training-free visual token pruning framework that generates a contrast-transformed view via the Chromatic Contrast Transformation (CCT) and computes cross-view token consistency to identify and preserve text-critical tokens. ConTra requires no attention matrices and only one batched vision-encoder pass over the two views, making it FlashAttention-compatible and readily deployable on off-the-shelf MLLMs without retraining. Across 22 benchmarks spanning multiple model scales, ConTra retains 90.1% performance at a 70% pruning rate on text-rich tasks—surpassing the strongest prior method by 8.1 points—while remaining fully competitive with query-guided methods on general tasks (98.1% retention) despite using no query information, and reduces FLOPs by up to 63%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Token Pruning, Multimodal Large Language Models, Contrast Invariance

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DbS1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ConTra, a training-free visual token pruning method designed to preserve text-critical information in MLLMs. It applies a contrast transformation to the input image and selects tokens based on cross-view token consistency, assuming that text-bearing tokens remain more stable than background tokens. The method does not require attention matrices, making it compatible with FlashAttention. Experimental results show improved performance over existing pruning methods on text-rich benchmarks.

Critical strengths
  1. This paper introduces a novel query-agnostic token selection method based on cross-view consistency under contrast transformations.
  2. The proposed method does not require explicit attention matrices, making it compatible with FlashAttention.
Critical weaknesses
  1. (Major) Insufficient evidence for text-critical token preservation.
  • Unclear qualitative evidence. In Figure 1, some text regions are removed even at 20%, 50%, and 70% pruning. These visualizations do not adequately support the paper's central claim that ConTra effectively preserves text-critical tokens.
  • Potential competition between text and non-text structures. Since CCT preserves not only text but also object contours and geometric structures, it remains unclear whether ConTra remains effective when these features coexist in an image.
  • Unclear text-background separation. It remains unclear whether consistency scores can reliably distinguish text from background tokens. Although Appendix C demonstrates score separability using GMM, the higher-mean component is designated as text without verifying its correspondence to actual text regions.
  1. Insufficient comparisons with relevant token pruning methods.
  • Missing comparisons with text-rich pruning methods. The paper neither cites nor compares recent methods (e.g., [A], [B], [C]) designed for text-rich images.
  • Insufficient comparison with query-aware methods. Although the paper claims to be competitive with query-aware methods, MMTok is the only such baseline evaluated. Additional comparisons with query-aware methods (e.g., [D], [E]) are needed.

+ Inconsistencies in presentation and reporting: Several inconsistencies throughout the paper should be carefully reviewed. For example:

  • Section 4.3 refers to consistency-score maps in Appendix A, but these are not provided.
  • Figure 3's caption does not match its content.
  • CV-Bench is listed in Section 4.1 but omitted from Table 3.

References:

[A] Choi, Joonmyung, et al. "DocPrune: Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruning." CVPR 2026.
[B] Kwon, Mincheol, et al. "Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding." CVPR, 2026.
[C] Wan, Ben, et al. "RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference." ICML 2026.
[D] Yu, Hanxun, et al. “VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration”, ICLR 2026.
[E] Zhang, Qizhe, et al. "Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs." NeurIPS 2025.

Questions for authors
  • Q1. In Fig. 1, ConTra removes some text regions while retaining background regions, even at moderate pruning rates. If the consistency score favors text tokens, why are these background tokens ranked above the text tokens?

  • Q2. Appendix C designates the component with the higher mean as text. Does this component correspond to the annotated text regions, and what is the AUROC of the consistency score in classifying tokens inside and outside these regions on CC-OCR?

  • Q3. When text and objects appear simultaneously in an image, how are the retained tokens divided between text regions and object contours as the pruning rate increases? Do object structures replace text tokens when text occupies a small area?

  • Q4. How does ConTra differ from [A], [B], and [C], which also target text-rich images? Does it outperform them, as well as text-aware baselines such as [D] and [E], on the text-rich benchmarks?

Reviewer_qCz4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

ConTra is a training-free visual token pruning method that ranks tokens by their representation consistency between an original image and a contrast-transformed view. Only the retained original tokens enter the language model. The paper evaluates fixed-ratio and threshold-based selection across text-rich and general vision-language benchmarks, with additional experiments on different model scales and architectures.

Critical strengths

The method is simple, requires no retraining and performs pruning before the language model. This makes it practical for existing inference pipelines. The reported per-benchmark results show meaningful improvements on text-rich tasks. Evaluation across multiple pruning rates and backbones provides useful evidence beyond a single setting.

Critical weaknesses
  1. Limited methodological contribution. The method combines contrast transformation, cosine similarity, and Top-K selection. No comparison with other photometric perturbations establishes the distinctive value of contrast-based consistency.
  2. Unsupported semantic inference. Property 1 establishes a pixel-level algebraic identity, not semantic invariance in a nonlinear encoder. It does not justify the claimed separation of textual and background information.
  3. Circular mechanism validation. Appendix C labels the higher-mean GMM component as “text” without text-region annotations. Demonstrating component separation therefore does not validate text-token identification.
  4. Inconsistent GMM statistics. Equation 8 and Table 5’s reported moments imply a mean component separation no greater than approximately 0.215, contradicting the reported separation of 0.375.
  5. Conflicting aggregate results. Identical benchmark scores for ConTra at 70% pruning yield 90.05% retention in Table 1 but 89.59% in Table 2.
  6. Unclear, scale-sensitive aggregation. Table 3 reports 98.07% retention, versus approximately 95.53% when averaging normalized benchmark scores. The reported value resembles raw-score aggregation, disproportionately weighting MME.
  7. Underspecified transformation implementation. CCT can produce values outside the stated image range. Clipping would invalidate the exact difference-preservation property for affected pixels.
  8. Overstated general-task safety. The claimed “no collateral damage” conflicts with per-task degradation: at 70% pruning, GQA falls from 60.85 to 57.60 even with the recommended moderate transformation.
  9. Incomplete efficiency evidence. Table 4 neglects the unpruned baseline and MMTok, and does not clearly define timing boundaries or memory accounting.
Questions for authors
  1. Using text-region annotations, what precision and recall does the consistency score achieve for identifying text-bearing tokens?
  2. What exact aggregation formula produces Avg, and how do you reconcile Tables 1 and 2? Please also report the mean of per-benchmark normalized scores.
  3. Could you provide the per-image GMM parameters and calculation code to reconcile Table 5 with Equation 8?
  4. How are out-of-range pixel values handled after CCT? Clipping would invalidate the exact difference-preservation property for affected pixels.
  5. Does the latency comparison include both image encodings, preprocessing, and generation under matched settings? What does the reported 1.51 GB memory measure include?
Reviewer_nS4BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ConTra, a training-free visual token pruning method that requires no access to attention matrices. Based on the principle of contrastive invariance, ConTra performs pruning using a single batched forward pass through the vision encoder for two views. The method achieves promising results on text-intensive tasks, including OCR and document understanding.

Critical strengths
  1. The method is simple and effective, relying solely on consistency between corresponding tokens across two views. It requires neither additional training nor text queries, and does not require modifications to pruning layers within the language model.

  2. The method demonstrates strong competitive performance on text-intensive tasks.

Critical weaknesses
  1. Some reported aggregate results appear inconsistent and require clarification. At 70% pruning, Tables 1 and 2 report different average retention scores for ConTra (90.05% vs. 89.59%) despite listing identical per-benchmark results. Furthermore, averaging the per-benchmark normalized scores gives approximately 89.46% for ConTra and 82.81% for MMTok, which does not reproduce the claimed 8.1-point improvement.
  2. The semantic interpretation of the “Contrast Invariance Principle” is insufficiently supported. Property 1 establishes only that pixel differences scale proportionally under the linear transformation; it does not imply that text representations remain stable while background representations drift after passing through the nonlinear vision encoder and projection layer. Backgrounds and objects also contain edges and shapes, while appearance cues such as color and texture may be directly relevant to the task. Although Appendix C labels the higher-mean GMM component as “text” and the lower-mean component as “background,” this correspondence is not validated against independent text-region annotations.
Questions for authors

Please refer to weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.