acceptodds
Withdrawn from ICLR 2027

Irreplaceability Matters: Counterfactual Intervention for Visual Token Pruning in MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) represent images as long visual-token sequences, making inference costly. Training-free pruning methods typically rank tokens by saliency or redundancy cues from a single forward pass, yet prominently scored tokens can remain compensable under tight budgets—importance alone does not ensure that visual evidence is hard to replace. We introduce CiVi, a plug-and-play inference-time selector that augments factual-view saliency and pairwise non-redundancy with a feature-space intervention-response proxy inspired by counterfactual reasoning. CiVi perturbs encoder features, compares clean and perturbed projected representations for ranking, and passes only retained clean tokens to the decoder. The intervention response provides a low-cost proxy for decoder-relevant sensitivity rather than an identified causal effect; decoder-level interventions and two-cue ablations support its decoder relevance and the complementarity of the three signals. Across LLaVA-, Qwen-, and video MLLM backbones, CiVi achieves a strong accuracy–efficiency trade-off, particularly under aggressive pruning. On LLaVA-1.5-7B, it preserves 95.7% of the unpruned nine-task average using only 64 of 576 visual tokens. Our code is available at https://anonymous.4open.science/r/CiVi-1082.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Vision-Language Models, Visual Token Compression, Token Pruning, Efficient AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JADFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CiVi, a training-free visual token pruning method based on the concept of irreplaceability. The authors argue that traditional metrics for saliency and redundancy fail to adequately capture whether visual information can be compensated for by other tokens. To address this, CiVi introduces an adaptive feature-space perturbation strategy to estimate intervention responses, combining this with factual saliency and non-redundancy for token selection. Experiments across various image and video MLLMs demonstrate that the method is competitive in terms of both performance and inference efficiency, particularly in high-intensity pruning scenarios.

Critical strengths
  1. The paper explores visual token pruning from the perspective of irreplaceability, offering an interesting angle that goes beyond traditional concepts of token importance.

  2. Experiments cover a wide range of backbone networks, pruning ratios, and image/video benchmarks, showing excellent performance at high compression rates.

  3. The paper includes component ablation studies, validation of decoder interventions, and inference latency measurements, providing strong empirical support for the proposed method.

Critical weaknesses
  1. The paper emphasizes that a token's irreplaceability depends on whether its information can be compensated for by other retained tokens. However, CiVi employs a static Top-K selection strategy without explicitly accounting for the dynamic changes in the retained set. Furthermore, while decoder interventions rely on text queries, the actual selection scores are derived entirely from visual features. Consequently, the method appears more like a combination of saliency, perturbation sensitivity, and non-redundancy rather than a direct estimate of irreplaceability.

  2. The method assigns higher perturbation ratios to visual features with more concentrated distributions, yet the rationale for this choice is not fully explained. For instance, images with uniform backgrounds containing only a few key objects might be subjected to stronger perturbations, potentially introducing additional noise into the token scoring process. Although the paper evaluates various perturbation types and sample sizes, it lacks a direct comparison against a fixed perturbation ratio, and the rationale behind several heuristic design choices is insufficiently explained.

  3. According to the formulation, unperturbed tokens receive an intervention response score of zero, and these scores feed directly into the log-domain score fusion process. Given that only three perturbation samples are used by default, token rankings may depend to some extent on the specific random masks and the frequency with which each token is perturbed. While deterministic seeds ensure reproducibility and the paper explores different sample sizes, it reports neither performance variance across independent random seeds nor the stability of the selected token sets.

  4. A strong response in the feature space to random perturbations does not necessarily imply a significant impact on the decoder's predictions. Although Figure 4 demonstrates some consistency between proxy rankings and decoder intervention results, that comparison relies on specific perturbation mechanisms and measures the impact of individual token replacements; it does not assess whether the retained token set can compensate for the removed visual evidence. Furthermore, a direct comparison with existing perturbation-based sensitivity analysis methods (such as ZOO-Prune) would help clarify the unique contribution of the proposed intervention response scores.

Questions for authors

Please see the weaknesses above.

Reviewer_jqDrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CiVi, a training-free visual token pruning method for Multimodal Large Language Models (MLLMs), motivated by the observation that token saliency alone does not necessarily reflect the irreplaceability of visual evidence. CiVi introduces an intervention-response proxy by applying stochastic perturbations to visual encoder features and comparing the clean and perturbed representations after the visual projector. It then combines three complementary signals—factual-view saliency, intervention response, and pairwise non-redundancy—to select informative visual tokens, while passing only the retained clean tokens to the frozen language decoder.

The method is evaluated across multiple MLLM backbones, including LLaVA, Qwen, and Video-LLaVA, on image and video understanding benchmarks. Experimental results demonstrate competitive performance under aggressive pruning. Notably, CiVi retains 95.7% of the unpruned nine-benchmark average performance on LLaVA-1.5-7B using only 64 of 576 visual tokens. The authors further provide ablation studies, inference efficiency measurements, and decoder-level intervention experiments to validate the contributions of the proposed scoring signals and their relevance to decoder sensitivity.

Critical strengths
  1. Well-motivated problem formulation and strong empirical performance.

The paper highlights a meaningful limitation of saliency-based visual token pruning: highly salient tokens may still contain redundant or compensable information. This observation provides an interesting perspective on preserving hard-to-compensate visual evidence under aggressive pruning. The experiments cover multiple MLLM backbones and diverse image and video understanding benchmarks, demonstrating a competitive accuracy–efficiency trade-off. Notably, CiVi preserves 95.7% of the unpruned average performance on LLaVA-1.5-7B while retaining only 64 of 576 visual tokens.

  1. Practical training-free design with supporting experimental validation.

CiVi operates without additional training or repeated decoder forward passes for token scoring, making it practical for inference-time deployment. The authors also go beyond standard benchmark evaluations by conducting decoder-level intervention experiments to examine the alignment between the proposed proxy scores and actual decoder sensitivity. Together with the ablation studies and efficiency measurements, these experiments provide useful empirical evidence for the effectiveness and complementarity of the proposed scoring signals.

Critical weaknesses
  1. The novelty relative to existing work is not sufficiently established.

CiVi combines factual-view saliency, intervention response, and pairwise non-redundancy for visual token pruning, but these underlying ideas have been extensively explored. In particular, ZOO-Prune (CVPR 2026) already estimates token sensitivity through random perturbations involving the visual projector and combines sensitivity with diversity for training-free pruning, closely resembling the core methodology of CiVi. Although CiVi introduces a different perturbation strategy, discrepancy enhancement, and score fusion mechanism, the lack of a direct comparison with ZOO-Prune makes it difficult to assess whether these modifications offer substantial methodological novelty or meaningful performance improvements.

  1. A gap remains between the theoretical motivation of irreplaceability and its practical approximation.

The paper emphasizes that a token's irreplaceability depends on whether its information can be compensated for by other retained tokens. However, CiVi employs a static pairwise non-redundancy score computed over all candidate tokens, without explicitly considering each token's marginal contribution relative to the retained set. Furthermore, the intervention response primarily measures changes in projected features rather than their direct impact on decoder predictions. Although the authors discuss the computational trade-off and provide supporting decoder-intervention experiments, the current evidence does not sufficiently establish the connection between the proposed proxy scores and retained-set-conditioned irreplaceability.

Questions for authors

Q1. Distinction from ZOO-Prune and methodological novelty.

ZOO-Prune already estimates token sensitivity through random perturbations involving the visual projector and combines it with diversity-aware selection for training-free pruning. Could the authors clarify the substantive methodological differences between CiVi and ZOO-Prune and provide a direct comparison under matched backbones, pruning ratios, evaluation protocols, and comparable inference costs? Such a comparison would be critical to assessing the paper's incremental contribution and could materially affect my overall rating.

Q2. Practical approximation of irreplaceability.

The authors have explained the high computational cost of direct decoder-level interventions. However, could they further justify the use of static pairwise non-redundancy computed over all candidate tokens, rather than a retained-set-conditioned non-redundancy score? Since the latter can also be approximated in feature space, it would be helpful to clarify the specific trade-off between computational efficiency and information coverage underlying the current design.

Q3. Stability of stochastic intervention-response scoring.

The paper discusses scoring stability through multi-sample aggregation and variance analysis. However, with the default setting of only three intervention samples, some tokens may remain unperturbed and consequently receive a zero intervention-response score. Could the authors report the Top-K selection overlap or ranking consistency across different random seeds, and compare the current sampling strategy with an alternative that ensures each candidate token is perturbed at least once? Such evidence would help assess the robustness of the scoring mechanism to stochastic perturbation masks.

Q4. Performance improvements over the unpruned model in video understanding.

Table 2 shows that CiVi achieves 102.3% of the unpruned Video-LLaVA-7B's average relative performance while retaining only 256 out of 2,048 visual tokens, whereas all competing pruning methods perform below the unpruned baseline on average. This is an interesting result. Could the authors further investigate whether the improvement primarily stems from removing redundant information, suppressing distracting visual evidence, or other factors? Additionally, reporting variance across independent runs or confidence intervals would help establish the robustness of this improvement.

Reviewer_qJWkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CiVi, a training-free visual-token pruning method for Multimodal Large Language Models. The method aims to preserve visual evidence that is difficult to replace under aggressive token pruning, rather than relying only on conventional token-importance scores. CiVi combines factual-view saliency, feature-space intervention response, and pairwise non-redundancy. Experiments across LLaVA, Qwen, and video MLLMs report competitive accuracy-efficiency trade-offs.

Critical strengths
  1. The paper addresses an important efficiency challenge in MLLMs. The observation that highly salient tokens may still contain redundant information under aggressive pruning is meaningful.

  2. CiVi is training-free and can be integrated into existing MLLMs without modifying their weights. Passing only retained clean tokens to the decoder avoids introducing perturbed representations into generation.

Critical weaknesses
  1. Saliency estimation, feature perturbation, and diversity-based selection have been explored previously. In particular, ZOO-Prune also estimates token sensitivity through projection-level perturbations and combines sensitivity with diversity for training-free visual-token pruning. Although CiVi uses a different intervention-response formulation and scoring strategy, the distinction is not sufficiently established.

  2. The intervention-response score measures representation changes under feature perturbations, while pairwise non-redundancy measures similarity among candidate tokens. Neither directly evaluates a token's contribution conditioned on the final retained set. The decoder-intervention validation measures agreement with token-wise decoder sensitivity under a related perturbation protocol, rather than directly demonstrating set-conditioned irreplaceability.

  3. In one reported LLaVA-1.5-7B setting, the average retention improvement over VisionZip is only 0.4 percentage points at 77.8% pruning, and the main tables do not report uncertainty estimates.

Questions for authors
  1. Does the intervention-response score correlate with the actual performance degradation caused by removing or replacing a token from the final retained set? Can the authors provide a more direct evaluation of set-conditioned token irreplaceability?

  2. Can the authors provide uncertainty estimates or robustness analyses for the reported accuracy improvements, and characterize the quadratic scoring overhead at larger visual sequence lengths and batch sizes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.