acceptodds
Withdrawn from ICLR 2027

CausalPrune: Counterfactual Causal Token Pruning for Efficient Large Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable progress in multimodal understanding, but their practical deployment is still limited by the substantial computation introduced by excessive visual tokens. Visual token pruning offers an effective solution by removing redundant visual tokens during inference. However, existing methods typically estimate token importance from correlational statistics, which cannot distinguish tokens that causally determine the model output from those spuriously associated with it. To address this issue, we propose CausalPrune, a counterfactual causal token pruning framework for efficient LVLMs. We formulate LVLM inference as a Structural Causal Model and introduce the Causal Token Importance Score (CTIS), which estimates the Average Treatment Effect of intervening on each visual token. To make causal estimation practical, we derive an efficient first-order approximation of CTIS and incorporate backdoor adjustment to mitigate latent cross-modal confounding. Within this causal formulation, layer-wise causal mediation decomposes token importance into direct and indirect effects through intermediate hidden states, enabling progressive pruning across transformer layers in a principled manner. Extensive experiments on various LVLM architectures, such as LLaVA-1.5 and Qwen2.5-VL, demonstrate that CausalPrune maintains comparable or even superior multimodal performance while achieving substantial reductions in FLOPs and prefill latency, consistently outperforming existing pruning methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Vision-Language Models, Visual Token Pruning, Counterfactual Causal Inference, Causal Token Importance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_BmRCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to investigate visual token pruning for accelerating the inference of large vision-language models. The authors use counterfactual interventions to define the causal importance of visual tokens, and employ a gradient-based approximation to reduce the cost of causal importance estimation, together with backdoor adjustment to mitigate spurious visual-text associations. They further incorporate layer-wise causal mediation analysis to progressively remove unimportant visual tokens across multiple Transformer layers, thereby reducing FLOPs, KV cache usage, and inference latency while preserving model performance. Experiments on models such as LLaVA, Qwen2.5-VL, and InternVL show that the proposed method can maintain competitive multimodal performance even under high visual token pruning ratios, while reducing inference latency.

Critical strengths
  1. The research problem addressed in this paper is meaningful. Moving beyond traditional correlation-based measures of visual token importance to consider causal effects provides a potentially useful perspective for LVLM pruning.
  2. The paper conducts evaluations across multiple LVLM architectures and a variety of multimodal benchmarks, providing broad experimental coverage.
Critical weaknesses
  1. My main concern is that the paper's core causal modeling and theoretical guarantees are not sufficiently supported. (i) The proposed backdoor adjustment does not appear to be valid under the paper's own causal graph. The authors explicitly identify the backdoor path Vi←UI→Vk→⋯→YV_i \leftarrow U_I \rightarrow V_k \rightarrow \cdots \rightarrow Y, but this path does not contain any text token TT. Therefore, conditioning on TT itself cannot block this path. The paper further argues that the independence between TT and UIU_I justifies the blocking, but such independence does not establish the required d-separation. (ii) CTIS is defined in terms of changes in the output logits YY, whereas the practical GradScore is computed from a scalar loss L\mathcal{L}. Moreover, the first-order Taylor term is ∇viL⊤vi\nabla_{v_i}\mathcal{L}^{\top}v_i, while the proposed GradScore is ∥∇viL⊙vi∥2\lVert \nabla_{v_i}\mathcal{L}\odot v_i \rVert_2. These two quantities are generally not equivalent, so the current derivation does not directly establish GradScore as a first-order approximation to CTIS. (iii) Theorem 3 claims that CTISapprox(vi)→pCTIS(vi)\mathrm{CTIS}_{\mathrm{approx}}(v_i) \overset{p}{\to} \mathrm{CTIS}(v_i) as n→∞n\to\infty. However, the proof actually retains a non-vanishing Taylor error ϵT(vi)\epsilon_T(v_i), and exact convergence additionally requires Λ→0\Lambda\to0, which is not included in the theorem statement.
  2. There is a conceptual mismatch between the causal objective defined in the paper and the spurious-correlation problem it claims to address. CTIS measures the intervention effect of a visual token on the model output YY, rather than the causal relevance of that token to the correct task answer. If the model itself has already learned to rely on a spurious background feature, removing that feature may still substantially change the model output, and the feature would therefore receive a high CTIS score. I am concerned that, even if CTIS could be estimated exactly, it would not necessarily distinguish between features that are genuinely required by the task and features that the model actually relies on due to spurious correlations. This appears to deviate from the core motivation illustrated in Figure 1.
Questions for authors
  1. What target is used to compute the inference-time loss? Is it the ground-truth answer or the model's own prediction?
  2. How are text-token positions aligned across calibration samples with different query lengths? I am also curious about how the token representations are converted into scalar values for covariance estimation.
  3. In Eq. (8), what distribution is the expectation taken over? How is this expectation used to calculate the pruning score for each test instance?
  4. I am a little confused about the ablation settings in Table 2. Are the components removed one by one, or added progressively? It would be helpful to clarify what each row represents.
  5. Algorithm 1 seems to require a full-token forward pass and a backward pass before pruning. Does the 14.8 ms/sample scoring overhead reported in Table 8 include these passes and all other required computations?
  6. In Figure 6, spurious correlation strength is defined using average cosine similarity. However, some values on the horizontal axis exceed 1. Could the authors explain how this metric is calculated?
  7. What motivated the decision to use additional low-rank projector compression in the LLaVA-NeXT experiments? Have the authors tested the method without this compression? I am also concerned about whether this additional compression could affect the fairness of the comparison with other pruning methods.
Reviewer_sNYuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To tackle the computational bottlenecks of massive visual tokens in Large Vision-Language Models (LVLMs), this paper proposes CausalPrune, a training-free framework that shifts token pruning from correlational statistics to counterfactual causal effect estimation. The core problem identified is that existing attention-, gradient-, or diversity-based pruning methods mistakenly preserve background tokens due to training-data co-occurrence biases. To solve this, the authors formulate LVLM inference as a Structural Causal Model (SCM), defining token importance via the Causal Token Importance Score (CTIS) based on Pearl's do-operator. To avoid O(N)O(N) forward passes, they derive a first-order Taylor gradient approximation coupled with an offline backdoor adjustment correction to remove confounding biases, alongside a layer-wise causal mediation analysis for progressive multi-stage pruning. Extensive experiments across 5 major LVLM architectures and 17 image/video benchmarks demonstrate that CausalPrune achieves superior accuracy-efficiency tradeoffs under high compression rates without retraining.

Critical strengths

1.Compelling Conceptual Motivation: Reframing visual token pruning from observation-based correlation to counterfactual causal contribution effectively targets the background-retention flaw of prior attention-based methods.  2. Rigorous Theoretical Formulation: The derivation utilizing Softmax second-order Taylor expansion and the Frisch-Waugh-Lovell theorem rigorously proves that traditional attention scores are biased estimators due to token covariance confounding.  3. Comprehensive Empirical Evaluation: Evaluated across 5 diverse LVLM backbones (including Qwen2.5-VL and InternVL3) and 17 image/video benchmarks, showing robust performance retention even at extreme 90%+ pruning ratios.

Critical weaknesses
  1. Unproven Inference Efficiency & Missing Runtime Benchmarks: The paper claims that CausalPrune accelerates LVLM prefill, yet Algorithm 1 requires a full forward pass and a costly backward pass (∇viL\nabla_{v_i}\mathcal{L}) before token pruning. The authors should provide end-to-end wall-clock latency breakdowns (in milliseconds) and peak memory usage during inference—including the overhead of gradient computation and calibration—to prove that the method does not incur a net negative speedup compared to standard attention-based pruning (e.g., FastV).
  2. Equation Disconnect in Causal Mediation Analysis: While NDE/NIE equations (Eq. 15–17) theoretically model cascading effects over other tokens M={Hkl:k≠i}\mathbf{M} = \{H_k^l : k \neq i\}, the actual implementation score (Eq. 18) drops all cross-token terms and only computes self-state sensitivity. The authors need to supply an ablation comparing Eq. 18 against a naive gradient magnitude baseline to justify whether this complex mediation formulation actually contributes to accuracy.
  3. Ambiguous Ablation Design: Table 2 labels suggest an incremental accumulation rather than a standard "w/o" ablation, making it difficult to assess individual module contributions.
Questions for authors

See the weaknesses. I have been struggling between weak reject and weak accept for a long time, and I have decided that it is weak accept. I hope the author can add corresponding experimental results based on my review comments during the rebuttal period and update them in different colors in the article for my subsequent review and final score determination.

Reviewer_ZGojOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CausalPrune, a visual token pruning framework for large vision-language models. It models inference using a structural causal model and defines a Causal Token Importance Score (CTIS) through the change in expected output logits under token interventions. The proposed algorithm combines a gradient-times-input score, a correction based on visual-text covariance, and a layer-wise mediation score for progressive token pruning. Experiments across multiple LVLM architectures and image/video benchmarks report favorable performance-efficiency trade-offs, with component ablations and an analysis of online scoring overhead.

Critical strengths
  1. Framing token selection in terms of its effect on model outputs is a useful motivation. The proposed combination of an intervention-based objective, a visual-text correction, and layer-dependent scoring is a potentially interesting direction, even though the claimed causal guarantees require correction. This framing and combination deserve consideration as a contribution in their own right.

  2. The evaluation covers several LVLM architectures and image/video benchmarks and reports meaningful gains in some settings: for example, Table 1 gives 73.9 versus CDPruner's 71.3 average score for Qwen2.5-VL-7B at 128 tokens. Tables 8 and 10 also explicitly report scoring overhead. These results provide evidence of practical promise and should be weighed positively despite the limitations below.

Critical weaknesses

The main theoretical contributions do not hold as stated. The failures concern the quantities being estimated and the validity of the causal arguments, rather than just loose bounds.

  1. Equation (8) targets a change in expected output logits, whereas Equations (10)-(11) use a scalar loss and replace a first-order inner product by the L2 norm of an element-wise product. Even identifying the output with the loss, let Y(v)=L(v)=v_1+v_2, v=(a,a), a>0, with a zero baseline and independent text. The exact intervention effect is 2a, but GradScore is sqrt(2)*a. The Hessian and all higher-order Taylor remainders are zero, so Equation (14) fails even without cancellation. This is a specific evaluated input, not a degenerate distributional assumption. Separately, Theorem 3 claims consistency as calibration size n increases, but Equation (43) retains a Taylor bias and then introduces a second limit, Lambda -> 0, absent from the theorem. For Y(v)=L(v)=v^2 with independent text and zero correction, the target is a^2 and the score is 2a^2 for any n. More calibration data cannot remove this approximation bias. An independent additive text term can be included without changing either counterexample.

  2. Appendix B.1.1, Lemma 2 argues that conditioning on T blocks V_i <- U_I -> V_k -> H^1 -> ... -> Y. T is not on this open path, and its independence from U_I does not block it. Thus the claimed automatic validity of the adjustment set in standard LVLMs is unsupported. Equations (15)-(17) also add a pure natural direct effect and a pure natural indirect effect, omitting treatment-mediator interaction. With M(v)=v and Y(v,m)=v*m, the change from 0 to 1 gives NDE=NIE=0 under the paper's definitions, whereas the total effect is 1. Cross-world independence cannot repair this identity. These failures undermine the justification of both the covariance correction and the layer-wise mediation score; repairing the paper requires connecting a valid estimand and decomposition to the implemented algorithm.

The experiments do not isolate the proposed causal components or establish the claimed efficiency advantage under complete, matched accounting. Reported accuracy improvements are encouraging, but several issues prevent attributing them to the stated mechanisms.

  1. Appendix D.2 additionally applies rank-128 low-rank factorization to LLaVA-NeXT's MM-projector. It is unclear whether this modification is shared by the baselines or separated from token pruning in matched controls, so the latency/FLOP improvements cannot be attributed solely to CTIS. Some baseline results are imported from CDPruner; their checkpoint, preprocessing, actual token budget, evaluation protocol, and projector settings need explicit matching. Table 2 labels 62.5/62.8/63.0 as 'w/o Backdoor Correction', 'w/o Layer-wise Pruning', and 'w/o Mediation Analysis', but Section 4.3 interprets these rows as sequential additions. Leave-one-component-out and cumulative ablations support different conclusions. Full performance is 63.2 versus 62.0 for GradScore alone and 63.0 without mediation; uncertainty from calibration sampling and finite evaluation sets is needed to assess the smaller component gains.

  2. Table 8 reports 14.8+32.0=46.8 ms total prefill latency, compared with 45.4 ms for CDPruner and 43.8 ms for DivPrune. Relative to CDPruner, this is a 0.6-point POPE F1 improvement (87.8 versus 87.2) at about 3.1% higher latency. It supports a trade-off and an approximately 5.26x speedup over the 246 ms unpruned baseline, but does not establish the fastest total prefill. More fundamentally, Algorithm 1 requires a full-sequence forward pass, cached hidden states, and backpropagation before selection, plus layer-wise Jacobian/JVP computations. Tables 8/10 do not make clear how all these operations fit within the reported overhead, and whether Table 4's FLOPs cover the same computation. Counting vector operations after gradients are available as O(Nd) does not account for the transformer work needed to obtain them.

  3. Appendix D.9 defines 'spurious correlation strength' as average cosine similarity, yet Figure 6 uses values up to 5, beyond its [-1,1] range. Any rescaling needs specification; even with corrected units, visual-text alignment alone does not distinguish genuine grounding from confounding. Benchmark accuracy and this proxy therefore do not establish that CTIS estimates the intended intervention effect.

Questions for authors
  1. How are the linear counterexample to Equation (14) and the fixed quadratic bias in Theorem 3 reconciled with the stated assumptions? Please specify one estimand shared by the definition, approximation, and implementation, and identify any additional limit required beyond increasing calibration size.

  2. Which node on the displayed open backdoor path is blocked by conditioning on T? For Equations (15)-(17), is the indirect effect intended to be evaluated at the treated token value, or is an assumption excluding treatment-mediator interaction intended? Please explain how valid adjustment and a corrected decomposition justify the covariance and layer-wise scores actually used.

  3. Can you clarify the existing experiment configurations: which methods receive the rank-128 projector factorization, how imported baseline settings are matched, and whether Table 2 is a leave-one-out or cumulative ablation? Please also specify Figure 6's numerical transformation and why the resulting statistic measures confounding rather than genuine visual-text alignment.

  4. What is the actual online computation and loss target, and which operations are included in the 14.8 ms overhead and reported FLOPs? If a truncated or cached proxy is used, please explain its relationship to Algorithm 1 and the claimed guarantees.

Reviewer_yfnGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CausalPrune, a counterfactual causal token pruning framework for large vision-language models (LVLMs), which employs structural causal models, average treatment effects, gradient-based approximation, backdoor adjustment, and layer-wise causal mediation to estimate visual token importance and achieve efficient progressive pruning, with experiments conducted on multiple LVLM architectures and multimodal benchmarks.

Critical strengths
  1. Introducing counterfactual causal inference into visual token pruning provides an interesting alternative to conventional correlation-based importance estimation and offers potential for identifying genuinely task-relevant visual tokens.

  2. The combination of CTIS, gradient-based approximation, backdoor adjustment, and layer-wise causal mediation forms a relatively comprehensive framework for token importance estimation and progressive pruning.

  3. The evaluation of both multimodal performance and inference efficiency, including FLOPs and prefill latency, addresses an important challenge in the practical deployment of LVLMs.

Critical weaknesses
  1. The proposed SCM does not appear to be sufficiently justified in terms of the causal relationships among visual tokens, textual representations, intermediate hidden states, and model outputs.

  2. Removing a visual token may simultaneously alter the input sequence structure, attention distributions, and subsequent hidden representations. Therefore, it is unclear whether CTIS captures the genuine causal contribution of a token or the combined effects of these structural changes.

  3. The manuscript does not sufficiently explain how the assumptions required for identifying individual token-level ATEs are satisfied, particularly given the strong dependencies and interactions among visual tokens.

  4. The first-order approximation of CTIS is central to the proposed framework, but its approximation error and ranking consistency under nonlinear transformations and strong token interactions require more rigorous theoretical analysis.

  5. The identification and estimation of latent cross-modal confounders are not sufficiently clear. It remains uncertain whether the proposed adjustment can effectively eliminate confounding effects and produce reliable causal estimates.

  6. The decomposition into direct and indirect effects across transformer layers requires clearer identification assumptions and theoretical justification, especially considering the complex interactions and dependencies among intermediate hidden states.

  7. Improvements in downstream performance and computational efficiency do not necessarily demonstrate the superiority of causal importance estimation. More direct counterfactual experiments are needed to verify whether CausalPrune can distinguish causally relevant tokens from spuriously correlated ones.

  8. The generalizability of the proposed method across different prompts, image resolutions, pruning ratios, and multimodal reasoning scenarios remains insufficiently established, particularly under the extremely high pruning ratio of 94.4%.

Questions for authors
  1. What assumptions are required to identify the ATE of an individual visual token in the proposed SCM? How are dependencies and interactions among visual tokens addressed when establishing causal identifiability?

  2. How is the intervention on an individual visual token formally defined? How do the authors distinguish the causal contribution of the token from the structural changes in the input sequence and attention distributions induced by its removal?

  3. How are latent cross-modal confounders identified and estimated in practice? What empirical evidence supports the effectiveness of the proposed backdoor adjustment?

  4. What assumptions are necessary for the direct and indirect causal effect decomposition across transformer layers? How are interactions among tokens and intermediate hidden states handled?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.