- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aims to investigate visual token pruning for accelerating the inference of large vision-language models. The authors use counterfactual interventions to define the causal importance of visual tokens, and employ a gradient-based approximation to reduce the cost of causal importance estimation, together with backdoor adjustment to mitigate spurious visual-text associations. They further incorporate layer-wise causal mediation analysis to progressively remove unimportant visual tokens across multiple Transformer layers, thereby reducing FLOPs, KV cache usage, and inference latency while preserving model performance. Experiments on models such as LLaVA, Qwen2.5-VL, and InternVL show that the proposed method can maintain competitive multimodal performance even under high visual token pruning ratios, while reducing inference latency.
Critical strengths
- The research problem addressed in this paper is meaningful. Moving beyond traditional correlation-based measures of visual token importance to consider causal effects provides a potentially useful perspective for LVLM pruning.
- The paper conducts evaluations across multiple LVLM architectures and a variety of multimodal benchmarks, providing broad experimental coverage.
Critical weaknesses
- My main concern is that the paper's core causal modeling and theoretical guarantees are not sufficiently supported. (i) The proposed backdoor adjustment does not appear to be valid under the paper's own causal graph. The authors explicitly identify the backdoor path , but this path does not contain any text token . Therefore, conditioning on itself cannot block this path. The paper further argues that the independence between and justifies the blocking, but such independence does not establish the required d-separation. (ii) CTIS is defined in terms of changes in the output logits , whereas the practical GradScore is computed from a scalar loss . Moreover, the first-order Taylor term is , while the proposed GradScore is . These two quantities are generally not equivalent, so the current derivation does not directly establish GradScore as a first-order approximation to CTIS. (iii) Theorem 3 claims that as . However, the proof actually retains a non-vanishing Taylor error , and exact convergence additionally requires , which is not included in the theorem statement.
- There is a conceptual mismatch between the causal objective defined in the paper and the spurious-correlation problem it claims to address. CTIS measures the intervention effect of a visual token on the model output , rather than the causal relevance of that token to the correct task answer. If the model itself has already learned to rely on a spurious background feature, removing that feature may still substantially change the model output, and the feature would therefore receive a high CTIS score. I am concerned that, even if CTIS could be estimated exactly, it would not necessarily distinguish between features that are genuinely required by the task and features that the model actually relies on due to spurious correlations. This appears to deviate from the core motivation illustrated in Figure 1.
Questions for authors
- What target is used to compute the inference-time loss? Is it the ground-truth answer or the model's own prediction?
- How are text-token positions aligned across calibration samples with different query lengths? I am also curious about how the token representations are converted into scalar values for covariance estimation.
- In Eq. (8), what distribution is the expectation taken over? How is this expectation used to calculate the pruning score for each test instance?
- I am a little confused about the ablation settings in Table 2. Are the components removed one by one, or added progressively? It would be helpful to clarify what each row represents.
- Algorithm 1 seems to require a full-token forward pass and a backward pass before pruning. Does the 14.8 ms/sample scoring overhead reported in Table 8 include these passes and all other required computations?
- In Figure 6, spurious correlation strength is defined using average cosine similarity. However, some values on the horizontal axis exceed 1. Could the authors explain how this metric is calculated?
- What motivated the decision to use additional low-rank projector compression in the LLaVA-NeXT experiments? Have the authors tested the method without this compression? I am also concerned about whether this additional compression could affect the fairness of the comparison with other pruning methods.