acceptodds
Withdrawn from ICLR 2027

From Patching to Pruning Visual Computation in Vision–Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision–language models (VLMs) incur substantial inference cost because every visual token is processed by the attention and MLP projections of every decoder layer, even when token-specific visual computation is unnecessary at many depths. We introduce Patch-to-Prune (P2P), inspired by Mechanistic Interpretability, a training-free framework that converts activation patching from a diagnostic tool into an inference-time computation bypass. P2P performs validation-guided forward and backward layer sweeps to identify decoder regions whose visual-token projection outputs can be replaced by fixed neutral proxy activation vectors within a user-specified accuracy tolerance. Unlike conventional token-pruning methods, P2P preserves the sequence length, token order, positional information, attention mask, and residual pathways, thereby pruning computation without removing tokens or modifying the pretrained model weights. We evaluate P2P on four VLMs from the Qwen2.5-VL and LLaVA families across seven multi-modal benchmarks using mutually disjoint calibration, validation, and test partitions. P2P at a 3% tolerance retains around 94% of dense accuracy while reducing FLOPs by 55%. Beyond these efficiency gains, our layer-wise analysis suggests that visual processing in VLMs is non-uniformly distributed across decoder depth: early and late layers often require little token-specific visual computation, whereas intermediate layers appear to perform most task-relevant visual integration, enabling later reasoning to rely largely on visual information already embedded in shared residual and textual representations. This makes P2P both an efficient inference framework and a causal lens into visual information processing in VLMs.

Primary area:
interpretability and explainable AI
Keywords:
Activation Patching, Mechanistic Interpretability, Vision Language Models, Token Pruning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vyvJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Patch-to-Prune (P2P), a training-free acceleration framework for VLMs. P2P uses activation patching on small calibration/validation sets to find early and late decoder layers where visual-token projection outputs can be replaced by cached mean activations. At inference, it bypasses these computations while preserving all tokens and the original sequence structure. Contributions include a tolerance-controlled layer-selection procedure, a computation-bypass mechanism, and optional LoRA finetuning. Experiments on Qwen2.5-VL and LLaVA across seven benchmarks show ~60% FLOPs and ~22% latency reduction at 3% tolerance, retaining ~94% dense accuracy.

Critical strengths
  1. Operationalizes activation patching as an inference-time computation bypass, preserving tokens and sequence structure. This offers a practical alternative to irreversible token pruning, despite limited conceptual novelty.
  2. Training-free, tolerance-controlled design enables easy integration and interpretable accuracy-efficiency trade-offs. It avoids architecture changes and parameter updates, making deployment straightforward.
Critical weaknesses
  1. Novelty of motivation is limited: ATP-LLaVA (CVPR 2025) already showed pruning sensitivity varies layer-wise (shallow sensitive, deep robust); DUET-VLM (CVPR 2025) confirmed late-layer vision tokens are redundant. Layer-wise sensitivity for pruning is well-established.
  2. Layer selection lacks principled design. Mean-activation replacement ignores cross-attention strength as a routing signal. Per-model exhaustive sweeps on 100+100 examples are impractical for a training-free method claiming plug-and-play usability.
  3. Baselines are outdated (FastV, PyramidDrop, both pre-2025) and too few. No evaluation on video, long-context, or multi-turn dialogue. Only two similar decoder-only model families (Qwen, LLaVA) — generalization across architectures remains unverified.
Questions for authors

Please see weaknesses.

Reviewer_UhiYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Patch-to-Prune (P2P) replaces the outputs of attention or MLP projections for selected visual tokens with cached, modality-conditioned mean activations. This approach preserves the sequence length, positional information, attention masks, and residual paths, while bypassing the projection matrix multiplications for the chosen visual tokens. The mean activations are estimated using 100 samples. Cumulative layer sweeps from the input and output ends are then evaluated on a separate set of 100 validation samples to select early and late decoder layers in which visual-token projection computations can be bypassed, subject to the specified validation-accuracy tolerance. The paper presents experiments with LLaVA and Qwen models, demonstrating reductions in actual latency, and discusses the compatibility of this activation-replacement computational graph with LoRA adaptation. In the primary results table, a configuration with a 3% accuracy tolerance threshold retains approximately 94% of the full model's accuracy across two different scales of LLaVA, while significantly reducing the reported FLOPs.

Critical strengths
  1. Intervention operations are closely linked to deployment operations. Unlike diagnostic approaches that calculate activations before applying overrides, P2P explicitly skips the projection multiplications for visual tokens. By preserving positional information and the residual stream, it functions as a distinct mechanism for reducing computational costs. Table 1 reports both actual runtime latency and FLOPs—a necessary inclusion, as these two metrics do not always improve in tandem.
  2. The paper employs non-overlapping datasets for prototype estimation, layer selection, and evaluation, and includes valuable ablation studies on proxy selection and calibration set size. It also distinguishes between frozen-model inference and optional LoRA-based recovery. Furthermore, the text transparently acknowledges that the test-set accuracy drop may exceed the selected tolerance and that latency variations may not always be monotonic, rather than claiming unconditional accuracy guarantees.
Critical weaknesses
  1. The current description does not establish whether the final deployed configuration, which skips both early and late layers, still satisfies the specified tolerance threshold. The paper evaluates the skipping ranges for early and late layers separately. However, combining them may introduce nonlinear interactions, and satisfying the threshold independently does not guarantee that the combined configuration will also satisfy it. The stopping rule is also ambiguous. If (L) denotes the position where the threshold criterion is violated for the second time, setting (L_{fwd}^* = L - 1) after two consecutive failures appears to retain the preceding position that already failed the criterion. The paper does not clarify whether the combined configuration is reevaluated on the validation set or how the method falls back when the tolerance threshold is violated.

  2. The proposed mechanism reduces projection computation but does not remove sequence positions. Retaining visual token positions means that replacing some projections with mean values does not automatically eliminate attention interactions or sequence-length-dependent memory costs. Consequently, the actual benefits depend on the relative costs of projection, attention, and memory operations, as well as image resolution and prompt length. The measured FLOPs and latency under different configurations provide useful evidence. However, without corresponding measurements, these results should not be taken as evidence of memory savings or faster long-context inference across different settings. The observed non-monotonic latency trends further highlight the need to distinguish these cases.

Questions for authors

Please address the issues identified as weaknesses.

Reviewer_P4b4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Patch-to-Prune (P2P), a training-free acceleration method for vision-language models. P2P uses activation patching to identify early and late decoder layers where visual-token projection outputs can be safely replaced with cached mean activation prototypes. By doing so, it bypasses visual-token attention and MLP projection computations in these "safe" layers while preserving the token sequence length and model architecture. Experiments demonstrate reductions in both FLOPs and latency, with only moderate accuracy degradation.

Critical strengths
  1. The paper presents an interesting perspective: accelerating VLMs by identifying where visual computation is necessary, rather than only deciding which visual tokens to remove.
  2. The method preserves the token sequence, positional structure, attention masks, and residual paths, avoiding irreversible token-removal decisions.
Critical weaknesses
  1. The observation that visual processing is concentrated in intermediate layers, while early and late layers exhibit redundancy, has been suggested by several prior works [1-3]. However, the paper presents this as a key novel finding without adequately discussing these closely related studies.

[1] VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models

[2] IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruning

[3] Cross-modal Information Flow in Multimodal Large Language Models

  1. P2P requires a separate calibration set to estimate prototypes and a validation set to select safe layers for each benchmark. Thus, the safe-layer mask is not universal and must be re-derived for a new task or deployment distribution, which limits its practicality for general-purpose inference acceleration.
Questions for authors
  1. Have you considered directly propagating visual hidden states from previous layers as an alternative to using a global mean prototype? This alternative could theoretically preserve fine-grained, token-specific information while still bypassing the current-layer visual projection computation. How does your global mean approach compare to this baseline, and why was it chosen?
  2. A critical limitation of P2P, compared to standard token pruning methods, is that it does not reduce the KV-Cache. Additionally, storing per-layer and per-module prototypes introduces extra memory overhead. Given these limitations, how do you justify the practical deployment advantage of P2P, especially in memory-constrained LLM inference scenarios?
  3. The ablations presented in Tables 4 and 5 are not evaluated under the final P2P setting. Instead, they use cumulative forward patching rather than the selected safe-layer masks. This discrepancy makes it difficult to assess the true contribution of these components in the final system. Can you provide these ablation results by running the full, actual P2P pipeline for each proxy type and reporting the final accuracy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.