- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Patch-to-Prune (P2P), a training-free acceleration framework for VLMs. P2P uses activation patching on small calibration/validation sets to find early and late decoder layers where visual-token projection outputs can be replaced by cached mean activations. At inference, it bypasses these computations while preserving all tokens and the original sequence structure. Contributions include a tolerance-controlled layer-selection procedure, a computation-bypass mechanism, and optional LoRA finetuning. Experiments on Qwen2.5-VL and LLaVA across seven benchmarks show ~60% FLOPs and ~22% latency reduction at 3% tolerance, retaining ~94% dense accuracy.
Critical strengths
- Operationalizes activation patching as an inference-time computation bypass, preserving tokens and sequence structure. This offers a practical alternative to irreversible token pruning, despite limited conceptual novelty.
- Training-free, tolerance-controlled design enables easy integration and interpretable accuracy-efficiency trade-offs. It avoids architecture changes and parameter updates, making deployment straightforward.
Critical weaknesses
- Novelty of motivation is limited: ATP-LLaVA (CVPR 2025) already showed pruning sensitivity varies layer-wise (shallow sensitive, deep robust); DUET-VLM (CVPR 2025) confirmed late-layer vision tokens are redundant. Layer-wise sensitivity for pruning is well-established.
- Layer selection lacks principled design. Mean-activation replacement ignores cross-attention strength as a routing signal. Per-model exhaustive sweeps on 100+100 examples are impractical for a training-free method claiming plug-and-play usability.
- Baselines are outdated (FastV, PyramidDrop, both pre-2025) and too few. No evaluation on video, long-context, or multi-turn dialogue. Only two similar decoder-only model families (Qwen, LLaVA) — generalization across architectures remains unverified.
Questions for authors
Please see weaknesses.