UG-Pruner: Unifying Different Visual Token Pruning Tasks for Multimodal Large Language Models in Graph

Visual token pruning is a research hot spot for reducing the computation and memory overhead of multimodal large language models (MLLMs). Despite great success, existing methods often require dedicated designs for different pruning tasks, such as the image, video or 3D ones, lacking a unified and generalized paradigm. In this paper, we propose a novel and unified pruning scheme termed UG-Pruner. In principle, UG-Pruner formulates token pruning as a problem of graph modeling, and then unifies different pruning tasks via constrained graph building. In practice, we concretize the major properties of token pruning tasks into graph constraint terms, including the 2D spatial, temporal, 3D geometry, and streaming causal ones. In this case, UG-Pruner can handle different token pruning in a unified framework via configuring proper constraints to graph modeling. To validate UG-Pruner, we apply it to 4 MLLMs and conduct extensive experiments on 17 benchmarks of image, off-line video, streaming video and 3D scenes. The experimental results not only show its competitive or even better performance against task-specific SOTA methods, e.g., achieving a new SOTA score of 94.27 on ScanQA, but also witness its superior generalization on all benchmarks. For instance, UG-Pruner can retain 96.72% performance under a pruning ratio up to 90% over 4 main tasks, portraying its great potential as a unified and generalized toolbox for existing MLLMs. Our code is given in the supplement.