acceptodds
Under review as a conference paper at ICLR 2027

UG-Pruner: Unifying Different Visual Token Pruning Tasks for Multimodal Large Language Models in Graph

Abstract

Visual token pruning is a research hot spot for reducing the computation and memory overhead of multimodal large language models (MLLMs). Despite great success, existing methods often require dedicated designs for different pruning tasks, such as the image, video or 3D ones, lacking a unified and generalized paradigm. In this paper, we propose a novel and unified pruning scheme termed UG-Pruner. In principle, UG-Pruner formulates token pruning as a problem of graph modeling, and then unifies different pruning tasks via constrained graph building. In practice, we concretize the major properties of token pruning tasks into graph constraint terms, including the 2D spatial, temporal, 3D geometry, and streaming causal ones. In this case, UG-Pruner can handle different token pruning in a unified framework via configuring proper constraints to graph modeling. To validate UG-Pruner, we apply it to 4 MLLMs and conduct extensive experiments on 17 benchmarks of image, off-line video, streaming video and 3D scenes. The experimental results not only show its competitive or even better performance against task-specific SOTA methods, e.g., achieving a new SOTA score of 94.27 on ScanQA, but also witness its superior generalization on all benchmarks. For instance, UG-Pruner can retain 96.72% performance under a pruning ratio up to 90% over 4 main tasks, portraying its great potential as a unified and generalized toolbox for existing MLLMs. Our code is given in the supplement.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Visual Token Pruning, Structured Graph Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_V1abOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes UG-Pruner, a training-free visual token pruning framework that aims to unify pruning across images, offline videos, streaming videos, and multi-view 3D scenes. The method first selects candidate tokens, then constructs task-specific graphs based on semantic, 2D spatial, temporal, causal-access, and 3D geometric conditions. It estimates node importance through score propagation and selects tokens to retain. The 3D setting additionally incorporates IFPS to balance importance and spatial coverage. The authors evaluate the complete framework on four models and seventeen benchmarks, demonstrating competitive performance retention and efficiency at high pruning ratios.

Critical strengths
  1. A configurable pruning framework covering multiple input types. The method represents the structural characteristics of different tasks as graph-construction constraints, sharing the candidate selection, graph scoring, and token selection pipeline while keeping model parameters frozen. This provides a clear structure for reusing pruning implementations across different visual tasks.

  2. Broad experimental coverage. The experiments cover images, offline videos, streaming videos, and 3D scenes across four models and seventeen benchmarks. They include comparisons with task-specific methods and actual efficiency measurements, providing some support for the practical utility of the complete framework in the evaluated settings.

Critical weaknesses
  1. Table 4 mainly compares semantic-only connections with the complete task-specific configuration, without adequately isolating the effects of the 2D spatial, temporal, and 3D geometric constraints. Consequently, the results do not clearly establish whether each constraint is necessary or how much it contributes to the performance gains.

  2. The method uses OR for image and 3D tasks but AND for video tasks, without sufficiently explaining these choices or comparing alternative combinations. Semantic-threshold sensitivity experiments do not establish the validity of these combination rules.

  3. In addition to geometric graph construction, the 3D setting introduces IFPS, but the experiments do not clearly separate their contributions. It is therefore difficult to determine whether the improvements primarily come from geometric constraints, IFPS, or their combination.

Questions for authors
  1. When the candidate nodes, final token budget, and all other settings are held fixed, how does offline video performance change when the 2D spatial or temporal constraint is removed individually? Can these results establish the individual contributions of the two constraints and the benefit of combining them?

  2. What is the rationale for using OR for image and 3D tasks but AND for video tasks? Have alternative combinations been compared, along with their corresponding graph densities and graph-construction overhead?

  3. Do the Semantic and Constraints variants for ScanQA in Table 4 use the same final sampling algorithm? Can the benefit of geometric connections be isolated while holding the sampling algorithm fixed, and can the contribution of IFPS be reported separately?

4 comments
AuthorsOpenReview ↗

Response to Reviewer V1ab (Part 1 of 2)

We highly appreciate your valuable time and effort in reviewing our paper. Below we respond to you concerns point by point.

W1&Q1: Table 4 mainly compares semantic-only connections with the complete task-specific configuration, without adequately isolating the effects of the 2D spatial, temporal, and 3D geometric constraints. How does offline video performance change when the 2D spatial or temporal constraint is removed individually? Can these results establish the individual contributions of the two constraints and the benefit of combining them?

R1: Thanks for your constructive comment. Following your suggestion, we provide additional component-wise ablations on image, offline video, streaming video, and 3D scene understanding. Each variant removes one constraint from the full UG-Pruner configuration.

Image (90% pruning ratio)

SettingsMMBenchGQA
UG-Pruner68.1262.18
w/o Spatial67.2761.12

Offline video (90% pruning ratio)

SettingsMLVULongVideoBench
UG-Pruner65.6457.37
w/o Spatial63.4854.38
w/o Temporal65.5556.10

Streaming video (75% pruning ratio)

SettingsOVO Avg.StreamingBench
UG-Pruner53.4971.58
w/o Spatial51.8770.98
w/o Temporal53.2771.34
w/o Causal53.1270.78

3D scene understanding (90% pruning ratio)

SettingsScanQA CIDErMulti3DRefer [email protected]SQA3D EM
UG-Pruner94.2754.8856.46
w/o Geometric93.4854.6255.66

From these results, we can see that removing any task-specific constraint reduces performance. For offline video, removing either the spatial or temporal constraint lowers performance on both benchmarks, confirming their individual contributions and the benefit of combining them.

As noted in our paper, different pruning tasks have distinct properties, and these ablations validate the effectiveness of our graph constraints in handling these properties. We will include these results in the revised paper.

W2&Q2: What is the rationale for using OR for image and 3D tasks but AND for video tasks? Have alternative combinations been compared, along with their corresponding graph densities and graph-construction overhead?

R2: Thanks for your careful review. We mainly consider these choices from the perspectives of token pruning and graph modeling.

For image and 3D tasks, the token pruning are directed applied to all input tokens, and the AND combination are strong constraints for graph building, which will result in an excessive number of isolated nodes. These isolated nodes prevent an accurate assessment of their importance in the information propagation of our UGPruner, which further affects the final token selection. In this case, the OR combination impose looser constraint on the construction of the graph, effectively preventing the aforementioned situation from occurring.

In contrast, for offline and streaming video tasks, UG-Pruner constructs the graph frame by frame, and these tasks often involves more visual tokens than the image and 3D ones. Using OR in this setting can produce an overly dense graph, making it difficult to distinguish important tokens from redundant ones. In this case, the AND combination of constraints can obtain a better token graph.

To better respond to your concern, we supplement the OR and AND ablation below, of which results well validate our motivations. We will give these results to our new version.

Image

SettingsTextVQAMMBench
Semantic AND 2D-Spatial53.8166.92
Semantic OR 2D-Spatial*57.9067.53

Offline video

SettingsMLVULongVideoBench
Semantic OR 2D-Spatial OR Temporal64.3855.80
Semantic AND 2D-Spatial AND Temporal*65.6457.37

3D scene understanding

SettingsScanQA CIDErScanQA EMScanRefer [email protected]
Semantic AND 3D-Geometric91.8227.1249.25
Semantic OR 3D-Geometric*94.2728.0449.34
AuthorsOpenReview ↗

Response to Reviewer V1ab (Part 2 of 2)

W3&Q3:Do the Semantic and Constraints variants for ScanQA in Table 4 use the same final sampling algorithm? Can the benefit of geometric connections be isolated while holding the sampling algorithm fixed, and can the contribution of IFPS be reported separately?

R3: Thanks for your comment. Since IFPS introduces additional spatial information into token selection, the Semantic setting in Table 4 uses only semantic constraint and Top-K to evaluate semantic information alone, while the Constraints setting incorporates both 3D-geometric constraint and IFPS.

To better address your concern, we provide ablations to separately examine the contributions of geometric constraint and IFPS selection.

Selector ablation

SelectorScanQA CIDErMulti3DRefer [email protected]SQA3D EM
Top-K92.9553.9756.03
IFPS*94.2754.8856.46

Constraint ablation

ConstraintsScanQA CIDErMulti3DRefer [email protected]SQA3D EM
Semantic-only93.4854.6255.66
Semantic + 3D-Geometric*94.2754.8856.46

From these results, we can see that both geometric constraints and IFPS improve performance. 3D-Geometric constraint connect neighboring nodes across views, while IFPS additionally considers spatial distance for broader scene coverage. Following your suggestions, we will add the above discussions to our new version.

AuthorsOpenReview ↗

Awaiting Your Feedback

Dear Reviewer V1ab:

Thanks again for your great effort in reviewing our paper.

In our rebuttal, we have carefully responded to all your concerns, including the contributions of graph constraints (Q1/R1), the rationale for the OR and AND settings (Q2/R2), and the role of the sampling algorithm in 3D (Q3/R3).

With the discussion period drawing to a close, we look forward to hearing from you and would be glad to answer any further questions you may have.

Best regards,

The Authors

AuthorsOpenReview ↗

Look Forward to Your Feedback

Dear Reviewer V1ab,

Thanks again for your valuable time and effort in reviewing our paper.

In our rebuttal, we have carefully responded to all your concerns in details.

  1. Contributions of graph constraints (Q1/R1). We follow your suggestion to add component-wise ablations across image, offline video, streaming video, and 3D tasks, further confirming the effectiveness of our graph constraint designs for different pruning tasks.

  2. Rationale for OR and AND (Q2/R2). We theoretically analyze the OR and AND combinations for image and video tasks from the perspective of graph construction and node estimation. Then, the quantitative ablations of two choices are also provided to further confirm their effectiveness.

  3. Role of the sampling algorithm in 3D (Q3/R3). We follow your suggestion to supplement the controlled experimentations of 3D geometric constraint and the IFPS selection, which well validates the distinct advantages of two designs.

Moreover, in our rebuttal, we have addressed the above concerns shared by the other two reviewers, and received their positive feedback.

With the discussion period drawing to a close, we look forward to hearing from you, and we would be delighted to answer any further questions you may have.

Best regards,

The Authors

Reviewer_SecCOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UG-Pruner investigates whether a common graph-based formulation can support visual token pruning across images, offline videos, streaming videos, and multi-view 3D scenes. The method constructs a graph over candidate visual tokens, combining feature similarity with task-specific spatial, temporal, causal, or geometric constraints. It then propagates importance scores through the graph to select tokens before the language model, with an additional spatial sampling procedure for 3D inputs. The framework requires no model training. Evaluation on four multimodal models and 17 benchmarks examines task performance and inference efficiency, reporting competitive accuracy at reduced token budgets and particularly favorable results on streaming video and 3D tasks.

Critical strengths
  1. The graph formulation provides a useful common implementation for structurally different visual inputs. Expressing spatial, temporal, and geometric relationships through edge construction is a practical way to reuse the scoring machinery, and evaluation across four task families is substantially broader than an image-only pruning study.

  2. The streaming and 3D results are promising at aggressive budgets. In Table 2, OVO average improves from 50.60 for STC-Pruner to 53.49 at 75% pruning, and ScanQA CIDEr improves from 92.11 for SeGPruner to 94.27 at 90% pruning. Table 4 also shows that the complete constraint configuration improves over semantic-only affinity. These results support the value of incorporating input structure.

Critical weaknesses
  1. The experiments do not adequately isolate the contribution of constrained graph construction from the other task-specific components. The propagation rule is inherited from graph-based pruning, G-Prune is the default candidate selector, and Appendix A.1/Algorithm 1 introduce IFPS specifically for 3D, whereas the main description emphasizes top-K propagated scores. Table 4 compares all structural constraints against semantic-only affinity but does not separate individual constraints or IFPS from graph scoring. Consequently, the 3D improvement cannot yet be attributed specifically to the proposed graph constraints, and the degree of unification is less clear than the main text suggests.

  2. The aggregate evidence is not sufficient for the breadth of the generalization claim. The headline 96.72% in Table 3 averages ten selected metrics, rather than all 17 benchmarks, and combines 90% pruning in several tasks with 75% pruning in streaming. Several baseline averages use different subsets of columns because results are missing, so these averages are not directly comparable. At matched task settings in Table 2, image retention is 94.32% versus CoIN's 94.29%, and offline-video retention at 90% pruning is 94.91% versus HoliTom's 94.98%. The evidence supports useful breadth and particular streaming/3D gains, but does not yet establish a general advantage attributable to the shared method.

Questions for authors
  1. Can you isolate graph constraints and the final selector with a controlled 3D comparison of semantic-only versus semantic-plus-geometric affinity, crossed with top-K versus IFPS, while keeping candidate tokens and budgets fixed? A direct G-Prune control under the same setup would clarify the incremental contribution.

  2. Can you report cross-task aggregates on identical metric sets, with every task's pruning ratio explicit, and provide per-constraint ablations for the streaming and offline-video settings? These results would determine whether the proposed abstraction provides benefits beyond task-specific implementation choices.

  3. For Fig. 3(a), were the 3D latency measurements obtained with the same checkpoint, inputs, output lengths, and timing boundaries, including candidate selection, graph construction, propagation, and IFPS? Table 2 notes that the Geo3DPruner result uses an unreleased tuned model, so the relationship between the accuracy and latency comparisons needs clarification.

6 comments
AuthorsOpenReview ↗

Response to Reviewer SecC (Part 1 of 3)

We highly appreciate your valuable time and effort in reviewing our paper, and also thanks a lot for your constructive and encouraging comments on our work. Below, we respond to you concerns point by point.

W1Q1-a: The experiments do not adequately isolate the contribution of constrained graph construction from the other task-specific components. Consequently, the 3D improvement cannot yet be attributed specifically to the proposed graph constraints, and the degree of unification is less clear than the main text suggests.

R1-a: Thanks for your constructive comment. Following your suggestion, we provide additional component-wise ablations on the taks of image, offline video, streaming video, and 3D scene understanding. * denotes the setting of our UGPruner.

Image (90% pruning ratio)

Image tasksMMBenchGQA
Semantic-only67.2761.12
+ Spatial*68.1262.18

Offline video (90% pruning ratio)

Offline-Video tasksMLVULongVideoBench
Semantic-only63.4353.70
+ Spatial65.5556.10
+ Spatial + Temporal*65.6457.37

Streaming video (75% pruning ratio)

Streaming video tasksOVO Avg.StreamingBench
Semantic-only51.6269.98
+ Causal53.0770.62
+ Spatial + Causal53.2771.34
+ Spatial + Temporal + Causal*53.4971.58

3D scene understanding (90% pruning ratio)

3D scene understandingScanQA CIDErMulti3DRefer [email protected]SQA3D EM
Semantic-only93.4854.6255.66
+ 3D-Geometric*94.2754.8856.46

From these results, we can see that although the semantic-only constraint lies the fundamental part of token pruning for all tasks, task-wise graph constraints are still critical for achieving better performance. As noted in our paper, different pruning tasks have distinct properties, and these ablations confirm that our proposed graph constraints can effectively handle these properties.

We will follow your suggestion to supplement these ablations to our new version.

W1Q1-b: Can you isolate graph constraints and the final selector with a controlled 3D comparison of semantic-only versus semantic-plus-geometric affinity, crossed with top-K versus IFPS, while keeping candidate tokens and budgets fixed?

R1-b: Thanks for your constructive comment. Considering multi-view 3D inputs contain repeated observations of the same physical regions, we introduce 3D-geometric constraint to connect neighboring nodes across view.

However, Top-K may still retain high-scoring tokens with similar information from nearby 3D regions. We therefore use IFPS to additionally consider spatial distance for broader scene coverage. To further examine graph modeling and final selection, we supplement the following comparisons.

MethodScanQA CIDErMulti3DRefer [email protected]SQA3D EM
FastV + Top-K91.0253.0154.65
UG-Pruner + Top-K92.9553.9756.03
UG-Pruner + IFPS*94.2754.8856.46

From these results, we can see that graph constraints bring clear improvements over the pure semantic-based methods such as FastV with the same Top-K selector. IFPS further considers spatial distance during token selection, preserving more complete visual information for scene understanding. We will follow your suggestion to add the above discussions to our new version.

AuthorsOpenReview ↗

Response to Reviewer SecC (Part 2 of 3)

W1Q1-c: The propagation rule is inherited from graph-based pruning, G-Prune is the default candidate selector. A direct G-Prune control under the same setup would clarify the incremental contribution.

R1-c: Thanks for your comment. Following your suggestion, we provide direct comparisons with G-Prune across different tasks under the same setup.

MethodsGQAMMBenchMLVULongVideoBenchOVO Avg.SQA3D EM
G-Prune61.1267.2758.6954.4549.2255.14*
UG-Pruner62.1868.1265.6457.3753.4956.46

*For a fair comparison, G-Prune also uses IFPS for final token selection on 3D tasks.

From these results, we can see that UG-Pruner consistently outperforms G-Prune. For images, combining semantic and spatial constraints further improve the strong performance of G-Prune, which was designed for image pruning. Incorporating 3D geometric constraints brings similar benefits to 3D scene understanding.

The gains are more pronounced on video tasks, where spatial and temporal constraints capture relationships across frames, while causal constraints further adapt the graph to streaming video inputs.

These results validate the effectiveness of introducing task-specific graph constraints into the existing graph propagation framework.

W2&Q2-a: Several baseline averages use different subsets of columns because results are missing, so these averages are not directly comparable. Could you report averages over identical metric sets with explicit task-wise pruning ratios, and provide component-wise constraint ablations for offline and streaming video?

R2-a: Thanks for careful review. Following your suggestion, we further group methods with the same transferred tasks and report average performance retention on their original and transferred tasks separately, using identical metrics within each group.

FastV and VisionZip

MethodOriginal taskTransferred tasks
ImageOffline video + Streaming video + 3D
FastV92.91%87.76%
VisionZip92.34%88.99%
UG-Pruner96.20%96.85%

HoliTom and ForestPrune

MethodOriginal taskTransferred task
Offline video3D
HoliTom93.89%83.48%
ForestPrune92.42%72.68%
UG-Pruner95.68%91.94%

STC-Pruner

MethodOriginal taskTransferred tasks
Streaming videoOffline video + 3D
STC-Pruner95.73%81.63%
UG-Pruner101.29%94.19%

SeGPruner

MethodOriginal task
3D
SeGPruner88.55%
UG-Pruner91.94%

From these results, we can see that UG-Pruner outperforms the compared methods on their original tasks and shows even greater advantages on transferred tasks, well confirming the generalization ability of our unified graph modeling.

AuthorsOpenReview ↗

Response to Reviewer SecC (Part 3 of 3)

Q2-b:Can you provide per-constraint ablations for the streaming and offline-video settings?

R2-b: Thanks for your constructive comment. Following your suggestion, we provide additional component-wise constraint ablations on offline and streaming video.

Offline video (90% pruning)

SettingsMLVULongVideoBench
UG-Pruner65.6457.37
w/o Spatial63.4854.38
w/o Temporal65.5556.10

Streaming video (75% pruning)

SettingsOVO Avg.StreamingBench
UG-Pruner53.4971.58
w/o Spatial51.8770.98
w/o Temporal53.2771.34
w/o Causal53.1270.78

The results show that using task-specific constraints together achieves better performance, further confirming the contributions of individual constraints and their combination. We will include these results in our new version.

Q3-a: For Fig. 3(a), were the 3D latency measurements obtained with the same checkpoint, inputs, output lengths, and timing boundaries, including candidate selection, graph construction, propagation, and IFPS?

R3-a: Thanks for your careful review. Yes, they were measured by the same settings you mentioned.

We measure 3D latency using the same public Video3D-LLM checkpoint, the same test samples, and a fixed output length of 1 token. Timing includes candidate selection, graph construction, propagation, and final selection, including IFPS. We will clarify this measurement protocol in the caption.

Q3-b: Table 2 notes that the Geo3DPruner result uses an unreleased tuned model, so the relationship between the accuracy and latency comparisons needs clarification.

R3-b: Thanks for your comment. Since Geo3DPruner has not released its tuned weights, we measure the latency of its pruning pipeline using the public Video3D-LLM checkpoint and its official code project.

The table below reports each method alongside its corresponding unpruned baseline. We will clarify these model sources in the table and figure captions.

MethodsPruning ratioScanRefer [email protected] / .5Multi3DRefer [email protected] / .5Scan2Cap B-4 / CIDErScanQA CIDEr / EMSQA3D EMAvg. Ret.
Baseline†0%62.0 / 55.160.1 / 54.642.6 / 89.0104.3 / 29.860.30100.00%
Geo3DPruner†90%58.0 / 51.853.9 / 49.341.3 / 82.392.1 / 26.855.8092.00%
Baseline0%58.16 / 51.7757.35 / 52.1040.56 / 80.28101.28 / 30.0158.43100.00%
UG-Pruner90%55.63 / 49.3454.88 / 49.8638.86 / 72.9094.27 / 28.0456.4694.68%

† Results quoted from the original Geo3DPruner paper.

Reviewer_VfeuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UG-Pruner, a unified training-free visual token pruning framework for multimodal large language models (MLLMs). UG-Pruner formulates token pruning as a constrained graph modeling problem, incorporating semantic, spatial, temporal, causal, and 3D geometric constraints to accommodate different visual tasks. Token importance is estimated through graph-based information propagation, followed by token selection under a predefined budget. Experiments on four MLLMs and 17 benchmarks demonstrate competitive performance across image, offline video, streaming video, and 3D scene understanding tasks, highlighting the potential of a unified pruning framework.

Critical strengths

(1) This paper incorporates task-specific properties into a shared graph-based framework, providing a potentially general solution for different MLLMs.

(2) The proposed framework explicitly models semantic similarity, spatial proximity, temporal relationships, causality, and 3D geometric structure through graph construction constraints, which can be flexibly combined to accommodate different task requirements.

Critical weaknesses

(1) The proposed framework relies on five graph constraints, but the ablation study in Table 4 mainly compares semantic-only graph construction with the complete constraint configuration, which fails to isolate the contributions of spatial, temporal, causal, and geometric constraints.

(2) Although UG-Pruner aims to improve inference efficiency, its graph construction involves computing pairwise relationships among candidate tokens, which can introduce quadratic complexity with respect to the number of candidates. Therefore, it could become expensive for long videos or high-resolution visual inputs containing a large number of tokens. It is expected to systematically discuss graph construction overhead, peak memory consumption, or scalability with increasing input length.

Questions for authors

(1) Could the authors provide component-wise ablations to isolate the contributions of spatial, temporal, causal, and 3D geometric constraints? Additionally, what motivates the use of OR combinations for image/3D tasks but AND combinations for video tasks?

(2) How does the computational and memory overhead scale with increasing video length or visual token count?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Vfeu (Part 1 of 2)

We highly appreciate your valuable time and effort in reviewing our paper, and also thanks a lot for your constructive and encouraging comments on our work. Below, we respond to you concerns point by point.

W1&Q1-a:Could the authors provide component-wise ablations to isolate the contributions of spatial, temporal, causal, and 3D geometric constraints?

R1-a: Thanks for your constructive comment. Following your suggestion, we provide additional component-wise ablations on tasks of image, offline video, streaming video and 3D scene understanding. * denotes the setting of our UGPruner.

Image (90% pruning ratio)

Image tasksMMBenchGQA
Semantic-only67.2761.12
+ Spatial*68.1262.18

Offline video (90% pruning ratio)

Offline-Video tasksMLVULongVideoBench
Semantic-only63.4353.70
+ Spatial65.5556.10
+ Spatial + Temporal*65.6457.37

Streaming video (75% pruning ratio)

Streaming video tasksOVO Avg.StreamingBench
Semantic-only51.6269.98
+ Causal53.0770.62
+ Spatial + Causal53.2771.34
+ Spatial + Temporal + Causal*53.4971.58

3D scene understanding (90% pruning ratio)

3D scene understandingScanQA CIDErMulti3DRefer [email protected]SQA3D EM
Semantic-only93.4854.6255.66
+ 3D-Geometric*94.2754.8856.46

From these results, we can see that although the semantic-only constraint lies the fundamental part of token pruning for all tasks, task-wise graph constraints are still critical for achieving better performance. As noted in our paper, different pruning tasks have distinct properties, and these ablations confirm that our proposed graph constraints can effectively facilitate these properties.

We will follow your suggestion to supplement these ablations to our new version.

Q1-b:Additionally, what motivates the use of OR combinations for image/3D tasks but AND combinations for video tasks?

R1-b: Thanks for your careful review. The OR and AND combinations are motivated by the properties of pruning task as well as their effective graph modeling.

For image and 3D tasks, the token pruning are directed applied to all input tokens, and the AND combination are strong constraints for graph building, which will result in an excessive number of isolated nodes. These isolated nodes prevent an accurate assessment of their importance in the information propagation of our UGPruner, which further affects the final token selection. In this case, the OR combination impose looser constraint on the construction of the graph, effectively preventing the aforementioned situation from occurring.

In contrast, for offline and streaming video tasks, UG-Pruner constructs the graph frame by frame, and these tasks often involves more visual tokens than the image and 3D ones. Using OR in this setting can produce an overly dense graph, making it difficult to distinguish important tokens from redundant ones. In this case, the AND combination of constraints can obtain a better token graph.

To better respond to your concern, we supplement the OR and AND ablation below, of which results well validate our motivations. We will give these results to our new version.

Image

SettingsTextVQAMMBench
Semantic AND 2D-Spatial53.8166.92
Semantic OR 2D-Spatial*57.9067.53

Offline video

SettingsMLVULongVideoBench
Semantic OR 2D-Spatial OR Temporal64.3855.80
Semantic AND 2D-Spatial AND Temporal*65.6457.37

3D scene understanding

SettingsScanQA CIDErScanQA EMScanRefer [email protected]
Semantic AND 3D-Geometric91.8227.1249.25
Semantic OR 3D-Geometric*94.2728.0449.34
AuthorsOpenReview ↗

Response to Reviewer Vfeu (Part 2 of 2)

W2&Q2:It could become expensive for long videos or high-resolution visual inputs containing a large number of tokens. It is expected to systematically discuss graph construction overhead, peak memory consumption, or scalability with increasing input length.

R2: Thanks for your kindly suggestion. In practice, the actual overhead of our UGPruner is not significant, and its primarily because that we only perform graph construction and token selection on the visual tokens after encoder and before LLMs, which has a size of RN×dR^{N \times d}.

To better respond to your concern, we report the actual cost of our UGPruner with longer videos or higher-resolution images below.

Offline-Video Tasks

FramesComplexity of Graph ConstructionLatency of Graph Construction + Token SelectionAdditional Memory for UG-PrunerPeak Memory of MLLMsLatency of MLLMs
64→128 frames364.2→752.5 MFLOPs3.21→3.76 ms86.3→170.6 MB23.63→24.04 GB1037→2097 ms
128→256 frames752.5→1545 MFLOPs3.76→4.27 ms170.6→341.4 MB24.04→25.07 GB2097→4243 ms
256→512 frames1545→3172 MFLOPs4.27→6.08 ms341.4→684.1 MB25.07→30.32 GB4243→8693 ms

Image Tasks

ResolutionComplexity of Graph ConstructionLatency of Graph Construction + Token SelectionAdditional Memory for UG-PrunerPeak Memory of MLLMsLatency of MLLMs
512×512→1024×102429.6→447.1 GFLOPs3.01→14.55 ms29.0→195.4 MB16.04→16.68 GB88.14→264.60 ms
1024×1024→2048×2048447.1→4178 GFLOPs14.55→93.88 ms195.4→1249 MB16.68→19.44 GB264.60→1586 ms

From the above statistics, we can see that compared to the expensive inference of the entire MLLM, the overhead of our UGPruner is in fact negligible.

AuthorsOpenReview ↗

Awaiting Your Feedback

Dear Reviewer Vfeu:

Thanks again for your great effort in reviewing our paper.

In our rebuttal, we have carefully responded to all your concerns, including the component-wise ablations of graph constraints (Q1R1) and the scaling of computational and memory overhead (Q2R2).

With the discussion period drawing to a close, we look forward to hearing from you and would be glad to answer any further questions you may have.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.