acceptodds
Under review as a conference paper at ICLR 2027

Representation Dynamics Reveal Semantic Saliency and Similarity for Visual Token Pruning in MLLMs

Abstract

Multimodal large language models (MLLMs) incur high inference latency from long visual token sequences. Existing pruning methods commonly use attention maps or output features to estimate token importance or redundancy. Several recent approaches also exploit representation changes, but when and how these changes reflect foreground saliency and semantic consistency remain insufficiently understood. We analyze visual token representation dynamics across encoder depth and uncover two findings. First, the relationship between token update magnitudes and foreground saliency is layer-dependent: large token updates concentrate on foreground regions in two depth intervals, separated by several sink-dominated layers at intermediate depths. Second, similarities between token update directions better distinguish same-class from different-class tokens than those between encoder output features. Building on these findings, we propose MSDG-Prune, a training-free method that uses update magnitudes and directions to preserve salient and diverse visual information. Specifically, we group tokens by update-direction similarity and use query-weighted saliency derived from update magnitudes across a chosen depth window for group-wise token pruning. Extensive experiments across four MLLMs demonstrate the effectiveness and generalizability of MSDG-Prune. On LLaVA-NeXT, it retains 91.9% of uncompressed performance on average with only 5.6% of visual tokens, while achieving a 7.8× prefilling speedup. We will release the source code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Multimodal Large Language Models, Visual Token Pruning, Training-Free Method

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DUqEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MSDG-Prune uses update magnitudes to estimate visual-token saliency and update directions to form semantic groups. Query relevance guides token selection. The training-free method is evaluated across four MLLM backbones.

Critical strengths

The analysis offers useful insights into visual-token dynamics. The method requires no additional training and shows competitive average performance across several MLLMs.

Critical weaknesses
  1. On NeXT-7B at 160 tokens, the average gain is largely driven by POPE, while four of seven tasks underperform PruneSID (Table 2). This weakens the claim that the method preserves visual evidence more broadly.

  2. The comparison with EvoCut, a closely related method using representation dynamics, is limited to LLaVA-1.5. Its absence from the NeXT and Qwen experiments, despite supporting both models, leaves the benefit of MSDG-Prune’s specific design over existing dynamic pruning methods insufficiently established.

  3. The reported 7.8× speedup applies to prefilling, while the scope of “Inference Time” in Table 5 remains unclear. Without explicit timing boundaries, generation lengths, and a breakdown covering vision encoding, pruning, and decoding, the actual end-to-end benefit is difficult to verify.

Questions for authors
  1. On NeXT-7B at 160 tokens, the average gain is mainly driven by POPE, while four of seven tasks score below PruneSID (Table 2). How does this support the broader claim of preserving visual evidence?

  2. EvoCut is missing from the NeXT and Qwen comparisons, although its original paper evaluates both models. Can the authors provide comparisons under matched settings to establish the benefit over this closely related method?

  3. The timing boundaries in Table 5 are unclear. Does “Inference Time” include vision encoding, pruning, and decoding? Please clarify the generation length and provide a breakdown of end-to-end latency.

4 comments
AuthorsOpenReview ↗

Response to Reviewer DUqE (1/3)

Thank you for reviewing our paper and for your valuable feedback. Below, we address your concerns point by point and we will incorporate the new results and corresponding clarifications into the revised manuscript. We would appreciate it if you could let us know whether your concerns are addressed by our response.

Question 1: Concerns about the generality of performance gains.

We also observed that the average gain on LLaVA-NeXT-7B at 160 tokens is mainly driven by POPE (Table 2), while our method and PruneSID achieve comparable overall performance across the other six benchmarks. The 160-token setting highlights the challenge of maintaining accurate object-presence recognition. On POPE, our method achieves an F1 score of 87.1, outperforming PruneSID (76.9) by 10.2 percentage points and VisionZip (74.8) by 12.3 percentage points. POPE asks whether a named object is present in the image, making the query directly informative about which visual evidence to retain. Our method combines query relevance with visual saliency to prioritize evidence for that object, while pruning within groups of similar tokens to reduce redundancy.

The broader claim is supported by results across additional models and tasks. The Table DUqE-1. summarizes the shared evaluation settings from the main experiments and Appendix, covering six models at three retention ratios. MSDG-Prune achieves higher aggregate RelAcc. than PruneSID in 17 of the 18 settings and matches it in the remaining one. The gains also extend beyond POPE. Across the 18 settings, excluding POPE yields a mean relative score improvement of 0.91% over PruneSID, computed as the arithmetic mean of (MSDG-Prune−PruneSID)/PruneSID(\text{MSDG-Prune} - \text{PruneSID}) / \text{PruneSID} over the non-POPE benchmark-level comparisons. At 11.1% retention, our method outperforms PruneSID on five of the six non-POPE benchmarks on Mini-Gemini-7B (Table 3) and four of the six on Qwen2.5-VL-7B (Table 4). These cross-model and non-POPE results support the preservation of useful visual evidence across multiple tasks and architectures.

Table DUqE-1. Benchmark-Level and Aggregate RelAcc. Win/Tie/Loss Counts Excluding LLaVA-NeXT-13B.

Method vs. BaselineBenchmark-level Win / Tie / Loss (138 comparisons)RelAcc. Win / Tie / Loss (18 comparisons)Mean Relative Score Improvement (138 comparisons)Mean Relative Score Improvement Excluding POPE (120 comparisons)
Ours vs. PruneSID89 / 7 / 4217 / 1 / 0+1.06%+0.91%
PruneSID vs. VisionZip82 / 12 / 4415 / 0 / 3+1.11%+0.80%
Ours vs. VisionZip101 / 3 / 3418 / 0 / 0+2.17%+1.69%

Table note. Benchmark-level comparisons count each model–retention–benchmark combination separately. RelAcc. comparisons count each model–retention setting once. Mean relative score improvement is computed as the arithmetic mean of 100×(Smethod−Sbaseline)/Sbaseline100 \times (S_{\mathrm{method}} - S_{\mathrm{baseline}})/S_{\mathrm{baseline}} across all benchmark-level comparisons, including wins, ties, and losses.

AuthorsOpenReview ↗

Response to Reviewer DUqE (2/3)

Question 2: Additional Comparisons with EvoCut on LLaVA-NeXT and Qwen2.5-VL.

Thank you for suggesting a comparison with EvoCut [1], which also uses representation dynamics for token pruning. The published EvoCut results cover a different benchmark set for LLaVA-NeXT and different token retention ratios for Qwen2.5-VL. As no official implementation was publicly available, we reimplemented EvoCut and evaluated it under the same settings and token budgets as MSDG-Prune (Ours). As shown in Table DUqE-2. and Table DUqE-3, MSDG-Prune achieves higher RelAcc. than EvoCut across all six configurations, demonstrating the consistent performance advantage of our method.

Table DUqE-2. Qwen2.5-VL-7B Results.

MethodGQAMMB-enMMEPOPESQAVQAv2HRB-8kRelAcc.
Upper Bound (100%)
Baseline60.983.9231086.388.982.968.1100.0%
Token Retention: 33.3% (↓66.7%)
EvoCut57.680.5229184.287.180.15795.1%
Ours58.582.6233284.787.880.561.997.2% (+2.1 pp)
Token Retention: 22.2% (↓77.8%)
EvoCut57.379.5223583.186.578.356.493.8%
Ours57.781.222978487.378.961.996.1% (+2.3 pp)
Token Retention: 11.1% (↓88.9%)
EvoCut52.876.6215279.484.674.459.490.8%
Ours55.478.4217881.386.3755992.4% (+1.6 pp)

Table DUqE-3. LLaVA-NeXT-7B Results.

MethodGQAMMB-enMMEPOPESQAVQAv2SEED-IRelAcc.
Upper Bound (100%)
Baseline64.267.9184286.470.280.170.2100.0%
Token Retention: 22.2% (↓77.8%)
EvoCut63.266.3180486.367.677.566.497.4%
Ours63.364.2180587.66877.667.597.5% (+0.1 pp)
Token Retention: 11.1% (↓88.9%)
EvoCut60.963.7178385.866.474.762.794.6%
Ours6163.1174787.667.574.864.495.1% (+0.5 pp)
Token Retention: 5.6% (↓94.4%)
EvoCut59.462165382.266.771.355.790.3%
Ours5860.9162787.168.571.560.991.9% (+1.6 pp)

Table note. Values in parentheses indicate changes in RelAcc. relative to EvoCut at the same token retention ratio, measured in percentage points (pp). Higher is better for every reported score.

[1] H. Lu, F. Zhang, W. Jin, et al. EvoCut: Multi-Layer Evolution-Aware Visual Token Compression for Efficient Large Vision-Language Models. arXiv:2606.01756, 2026.

AuthorsOpenReview ↗

Response to Reviewer DUqE (3/3)

Question 3: Clarification of inference time and detailed analysis of end-to-end latency

  • Inference time is measured from when the input tensors are ready on the GPU to the completion of the final generated token. It includes vision encoding, visual projection, pruning, multimodal input preparation, LLM prefilling, autoregressive decoding, and other model-side overhead.
  • Output length The original latency evaluation for MSDG-Prune was conducted on POPE [1] using natural EOS stopping. POPE requires binary yes/no answers; in this setting, each response consisted of one answer token ("yes" or "no") followed by an EOS token, giving a total output length of L=2L=2.

We conducted a stage-wise timing analysis of VisionZip, PruneSID and MSDG-Prune (ours) under the same evaluation configuration. Specifically, we measured inference latency of LLaVA-NeXT-7B on a single NVIDIA A800-80GB GPU with FP16 precision, batch size 1, and greedy decoding with KV caching. The pruning budget is 160 visual tokens (5.56% retention ratio). We evaluate EOS stopping (2 generated tokens, including EOS) and fixed-length generation of 8 tokens. Table DUqE-4. show these results.

Table DUqE-4. End-to-end latency breakdown on LLaVA-NeXT-7B (ms).

MethodOutput lengthRequest pre/ post-processingOther overheadVision encodingPruningPrefillDecodingInference timeEnd-to-end latency
BaselineL=236.3411.93924.07—211.45621.27258.735295.076
VisionZipL=235.4781.83624.1961.12127.48321.07875.714111.192
PruneSIDL=236.5092.32123.80117.1527.58920.04690.907127.416
MSDG-Prune (Ours)L=237.6942.61223.8322.36227.46120.1396.395134.089
BaselineL=837.0751.9323.88—210.55145381.36418.435
VisionZipL=837.3271.92124.2951.18127.497144.742199.636236.963
PruneSIDL=838.7812.33623.85617.1527.576135.124206.042244.823
MSDG-Prune (Ours)L=838.0042.60524.1422.36827.523138.71215.346253.35

Compared with the full model, MSDG-Prune reduces prefill latency from 211.46 to 27.46 ms under EOS stopping. Including the 22.36 ms pruning cost, inference time decreases from 258.74 to 96.40 ms, and end-to-end latency decreases from 295.08 to 134.09 ms. With a fixed output length of 8 tokens, autoregressive decoding accounts for a larger share of runtime, and the end-to-end speedup decreases from 2.20× to 1.65×.

Please note that the results for VisionZip and PruneSID in this table were obtained through re-evaluation and differ slightly from those reported in the manuscript, which were taken from the corresponding original papers. We will update the manuscript with these re-evaluated results.

[1] Y. Li, Y. Du, K. Zhou, et al. Evaluating Object Hallucination in Large Vision-Language Models. In EMNLP, 2023.

Dear Reviewer DUqE,

Thank you again for carefully reviewing our paper. As the discussion period closes on November 18, we wanted to kindly follow up on our response.

Your review directly motivated several new experiments. Following your suggestions, we summarized comparisons with PruneSID across all model configurations in our manuscript. We added comparisons with our reimplementation of EvoCut under matched evaluation settings on LLaVA-NeXT-7B and Qwen2.5-VL-7B. We also clarified the definition of “Inference Time” and generation lengths used in the original manuscript and conducted new timing experiments to provided a breakdown of end-to-end latency.

We hope that these results address your concerns. Thank you!

Reviewer_VSxgOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how visual-token updates across encoder depth relate to foreground saliency and semantic similarity. It identifies foreground-enhanced intervals separated by sink-dominated layers, and finds that directions between normalized early and late states improve same-class token retrieval. MSDG-Prune filters sink tokens, clusters these update directions, and combines windowed update magnitudes with query relevance to allocate group budgets and select tokens before LLM prefilling. Across four MLLM pipelines, it achieves strong performance retention, including 91.9% on LLaVA-NeXT with 5.6% of visual tokens. The diagnostic analysis is useful, although its semantic interpretation and some evaluation details need tighter qualification.

Critical strengths
  1. The analysis goes beyond illustrative heat maps. Foreground recall is measured over 4,952 COCO images, and semantic retrieval uses explicit patch labels and interior-anchor rules. On LLaVA-1.5, update directions improve image-mean AP from 0.543 to 0.649, with gains on 162 of 171 classes.

  2. The specific use of representation dynamics is well motivated. Building on representation-change scoring [1], evolution analysis [2], and group-wise compression [3], the paper connects stage-specific saliency with normalized early-to-late directions as the grouping representation. Both signals have direct downstream ablations.

  3. The evaluation and appendix are substantial. Results cover fixed-resolution, tiled, dual-encoder, and native-resolution pipelines, with additional 13B/32B models. Grouping, sink filtering, budget allocation, update aggregation, endpoints, window width, and signal assignment are examined, including cases where alternatives match or outperform the default.

  4. The implementation is concrete and the limitations are candid. The appendix specifies deterministic centroid initialization, stopping criteria, and bounded integer quotas. Small-object and attribute-comparison failures are shown, and the authors acknowledge that adapting to new encoders requires model-specific analysis.

Critical weaknesses
  1. Does better same-class retrieval imply better redundancy estimates? Appendix B.2 evaluates within-image retrieval and requires at least eight class-labeled patches, favoring larger regions; spatial proximity or shared texture may contribute to the improvement. A distance-matched analysis that separates patches within one instance from distinct same-class instances would help connect semantic grouping to preserving counting and relational evidence.

  2. The configuration protocol could be clearer. The stage-based window rule and fixed settings are stated. Is the analysis set D in Appendix C.1 the COCO set from Appendix B.1, and which data and criteria fixed the diagnostic coordinates, threshold 50, and group count 20 before downstream evaluation?

  3. Some aggregate comparisons use different benchmark sets. Missing entries are excluded from RelAcc., so HiPrune, EvoCut, and MSDG-Prune need not be averaged over the same tasks in Table 1. Please report common-task averages and paired uncertainty from existing predictions, particularly for the 0.3–0.4-point LLaVA margins.

  4. The latency result needs a fuller accounting. The 7.8× gain concerns prefilling; Table 5 shows 2.6× total acceleration and 96 ms total latency versus 89 ms for PruneSID and 84 ms for VisionZip. A breakdown of encoder-state handling, clustering, and selection, together with batch size, output length, precision, and timing variability, would make this trade-off easier to assess.

  5. How exactly are the pruning signals merged and scaled? Appendix C.2 says Qwen's patch-level directions and saliency are aggregated across each four-patch merger, without specifying the aggregation or renormalization. Please give this operation and confirm whether the implementation follows the signed, unscaled scores si=αiuis_i = \alpha_i u_i in Eqs. (2) and (4), or adds clipping, normalization, or a softmax temperature.

  6. The generalization claim remains tied to two encoder families. LLaVA and Mini-Gemini reuse CLIP, while the larger-model evaluations retain the corresponding encoders. This supports transfer across decoder sizes and visual pipelines, but the five-stage pattern and fixed sink-coordinate procedure should be presented as observations for the tested encoders rather than a general encoder characterization.

Questions for authors

Please refer to the weaknesses shown above.

10 comments
AuthorsOpenReview ↗

Response to Reviewer VSxg (1/8)

Thank you for reviewing our paper and for your valuable feedback. Below, we address your concerns point by point and we will incorporate the new results and corresponding clarifications into the revised manuscript. We would appreciate it if you could let us know whether your concerns are addressed by our response.

Question 1: Clarification and further analysis of semantic grouping and redundancy estimation.

Thank you for raising this thoughtful question. In our manuscript, we conducted a semantic similarity analysis to investigate whether update directions (UD) better reflect semantic consistency than encoder output features (OF). To this end, we compared UD and OF in terms of same-class retrieval performance, as shown in Figure 1 and Appendix B.2 of our manuscript.

We agree that same-class retrieval alone is insufficient to establish the accuracy of redundancy estimation, particularly when considering spatial proximity and instance-level distinctions. Nevertheless, our experiments show that UD better reflects semantic consistency than OF. We therefore use UD as the representation for semantic grouping in our proposed method rather than OF. The ablation results in Table 6 further support this choice. On Qwen2.5-VL-7B, using UD instead of OF for grouping improves RelAcc. by 1.0, 1.2, and 0.6 percentage points at 33.3%, 22.2%, and 11.1% token retention, respectively, demonstrating the benefit of UD-based grouping within our method.

The factors highlighted in this question, including region size, spatial distance, and instance identity, provide useful perspectives for further assessing UD and OF as representations for semantic grouping. To examine these factors, we conducted additional comparisons of UD and OF through retrieval analyses across region sizes, distance-matched analyses of semantic consistency and instance distinction, and controlled evaluations of instance retention and downstream counting and relational performance. For the three COCO-based analyses, a patch is assigned to an instance if the instance mask covers at least 50% of its pixels. UD and OF are computed for the same set of visual tokens.

(1) Effect of Region Size.

To examine whether the retrieval improvement is driven primarily by large object regions, we conduct an additional retrieval evaluation using COCO instance masks from the COCO 2017 validation set [1], stratifying instances into three size ranges: 2–3, 4–7, and ≥8 labeled patches. For each query patch, positives are patches from the same instance, and negatives are patches from any other instance within the same image. Candidates are ranked by cosine similarity.

Table VSxg-1. Instance-level retrieval AP (↑) by Instance Size.

Patches per InstanceLLaVA UDLLaVA OFQwen UDQwen OF
2–30.568 (+0.052)0.5160.634 (+0.05)0.584
4–70.548 (+0.045)0.5030.582 (+0.054)0.528
≥80.662 (+0.034)0.6280.664 (+0.068)0.596

Table notes. Instance size is measured by the number of patches assigned to each instance. We report average precision (AP), with higher values indicating that same-instance patches tend to appear earlier in the retrieval ranking. Values in parentheses indicate the AP improvement of UD over OF. Boldface is used only to highlight UD results.

UD outperforms OF across all size groups on both models. For the smallest instances (2–3 patches), AP improves by 0.052 on LLaVA and 0.050 on Qwen, showing that the retrieval advantage extends to small instances as well.

AuthorsOpenReview ↗

Response to Reviewer VSxg (2/8)

(2) Semantic Consistency and Instance Distinction under Distance Matching.

The retrieval analyses assess how feature similarity organizes semantically related patches. To examine the contribution of spatial proximity and distinguish class-level consistency from instance identity, we divide patch pairs into three groups: same-instance pairs (AA), same-class different-instance pairs (BB), and different-class pairs (CC). We conduct distance-matched AUROC analyses for category discrimination (A∪BA \cup B vs. CC) and within-class instance discrimination (AA vs. BB). Within each distance bin, pairs are sampled on a per-image basis, and the three pair groups are balanced to equal counts. Each pair is scored by the cosine similarity between its patch representations.

Table VSxg-2. Category discrimination (A∪BA \cup B vs. CC): AUROC (↑) across spatial distance bins.

Distance (patch units)LLaVA UDLLaVA OFQwen UDQwen OF
[1, 3)0.643 (+0.007)0.6360.686 (+0.073)0.613
[3, 6)0.663 (−0.003)0.6660.683 (+0.066)0.617
[6, 10)0.653 (−0.009)0.6620.671 (+0.066)0.605
≥100.614 (+0.001)0.6130.655 (+0.079)0.576

Table VSxg-3. Within-class instance discrimination (A vs. B): AUROC (↑) across spatial distance bins.

Distance (patch units)LLaVA UDLLaVA OFQwen UDQwen OF
[1, 3)0.656 (+0.058)0.5980.644 (+0.115)0.529
[3, 6)0.576 (+0.034)0.5420.555 (+0.070)0.485
[6, 10)0.549 (+0.027)0.5220.519 (+0.062)0.457
≥100.526 (+0.028)0.4980.502 (+0.060)0.442

Table notes. Distance is measured as the Euclidean distance between patch centers on the patch grid. AUROC measures how well cosine similarity separates A∪BA \cup B from CC for category discrimination, and AA from BB for within-class instance discrimination. Higher values indicate better discrimination, with 0.5 corresponding to random guessing. Values in parentheses report UD − OF. Boldface is used only to highlight UD results.

As shown in Table VSxg-2, UD outperforms OF across all distance bins on Qwen2.5-VL, while the two representations perform comparably on LLaVA. Category AUROC for UD remains above 0.6 in every distance bin on both models, indicating that class-level semantic consistency extends beyond nearby patches. The consistent improvement over OF on Qwen2.5-VL further supports a semantic-consistency advantage after controlling for spatial distance.

As shown in Table VSxg-3, UD achieves higher AUROC than OF across all distance bins on both models. Absolute discrimination decreases with distance, reaching 0.526 on LLaVA and 0.502 on Qwen2.5-VL in the farthest bin. Thus, although UD provides better instance discrimination relative to OF, its similarity scores offer limited discrimination between same-instance and different-instance pairs at large spatial distances. This analysis characterizes the extent to which similarity reflects instance membership within a shared semantic class.

These analyses characterize semantic consistency and sensitivity to instance identity. Here, sensitivity to instance identity refers to how well feature similarity distinguishes same-instance patch pairs from pairs belonging to different instances of the same class. Better retrieval does not, by itself, imply better redundancy estimates. Patches from distinct same-class instances can provide separate counting or relational evidence, while patches within one instance can describe complementary parts. In MSDG-Prune, similarity-based grouping works together with visual saliency, budget allocation, and token selection to determine which patches are retained. We therefore examine instance coverage in a query-free setting with visual saliency and other pruning components held fixed, followed by downstream evaluations of the full method on counting and relational questions.

AuthorsOpenReview ↗

Response to Reviewer VSxg (3/8)

(3) Instance Retention and Downstream Performance.

We evaluate the contribution of UD-based grouping through two complementary experiments: a query-free instance-retention analysis under controlled visual saliency, and downstream ablations of the full method on counting and relational questions.

Query-free instance retention. Using the same instance annotations, we compare UD and OF as grouping representations in a query-free version of MSDG-Prune at token retention ratios of 33.3% and 11.1%, holding all other pruning components fixed. An instance is considered covered if at least one assigned patch is retained. We report coverage over all instances, instances with same-class counterparts in the image, and small instances containing 1–3 assigned patches.

Table VSxg-4. Instance coverage (↑) under query-free pruning.

RetentionGrouping featureAll instances covered (%)Same-class multi-instance coverage (%)Small instances covered (%)
LLaVA-1.5
33.3%OF85.382.967.3
33.3%UD85.9 (+0.6)83.5 (+0.6)69.2 (+1.9)
11.1%OF53.445.322.8
11.1%UD55.7 (+2.3)47.9 (+2.6)26.2 (+3.4)
Qwen2.5‑VL
33.3%OF84.983.365.7
33.3%UD85.3 (+0.4)83.4 (+0.1)69.5 (+3.8)
11.1%OF60.556.130.1
11.1%UD62.6 (+2.1)57.5 (+1.4)33.6 (+3.5)

UD improves all three coverage metrics on both models at both retention ratios. The gains in overall and same-class multi-instance coverage are larger at 11.1% retention, indicating a greater benefit from UD-based grouping under tighter token budgets.

Table VSxg-5. Downstream Performance. We evaluate the full MSDG-Prune method with UD or OF as the grouping representation on MME counting, MME position , and GQA relational questions at 11.1% token retention. All other pruning components are held fixed, and the unpruned model is included as a reference.

MethodRetention RatioMME count (200)MME position (200)GQA rel (100%)
Qwen2.5-VL
Baseline100%16015550.92%
MSDG-Prune (OF grouping)11.10%108.3126.740.26%
MSDG-Prune (UD grouping)11.10%128.3 (+20.0)145 (+18.3)42.95% (+2.69)
LLaVA-1.5-7B
Baseline100%155128.353.8%
MSDG-Prune (OF grouping)11.10%131.711050.9%
MSDG-Prune (UD grouping)11.10%135 (+3.3)110 (+0.0)51.4% (+0.5)

Using UD for grouping improves all three metrics on Qwen2.5-VL. On LLaVA-1.5, it improves counting and relational performance while maintaining the same position score. These results support the contribution of UD-based grouping to downstream performance within MSDG-Prune, with larger gains on Qwen2.5-VL.

Therefore, we use retrieval to assess semantic consistency and examine its relevance to redundancy estimation through controlled pruning and downstream ablations. These experiments support the contribution of UD-based grouping to instance preservation and counting and relational performance, with larger gains in overall and same-class multi-instance coverage under tighter token budgets.

[1] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, 2014.

Reviewer_YciHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MSDG-Prune, a training-free visual token pruning method for MLLMs. MLLMs typically encode an image into hundreds to thousands of visual tokens, with the number growing with image resolution, which substantially increases prefilling and GPU memory pressure. At the same time, visual tokens are highly redundant, with pronounced semantic duplication among tokens. This makes visual token pruning a meaningful research direction. Compared with conventional attention-based and feature-similarity-based work, this paper provides an insightful analysis of how token representations change layer by layer in the vision encoder, and uses these changes for semantic clustering and importance ranking of tokens. Combined with textual relevance and the removal of high-norm sink tokens, it achieves effective token compression and improves both prefilling and total inference speed. Compared with contemporary methods at the same pruning ratios, it achieves the highest or tied-highest RelAcc. at every budget. Ablation studies also support the design choices.

Critical strengths
  1. It establishes a detailed characterization of token behavior at different stages of the vision encoder and of the properties of the tokens themselves, and the method effectively avoids the encoder's inherent layer-dependent influence on token importance judgments.

  2. The experimental results are strong. The method is compared with eight comparable methods on nine datasets, and its performance is leading among comparable work.

Critical weaknesses
  1. The proposed model prunes image tokens based on the question-image pair. Although it achieves state-of-the-art performance, it does not show a significant advantage over query-agnostic methods (like PruneSID) on some models.

  2. The paper introduces textual similarity as part of token visual saliency but does not ablate the removal of textual similarity, so the marginal benefit of the textual-similarity term cannot be quantified.

Questions for authors
  1. The layer-wise phenomenon was found on two vision encoders. Has a highly similar pattern, such as the sink-dominated stage, been observed on vision encoders of other families and scales?

  2. For this question-based pruning method in real multi-turn dialogue scenarios: if pruning is not re-performed, does performance degrade? When subsequent questions are inconsistent with the initial question, how is performance affected? If re-pruning is performed, is there an efficiency advantage over not pruning?

7 comments
AuthorsOpenReview ↗

Response to Reviewer YciH (1/4)

Thank you for reviewing our paper and for your valuable feedback. Below, we address your concerns point by point and we will incorporate the new results and corresponding clarifications into the revised manuscript. We would appreciate it if you could let us know whether your concerns are addressed by our response.

Weakness 1: Concerns about performance relative to query-agnostic methods (e.g., PruneSID) under certain settings.

Thank you for raising this concern. We acknowledge that our method achieves modest gains over some query-agnostic methods such as PruneSID under certain settings (e.g. LLaVA-1.5-7B at 11.1% token retention). However, the performance margin varies across settings. Across the seven evaluated model configurations and three retention settings per model in the main experiments and Appendix, MSDG-Prune achieves higher reported RelAcc. than PruneSID in 19 of the 21 settings and matches it in the remaining two, with an average improvement of 0.94 percentage points.

The gains extend across model families, sizes, and token budgets. On LLaVA-1.5-7B, MSDG-Prune improves RelAcc. by 0.7 percentage points at 22.2% retention, while larger gains are observed on Mini-Gemini-7B and Qwen2.5-VL-7B at 11.1% retention, reaching 2.1 and 1.5 percentage points, respectively. The advantage also extends to larger models: MSDG-Prune improves RelAcc. by 2.4, 2.7, and 2.8 percentage points over PruneSID on Qwen2.5-VL-32B at 33.3%, 22.2%, and 11.1% token retention, respectively.

Overall, our method achieves better aggregate performance than PruneSID across the evaluated model families and sizes.

Table YciH-1. Benchmark-Level and Aggregate RelAcc. Win/Tie/Loss Counts Across All Evaluated Models.

Method vs. BaselineBenchmark-level Win / Tie / Loss (165 comparisons)RelAcc. Win / Tie / Loss (21 comparisons)Mean Relative Score Improvement (165 comparisons)
Ours vs. PruneSID99 / 8 / 5819 / 2 / 0+0.97%

Table note. All statistics are computed from the reported results in Tables 1–4 and Tables 9–11 in Appendix C.3. Benchmark-level comparisons count each model–retention–benchmark combination separately, yielding 165 comparisons. RelAcc. comparisons count each model–retention setting once, yielding 21 comparisons. Wins, ties, and losses are determined from the reported scores. Mean relative score improvement is calculated as the arithmetic mean of (MSDG-Prune−PruneSID)/PruneSID(\text{MSDG-Prune} - \text{PruneSID}) / \text{PruneSID} across all 165 benchmark-level comparisons, including wins, ties, and losses. The mean RelAcc. gain is calculated as the arithmetic mean of the RelAcc. differences across the 21 settings.

AuthorsOpenReview ↗

Response to Reviewer YciH (2/4)

Weakness 2: Absence of ablation on query relevance.

Thank you for this suggestion. We conducted an ablation by removing the query relevance component from MSDG-Prune and comparing the resulting variant with the full method. The evaluation was performed on Qwen2.5-VL-7B across GQA, MME, POPE, and SQA. As shown in Table YciH-2, the full method achieves RelAcc. of 98.5%, 97.4%, and 94.1% at token retention ratios of 33.33%, 22.22%, and 11.11%, respectively, outperforming the variant without query relevance at all three ratios. The gains increase as the token budget decreases, demonstrating that query relevance contributes to the performance of MSDG-Prune, with greater benefits under tighter token budgets. These results demonstrate the effectiveness of incorporating textual query information into our method. We will include this ablation in the revised manuscript.

Table YciH-2. Ablation on Query relevance on Qwen2.5-VL-7B.

MethodGQAMMEPOPESQARelAcc.
Upper Bound (100%)
Baseline60.9231086.388.9100.0%
Token Retention: 33.33% (↓66.67%)
Ours w/o Query relevance57.9231584.687.998.1%
Ours58.5233284.787.898.5% (+0.4 pp)
Token Retention: 22.22% (↓77.78%)
Ours w/o Query relevance56.4227082.587.396.2%
Ours57.722978487.397.4% (+1.2 pp)
Token Retention: 11.11% (↓88.89%)
Ours w/o Query relevance53.9212778.986.292.2%
Ours55.4217881.386.394.1% (+1.9 pp)

Table note. RelAcc. is the mean of the scores on the four listed benchmarks normalized by the corresponding uncompressed baseline scores, expressed as a percentage. Values in parentheses indicate gains in RelAcc. over the variant without query relevance at the same token retention ratio, measured in percentage points (pp).

AuthorsOpenReview ↗

Response to Reviewer YciH (3/4)

Question 1: Extended visualization of layer-wise patterns across additional vision encoder families and scales.

Thank you for asking about the consistency of this phenomenon across vision encoder families and scales. In addition to the two encoders analyzed in the original submission, we extended the layer-wise analysis to CLIP ViT-B/16 [1], DINOv2-B and DINOv2-L [2], InternViT-300M-448px-V2.5 [3] and SigLIP-SO400M [4] to examine whether similar patterns occur across additional encoder families and scales. We have included these analyses and visualizations in Appendix B.3 of the updated manuscript. The results reveal similar stage-wise patterns, with differences in the layer ranges and strength of individual stages.

In this response, we summarize the evaluated vision encoders in Table YciH-3. Specifically, InternViT-300M-448px-V2.5 and SigLIP-SO400M/14-384 exhibit the same sequence of stages observed in the main experiments: an Initialization Stage, an Early Foreground-Enhanced Stage, a Sink-Dominated Stage spanning multiple intermediate layers, a Late Foreground-Enhanced Stage, and a Background-Enhanced Stage toward the output. The layer ranges associated with these stages differ between the two encoders. The DINOv2 encoders exhibit a broadly similar progression, although neither shows a pronounced Background-Enhanced Stage. Within the CLIP family, CLIP ViT-B/16 also exhibits the stage-wise patterns observed in CLIP ViT-L/14 despite its smaller scale and fewer layers, with a particularly distinct Sink-Dominated Stage.

We also identified the diagnostic layers and feature coordinates for these additional encoders using the procedure described in Appendix C.1 to facilitate reproduction and reuse.

Table YciH-3. Summary of Vision Encoders and Sink-Stage Diagnostics.

Vision EncoderTotal LayersDiagnostic Layer Index (0-based)Diagnostic CoordinateDistinct Stage-wise Progression
Encoders in the Original Submission
CLIP-ViT-L/142412650✓
Qwen2.5-VL vision encoder3217849✓
Additional Evaluated Vision Encoders
CLIP ViT-B/16127474✓
DINOv2-base129415✓
DINOv2-large2419353✓
InternViT-300M-448px-V2.52412650✓
SigLIP-SO400M/14-38427171076✓

Table note. Total Layers denotes the number of transformer blocks. Diagnostic layer indices are zero-based, consistent with the convention used in the paper; an encoder with LL blocks therefore has indices 0,…,L−10,\ldots,L-1. Diagnostic coordinate denotes the hidden-feature dimension index used for sink identification.

These additional results extend the evidence for stage-wise representation dynamics and sink-dominated behavior beyond the two encoders originally studied, supporting their occurrence across multiple vision encoder families and scales.

[1] A. Radford, J. W. Kim, C. Hallacy, et al. Learning Transferable Visual Models From Natural Language Supervision. In ICML, 2021.

[2] M. Oquab, T. Darcet, T. Moutakanni, et al. DINOv2: Learning Robust Visual Features without Supervision. TMLR, 2024.

[3] Z. Chen, W. Wang, Y. Cao, et al. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. arXiv preprint arXiv:2412.05271, 2024.

[4] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid Loss for Language Image Pre-Training. In ICCV, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.