acceptodds
Under review as a conference paper at ICLR 2027

MiCo: Mutual Information Coverage Optimization through Semantic Erasure Modeling for Efficient MLLM Inference

Abstract

Multimodal large language models (MLLMs) have demonstrated impressive performance in multimodal understanding, but processing large numbers of visual tokens results in high computational costs. While many methods have been proposed to reduce the number of visual tokens, most of them rely on heuristics and are prone to discarding substantial visual information during pruning, leading to degradation in model performance. In this work, by using a semantic erasure model, we derive a general mutual information coverage objective from task log-loss and propose MiCo, a training-free two-stage pruning method. MiCo first uses visual signals to select a representative candidate pool before visual tokens enter the language model, then performs task-aware subset selection within it. At each stage, suitable observable proxies instantiate the derived objective as a monotone submodular coverage function, which MiCo greedily optimizes under the token budget. MiCo is evaluated on diverse MLLMs ranging from 7B to 13B parameters across a broad range of image and video benchmarks spanning general visual reasoning, fine-grained OCR and grounding, hallucination detection, and long-video understanding. MiCo consistently achieves the best performance across nearly all evaluated models under all pruning ratios. On LLaVA-NEXT-13B, MiCo uses only 5.6% visual tokens, retains 97.5% of baseline performance, and achieves a 3.8 inference speedup. Our experiments demonstrate the effectiveness of MiCo and our mutual information coverage objective for visual token pruning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM, Efficiency, Visual Token Pruning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zF8ROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies training-free visual token pruning for multimodal large language models (MLLMs) from an information-theoretic perspective. The authors formulate token subset selection through four factors: information amount, representation reliability, task relevance, and coverage. Based on this formulation, they propose MiCo, a two-stage pruning framework. In Stage 1, MiCo performs task-agnostic selection on vision-encoder features, using feature norm and CLS/pooled visual attention to weight visual tokens while encouraging representative coverage. In Stage 2, after several LLM layers, MiCo performs query-aware refinement using contextualized visual features and text-to-visual attention, again selecting a representative subset under a fixed token budget. The resulting objective is instantiated as a weighted coverage function and optimized greedily. Experiments across multiple MLLMs, image/video benchmarks, pruning ratios, and efficiency settings show consistently strong performance and substantial inference acceleration.

Critical strengths

1. The two-stage pruning design is well motivated and natural.

It is reasonable to first remove visually redundant tokens before language–vision interaction, while retaining a relatively large candidate pool, and then perform more aggressive query-aware pruning after the model has access to the textual context. This design combines visual representativeness and task relevance in a clean way.

2. The experimental evaluation is extensive.

The paper compares against a broad set of pruning baselines across multiple MLLM architectures and many benchmarks, including general visual understanding, fine-grained, hallucination-related evaluation, and video understanding. Importantly, the paper also reports practical efficiency metrics such as latency, memory consumption, and FLOPs rather than only comparing accuracy under fixed pruning ratios.

Critical weaknesses

1. The presentation is unnecessarily theory-heavy and obscures a relatively simple algorithmic idea.

The information-theoretic formulation is mathematically reasonable under the stated assumptions and does motivate the factorized weighted-coverage structure. However, the latent quantities are not directly observable, and the executable method ultimately depends on empirically chosen proxies. The connection from task log-loss to the concrete pruning rule therefore involves several layers of abstraction, including the semantic-query model, the semantic-erasure assumption, and the observable proxy assignments. Given that a substantial portion of the method section is devoted to this derivation while the practical algorithm is comparatively simple, I find the presentation harder to follow than necessary and the theoretical framing somewhat stronger than the evidence supporting the specific proxy instantiation.

2. The theoretical proxies are not sufficiently validated, especially in Stage 1.

MiCo relies on feature norm, visual attention, and text-to-visual attention as proxies for information amount, representation reliability, and task relevance. However, the reported statistics do not provide strong support for these assumptions. In Stage 1, the per-token Spearman correlation between feature norm and the proposed information reference is only ρ=0.00699\rho=0.00699, indicating essentially no useful token-level ranking relationship, while the confidence intervals for the CLS-attention reliability test include the chance level. This is particularly concerning given prior observations that high-norm ViT tokens can correspond to representation artifacts rather than informative regions [1]. Since Stage-1 pruning is irreversible and occurs before query-aware interaction, errors at this stage cannot be recovered later. Stage 2 shows a statistically significant but still very weak association between text-to-visual attention and task relevance (ρ=0.0697\rho=0.0697).

3. Scalability to larger visual inputs and larger MLLMs is insufficiently evaluated.

The experiments cover diverse architectures, but do not systematically evaluate scaling with input resolution or model size. This is particularly important for MiCo because its coverage selector requires pairwise visual-token similarities, whose overhead grows quadratically with the number of visual tokens. A controlled evaluation with increasing image resolution/token count would clarify whether the reported speedups remain meaningful for high-resolution inputs. Similarly, the largest evaluated MLLM is 13B, leaving it unclear whether the method and its proxy choices generalize to substantially larger VLMs.

4. The video comparison could be more complete.

MiCo shows particularly strong gains on video benchmarks, but the reported baselines are mostly general visual-token pruning methods. There are video-specific approaches that explicitly exploit temporal or inter-frame redundancy, e.g., through frame similarity, key-frame selection, or cross-frame token merging/pruning. This comparison is especially relevant because MiCo performs Stage-1 pruning independently for each frame, while cross-frame redundancy is only handled later in Stage 2.

[1]. Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2024, May). Vision transformers need registers. In International conference on learning representations (Vol. 2024, pp. 2632-2652).

Questions for authors
  1. How should the pruning budget be chosen in practice? Different tasks may require very different amounts of visual information; for example, OCR/grounding and reasoning-heavy tasks may have different sensitivity to pruning. Can the authors provide a task-adaptive strategy or clearer guidance for selecting the token budget?

  2. See weakness 3

  3. See weakness 4

8 comments
AuthorsOpenReview ↗

Theoretical Explanation, Proxy Validation, Scalability, Video Comparisons, and Adaptive Budgets

We appreciate your careful review and questions on theory, proxies, scalability, and budget selection. We hope our expanded analyses and experiments address your concerns and deepen your understanding of MiCo's motivation and effectiveness.

1. Theory and practical algorithm (W1). Most current token-pruning methods use heuristic rules, motivating us to derive a general pruning objective from theory. Starting from task log-loss, we introduce a semantic-erasure model and necessary approximations to obtain a source-weighted mutual-information coverage objective (Section 3; Appendix B). For efficient single-pass estimation, we follow PACT, VisionZip/VisPruner, and FastV/SparseVLM, using norm/CLS attention/text-to-visual attention for information/reliability/relevance, respectively. We acknowledge that the derivation is involved and will streamline its presentation. In practice, these proxies draw on established pruning methods and are readily available during inference, keeping MiCo training-free, simple to implement, and easy to use.

2. Proxy validation and early pruning (W2).

  • Information amount (Figure 9). Original rho=0.00699 pools cross-image norms against position-level entropy, rather than testing within-image subset selection. Considering different rankings can still select identical Top-k sets, we therefore test set agreement on independent A/B pools. Overlaps at k=32/64/128 are 30.28/46.98/38.98% and 35.26/49.51/39.17%, versus random 6.04/12.07/24.14%. Enrichment reaches 5.02x/5.84x, supporting norm-entropy set agreement.
  • Reliability (Figure 10/Table 10). Original NeXT concordances 0.557/0.564 showed a positive trend above 0.5. New LLaVA-1.5 linear/MLP estimates 0.5741/0.5972, with 95% CIs [0.5142,0.6318]/[0.5413,0.6507], strengthen support for CLS attention's reliability role.
  • Early pruning. Task-agnostic Stage 1 retains a larger informative pool for query-aware Stage 2. Adaptive results (Figure 8; Appendix I.1) show substantial token reduction with smaller score declines, supporting redundancy and effective information retention. Appendix E.4 tests pool-size sensitivity.
  • Task relevance (Figure 3/Table 1). Following original token rho=0.0697, we conduct bin analysis and reveal that target mass rises through bins 1-9 and falls only in bin 10. This attention-shift tail (Figures 11-13) lowers full-range bin correlations to 0.5636/-0.2606. Bins covering 90.23% of positions yield rho=1.0000, r=0.7403, supporting positive bin-mean alignment.

3. Resolution and model scalability (W3, Q2; Appendices D.5-D.6). At fixed T=320 on NeXT-7B:

Input tokensVanilla msMiCo msSpeedupMiCo peak GiB
1728193.4896.952.00x13.55
2304239.9499.692.41x13.55
2880290.0299.442.92x13.55

Stage 1 covers fixed-size crops; at fixed T, Stage 2 receives about 640 candidates, limiting selector growth. Fixed 88.9% compression additionally yields 2.23x/2.57x/2.92x speedups.

With unchanged proxy forms at approximately 89% compression, larger models give:

ModelSpeedupMiCo MMEVanilla MME
NeXT-34B5.34x2001.722024.00
Qwen3-VL-32B2.02x2357.312477.72
InternVL3-38B1.92x2330.792516.57

MiCo leads all three comparisons by 105-311 MME points, retaining 98.9% on NeXT-34B.

4. Video-specific baselines (W4, Q3; Appendix H.5). We add FastVID, HoliTom, and FlashVID, which explicitly exploit temporal redundancy. Three-task means on VideoMME/MVBench/MLVU are:

TMiCoFastVIDHoliTomFlashVID
3261.5061.1759.7161.06
1658.8457.4357.7159.19

At T=32, MiCo retains 97.3% of Vanilla performance and leads VideoMME/MLVU. At T=16, it retains 93.1%, with the fastest measured complete-answer latency (835.0 ms, 2.26x) and 21.32 GiB peak memory. Stage 1 preserves per-frame candidates; Stage 2 selects over the concatenated, contextualized pool, allowing cross-frame redundancy to be covered after text interaction.

5. Adaptive token budget (Q1). We introduce MiCo-adaptive, which adapts both stages' token counts to each image-question pair using a shared information-retention threshold tau. Figure 8 evaluates tau=0.990-0.999; Appendix I.1, Figure 23 reports all 19 thresholds from 0.900 to 0.999 across ten benchmarks. As tau decreases, retained-token counts fall substantially, while relative scores generally decline much less, showing that MiCo preserves task-useful visual information under substantial compression. Thus tau provides a flexible accuracy–computation trade-off, allowing both stages to adapt their budgets to visual complexity and query relevance.

If our responses can address your questions and concerns, we would be very grateful if you would consider raising your score.

AuthorsOpenReview ↗

Complete MiCo-Adaptive Results at tau=0.990–0.999 (Q1)

We introduce MiCo-adaptive, a new adaptive-budget design that uses a shared coverage-retention threshold tau instead of a fixed token count. At each stage, selection stops when retained/full weighted coverage reaches tau. Because visual features and query-dependent attention vary across inputs, the retained count adapts to image complexity and the visual information demanded by the question. Tau provides a simple control over the accuracy–computation trade-off.

PDF: Figure 8; Appendix I.1, Figure 23. The tables below report all ten thresholds from 0.990 to 0.999 on LLaVA-1.5-7B across ten benchmarks.

1. Mean tokens: Stage 1 / Stage 2 / layer average

Vanilla: 576 tokens. Each cell averages the benchmark’s samples.

Benchmark0.9900.9910.9920.9930.9940.9950.9960.9970.9980.999
GQA158.2/50.8/74.3168.0/57.3/81.6178.9/65.4/90.2191.0/74.5/100.0204.7/85.8/111.8220.3/100.5/126.7238.6/119.7/145.7260.6/145.0/170.3288.2/180.7/204.2326.8/239.5/258.6
SQA-IMG78.4/27.0/38.384.9/30.4/42.392.4/34.6/47.2101.2/39.8/53.2111.7/46.4/60.7124.2/55.4/70.4139.8/67.3/83.2160.0/84.3/100.8187.6/110.9/127.7231.1/161.3/176.5
TextVQA144.5/55.8/75.2153.1/61.8/81.7162.7/68.8/89.3173.5/77.2/98.2185.9/87.6/109.1200.2/100.7/122.4217.1/117.5/139.3237.9/140.0/161.4264.7/172.2/192.4303.5/225.6/242.6
POPE154.2/46.2/69.8163.9/52.1/76.5174.6/59.4/84.6186.5/68.4/94.3200.1/79.8/106.2215.8/94.4/121.0234.0/113.5/139.8256.0/138.8/164.4283.8/175.1/198.9322.7/233.9/253.3
MME139.3/41.6/63.0148.3/46.8/69.0158.4/53.2/76.2169.6/61.1/84.8182.4/71.2/95.5197.2/84.2/108.9214.5/101.5/126.2235.5/125.0/149.2262.2/159.4/181.9300.5/216.0/234.5
MMB-EN107.1/31.7/48.2115.0/35.8/53.2124.1/40.9/59.1134.4/47.7/66.6146.3/56.0/75.7160.3/66.8/87.2177.2/81.4/102.4198.1/102.0/123.1225.6/132.5/152.9266.4/185.4/203.1
MMB-CN107.1/31.5/48.0115.1/35.7/53.0124.2/40.8/59.1134.5/47.6/66.6146.4/55.9/75.7160.4/66.7/87.2177.3/81.5/102.5198.2/102.2/123.2225.7/132.7/153.0266.5/185.7/203.4
SEED-IMG139.4/45.8/66.3148.5/51.4/72.6158.6/58.3/80.3170.1/66.6/89.2183.1/77.2/100.3198.2/90.5/114.1216.0/108.0/131.6237.6/131.6/154.8265.4/165.8/187.6305.2/222.2/240.4
AI2D118.9/49.3/64.6126.1/54.7/70.3134.2/60.9/77.0143.2/68.5/84.9153.7/78.0/94.5165.8/89.7/106.4180.1/104.6/121.1197.9/125.0/140.9221.1/153.5/168.3255.7/199.5/211.8
MMMU90.9/35.3/47.497.3/39.5/52.1104.6/44.4/57.6113.1/50.5/64.2122.9/58.3/72.4134.7/68.1/82.6149.1/81.2/96.0167.4/99.0/114.0192.3/125.8/140.4231.1/171.5/184.6

2. Relative performance (% of each benchmark’s Vanilla)

Benchmark0.9900.9910.9920.9930.9940.9950.9960.9970.9980.999
GQA93.6894.5794.9595.6395.9696.6096.8397.0498.0898.55
SQA-IMG99.0099.5799.2198.8699.5799.4398.5098.8698.6498.43
TextVQA97.7998.0098.0498.0698.8299.0399.3199.3698.8199.59
POPE97.9699.1299.29100.00100.25100.2599.80100.28100.45100.06
MME91.4292.7593.8794.6196.0296.3296.7996.4297.0297.51
MMB-EN93.4092.6191.8293.8095.2595.6595.6596.0497.3699.21
MMB-CN92.9293.3693.5193.9595.7297.0597.2098.23100.0098.53
SEED-IMG92.2192.5593.1093.7594.6595.3896.1096.5697.8198.51
AI2D100.00100.2399.9499.5399.6599.2499.4798.9499.1899.30
MMMU97.26100.0098.4899.70100.0099.7098.7899.0997.8797.56

Across ten benchmarks (equal weights), lowering tau from 0.999 to 0.990 reduces the mean layer-average count from 220.88 to 59.51 tokens, a 73.06% reduction, while relative performance remains 95.56%, compared with 98.72% at 0.999. These results support MiCo’s ability to retain a core set of task-useful visual tokens: adjusting tau substantially reduces computation while preserving most performance. MiCo-adaptive therefore combines a controllable retention threshold with input-dependent token budgets, allocating tokens according to image complexity and question-specific information needs. Figure 23 reports the wider sweep of all 19 thresholds (0.900–0.999).

AuthorsOpenReview ↗

Complete Proxy-Validation and Substitution Results (W2)

1. Norm–entropy agreement

Appendix C.2, Figure 9. The original token-level correlation is 0.00699; the position-mean correlation is 0.35943. These measure different associations. Pruning retains a subset, so we additionally test Top-k agreement: different full rankings can retain identical sets. Norms are ranked within each image using independent COCO-Stuff A/B pools. Overlap = intersection/k; random uses the same candidate set.

Measurek=32k=64k=128
A→B overlap30.28%46.98%38.98%
B→A overlap35.26%49.51%39.17%
Random expectation, approx.6.04%12.07%24.14%
A→B / random5.02×3.89×1.61×
B→A / random5.84×4.10×1.62×

All overlaps exceed 30%; five exceed 35%. Enrichment reaches 5.02×/5.84×, supporting stable norm–entropy set agreement.

2. CLS attention and semantic recovery

Appendix C.2, Figure 10, Table 10. Concordance compares correct/incorrect decoding under matched conditions and is averaged within and across eligible images.

MetricLinearMLP
CLS-attention concordance0.57410.5972
95% CI[0.5142, 0.6318][0.5413, 0.6507]
Eligible images / 200116 / 200133 / 200
Raw p0.015100.00090
Holm-adjusted p0.045300.00360

Both new confidence intervals lie above 0.5, addressing the concern about intervals including chance and supporting positive association with successful semantic recovery.

3. Text attention and target relevance

Figure 3, Table 1; Appendix C.2, Figures 11–13. Target mass is normalized token–target overlap; correlations use image-macro bin means.

Retained binsNPositions includedSpearman ρPearson r
1--10256100.00%0.5636-0.2606
1--923190.23%1.00000.7403
1--820680.47%1.00000.8335
1--718170.70%1.00000.8912
1--615660.94%1.00000.9414

Target mass rises through bins 1–9 and falls only in bin 10. Bins covering 90.23% of positions show strong positive alignment. Figure 13 shows bottom-row maxima outside the target across all 32 layers in the example, echoing attention shift.

4. PCA component ablations

Figure 4; Appendix I.2, Figure 24. Selected-to-full convex-hull area ratios in the illustrated example. S2 is an independent Stage-2-only diagnostic with Stage 1 disabled.

SettingStage 1Independent Stage 2
MiCo0.9320.957
Without coverage0.4900.905
Without corresponding attention0.9100.940
Without norm0.9300.953

Removing each component reduces projected coverage in this example.

5. Proxy substitutions

Table 8; Appendix E.2. Qwen2.5-VL-7B, T=128. Acc₈ includes MMB-CN and uses MME/20.

Replacement / methodAI2DPOPEHallBMMEMMB-ENMMB-CNMMStarSQA-IMGAcc₈Rel
MMTok (ICLR'26)76.384.243.32120.779.877.553.781.475.390.8%
Coverage → visual-mean similarity (both stages)70.1478.1035.801688.6975.6973.7147.2081.3168.3082.34%
Norm → hidden-state entropy (both stages)78.4784.4945.432152.6080.8478.2652.6083.0476.3592.04%
Text attention → text–visual similarity (S2)79.0885.5445.882142.0179.7378.9556.4784.2877.1392.98%
CLS attention → global-token similarity (S1)79.8384.5946.132207.7581.1078.9556.7383.3477.6393.59%
MiCo80.4485.4648.602252.3780.7679.1256.5384.5878.5194.65%

The three alternative weights exceed MMTok, showing that gains do not depend on one specific signal. MiCo’s combination performs best; the coverage replacement highlights the value of set coverage.

Reviewer_CjT2OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper introduces MiCo, a training-free method for visual token pruning in multimodal language models. Leveraging a "semantic erasure model," the authors derive a weighted coverage objective by bounding the excess Bayes log-loss. In this approach, feature norms and attention scores serve as proxies for informativeness, reliability, and relevance, while positive cosine similarity is used to gauge the degree of coverage. Token selection proceeds in two stages: first, an initial pool of visual candidates is constructed, which is subsequently refined using a "text-to-visual" attention mechanism within the decoder. The token budget is defined as an average value across the decoder layers. The experiments encompass eight image-based models and one video-based model, with a comparative analysis covering dimensions such as token budgets, component ablation, performance on additional visual tasks, and inference costs.

Critical strengths
  1. The breadth and consistency of the experimental results stand out as the paper’s most significant strengths. Across 11 out of 12 LLaVA configurations—and all 8 Qwen/InternVL configurations—MiCo achieved the best overall score among all compared methods. Even under the most stringent resource constraints for the NeXT-13B model, it retained 97.5% of the performance of the unpruned model. These results compellingly demonstrate the method’s robust applicability across diverse architectures and resource-constrained environments. Furthermore, the paper includes insightful ablation studies; for instance, with Qwen2.5-VL-7B (T = 128), MiCo-S1 achieved a score of 76.9, which improved to 78.5 when the full method was applied (for comparison, MMTok scored 75.3).

  2. The paper’s focus on real-world application performance—particularly regarding tasks prone to the loss of visual information—is highly commendable. The evaluation encompasses chart and document understanding, visual grounding, and the VTC-Bench benchmark, which is notably sensitive to downsampling. The latency measurements provided are also highly valuable: in the LLaVA-Video (T = 16) scenario, the latency for a complete response was reduced from 1855.0 ms to 804.7 ms, while still preserving 93.0% of the overall performance. The appendix further details the overhead introduced by the selector and the actual number of generated tokens—a crucial distinction, as a simple reduction in token count does not necessarily reflect actual runtime performance gains. By moving beyond evaluations based solely on general Visual Question Answering (VQA) scores and estimated FLOPs, this work offers a much more persuasive case for its practical utility.

Critical weaknesses
  1. I am not yet convinced that these experiments sufficiently substantiate the role of the "information factor" proposed in the paper. The most immediate issue concerns Figure 5 and Table 15. As noted in the appendix, some of the component ablations utilize an older implementation of the Stage-1 attention mechanism. In this version, the "no-H" configuration yields a result of 77.1, while the baseline configuration—which includes all components—also yields 77.1 (given the precision shown in the text). In contrast, the full model using the updated implementation achieves a score of 78.4. Consequently, comparing 77.1 to 78.4 does not allow for the isolation of the specific impact of "H"; yet, the main text concludes that all signals contribute to the performance. While the ablation experiments using the older implementation do show a decline in performance when "coverage" or certain attention signals are removed—thereby providing some evidence for the efficacy of these components—it remains unclear how much the "information-amount weight" actually contributes in the specific implementation used to generate the primary experimental results.

  2. At the token level, the norm–entropy correlation in Stage-1 is only 0.00699, whereas the larger value of 0.35943 is derived from averages calculated at the position level. Furthermore, the confidence intervals for the "reliability probe" include the level of random chance, and the correlations observed in Stage-2 (the "correlation metrics") are also weak. These observations cast doubt on the theoretical justifications provided for these specific weights. A consistent set of ablation experiments would help clarify the extent to which this theoretical framework actually contributes to the effectiveness of the final algorithm.

  3. The relationship between this work and the most relevant "coverage"-based methods requires a more detailed and specific comparison. The SCOPE method combines saliency with marginal coverage gain, while CoverPruner employs a query-weighted source coverage approach, backed by standard greedy algorithm guarantees. Although the authors cite both methods, they are not included in the experimental comparisons; furthermore, the discussion in the "Related Work" section fails to clearly articulate which specific differences drive the observed performance gains. Given the impressive experimental results reported in the paper, a deeper exploration of the distinctions—specifically regarding source weighting schemes, feature spaces, and the two-stage schedule—would be highly valuable.

  4. Regarding experimental evaluation, I suggest prioritizing a comparison with SCOPE, as it was publicly available prior to this submission. CoverPruner appeared more recently; therefore, even if conducting additional experiments is not feasible, a rigorous comparison of the two methods—focusing on their respective objective functions and underlying assumptions—would be very beneficial. While the comparisons with MMTok, ApET, and other methods provide substantial evidence, they do not fully clarify the specific advantages of the proposed approach when measured against the most relevant coverage-based methods.

Questions for authors
  1. Could the authors provide experimental results for the "full version," "version without the H-term," "version without the coverage term," and "version without the attention term," ensuring that the implementation used is consistent with that of the main evaluation? Please also report the paired differences without rounding; in the previous implementation, both the "full version" and the "version without the H-term" yielded a result of 77.1. The core question is: does the "information-amount weight" actually play a functional role in the final implementation?

  2. How does MiCo perform compared to SCOPE when using the same backbone network, token-counting method, and evaluation protocol? Furthermore, it is recommended to disentangle and clarify the impact of the "source-weighted objective" (as used in CoverPruner) from the gains attributed to the "two-stage pruning mechanism." This would facilitate a more precise assessment of MiCo’s specific contributions.

  3. Regarding the comparisons of latency and memory usage, what attention backends and specific attention-map implementation settings were employed? While Appendix D.1 mentions that NeXT-7B MiCo utilizes FlashAttention2, the paper does not explicitly detail the configurations used for each method. It is recommended to conduct comparisons under a unified backend environment across representative computational budgets (provided that the methods in question support such an environment). This would help clarify the extent to which observed performance differences stem from the token-pruning strategy versus specific implementation choices.

8 comments
AuthorsOpenReview ↗

Ablation Clarification, Proxy Validation, Coverage-Method Comparisons, and Backend Efficiency

We appreciate your careful review and constructive feedback on the ablations, coverage objectives, and attention backends. We have added new analyses and experiments, and hope that these further analyses and results address your questions and clarify your concerns.

1. Functional contribution of H (W1, Q1). We apologize for an editing error in the original Table 15 note: it mislabeled 77.1 as full MiCo; the experimental scores are correct. Under the same implementation, full MiCo scores 78.4 and no-H 77.1. We removed the note. In no-H, both stages set the information weight to one while retaining attention, coverage, and greedy selection.

SettingAcc7Drop from full
Full MiCo78.40.0
Without H77.11.3
Without coverage74.44.0
Without both attention signals75.72.7

Differences use displayed precision. Full MiCo also exceeds no-H on all seven tasks: AI2D 80.4/79.6, POPE 85.5/84.4, HallB 48.6/45.6, MME 2252.37/2202.81, MMB-EN 80.8/80.4, MMStar 56.5/55.3, and SQA-IMG 84.6/84.0. Thus H contributes across tasks (Figure 7; Appendix E.1, Table 20). Replacing norm by hidden-state entropy lowers eight-task Acc from 78.51 to 76.35 (Table 8).

2. Empirical proxy validation (W2). Appendix C.2 adds three complementary checks.

  • Norm-entropy (Figure 9): the original token correlation pools cross-image norms against a position-level reference. Cross-image variation and noise-sensitive fine rankings motivate testing the retained set directly: identical Top-k sets can coexist with different full rankings. Independent A/B splits give overlaps of 30.28/46.98/38.98% and 35.26/49.51/39.17% at k=32/64/128, versus approximately 6.04/12.07/24.14% for random. At k=32, enrichment is 5.02x/5.84x.
  • Reliability (Figure 10, Table 10): new LLaVA-1.5 linear/MLP concordances are 0.5741/0.5972, with 95% CIs [0.5142,0.6318]/[0.5413,0.6507], supporting positive association between CLS attention and successful category decoding.
  • Relevance (Figure 3, Table 1): bins 1-9 cover 90.23% of positions and yield rho=1.0000, r=0.7403; all ten bin means give 0.5636/-0.2606. Figures 11-13 investigate the highest-attention, low-target-overlap tail, consistent with the attention shift discussed by VisPruner.

PCA ablations additionally yield Stage-1/independent Stage-2-only hull ratios 0.932/0.957, versus 0.490/0.905 without coverage (Figure 4; Appendix I.2).

3. Coverage objectives and controlled comparisons (W3-W4, Q2). MiCo uses factorized information/reliability/relevance weights on the covered source. SCOPE multiplies a candidate's marginal coverage gain by its CLS saliency. CoverPruner also weights sources, but uses query-derived exponential weights and row-centered, nonnegative cosine affinities. MiCo instead combines its stage-dependent factors with nonnegative cosine coverage. Appendix G.4 specifies feature spaces, weight placement, and both stages of each extension.

We extend SCOPE, CoverPruner, and DivPrune to MiCo's schedule on two backbones, three budgets, and ten benchmarks. SCOPE retains aligned CLS weighting in Stage 2. Ten-task means (MME/20) are:

ModelTMiCoSCOPECoverPrunerDivPrune
LLaVA-1.5-7B12862.0361.2560.6760.85
6460.5659.7458.6758.66
3258.7657.7055.8856.42
NeXT-7B64064.0563.9062.7963.12
32063.5963.1261.7861.97
16062.4461.3959.9159.88

MiCo leads all six means (Tables 29-30). Single-stage MiCo-S1 also achieves eight wins/two ties/two losses against MMTok across 12 LLaVA settings (Appendix G.1-G.2), supporting the selector before decoder refinement.

4. Attention backends and efficiency (Q3; Appendix D.4, Table 17). We compare 11 methods using FlashAttention2 (FA2) and SDPA on NeXT-7B (FP16; layer-average budget T=320). The vision encoder and decoder layers extracting pruning attention weights use eager; other decoder layers use the specified backend.

One-token latency includes vision encoding, score extraction and selection. Full results, including memory, are in Table 17. MiCo achieves 2.92×/2.86× speedups over Vanilla under FA2/SDPA, and MiCo-S1 achieves 3.34×/3.25×, supporting our design's efficiency across backends.

MethodFA2: ms/speedupSDPA: ms/speedup
Vanilla290.02/1.00×294.09/1.00×
MiCo99.44/2.92×102.80/2.86×
MiCo-S186.81/3.34×90.50/3.25×
FastV107.01/2.71×113.20/2.60×
SparseVLM130.35/2.22×134.11/2.19×
DivPrune86.50/3.35×90.84/3.24×
VisionZip102.35/2.83×107.64/2.73×
VisPruner112.39/2.58×118.37/2.48×
HoloV102.57/2.83×108.00/2.72×
MMTok151.88/1.91×156.36/1.88×
ApET103.45/2.80×105.83/2.78×

If our effort can address your questions and concerns, we would be very grateful if you would consider raising your score.

AuthorsOpenReview ↗

Complete Experimental Results for Component Ablations, Proxy Validation, and Proxy Substitutions (W1, W2, Q1)

1. Requested ablations (W1, Q1)

Figure 7; Appendix E.1, Table 20. The original Table 15 note mislabeled 77.1 as full MiCo; it has been removed. No-H sets the information weight to one in both stages, retaining attention, coverage, and greedy selection.

SettingAI2DPOPEHallBMMEMMB-ENMMStarSQA-IMGAcc₇
MiCo80.485.548.62252.3780.856.584.678.4
no-H79.684.445.62202.8180.455.384.077.1
no-coverage77.582.640.92035.5179.955.083.074.4
no-both-attn78.185.043.02099.8879.855.083.875.7

Acc₇ averages the seven tasks above, with MME/20. MiCo exceeds no-H on all seven tasks.

Differences before display rounding, recomputed from archived per-task scores:

SettingAcc₇ from source scoresFull − setting
MiCo78.42573949204420.0
no-H77.056493098984941.3692463930592567
no-coverage74.376891769660834.048847722383371
no-both-attn75.676138420181062.7496010718631396

2. Norm–entropy agreement (W2)

Appendix C.2, Figure 9. Original token/position-mean correlations are 0.00699/0.35943. Pooled cross-image norms and fine ranking variation motivate within-image set evaluation: pruning retains a subset, and different rankings can select identical Top-k sets. Independent A/B pools; overlap = intersection/k; random uses the same candidate set.

Measurek=32k=64k=128
A→B overlap30.28%46.98%38.98%
B→A overlap35.26%49.51%39.17%
Random expectation, approx.6.04%12.07%24.14%
A→B / random5.02×3.89×1.61×
B→A / random5.84×4.10×1.62×

All overlaps exceed 30%; five exceed 35%. Enrichment reaches 5.02×/5.84× at k=32, supporting stable norm–entropy set agreement.

3. CLS attention and semantic recovery (W2)

Appendix C.2, Figure 10, Table 10. Correct/incorrect decoding is compared under matched conditions, using eligible images.

MetricLinearMLP
CLS-attention concordance0.57410.5972
95% CI[0.5142, 0.6318][0.5413, 0.6507]
Eligible images / 200116 / 200133 / 200
Raw p0.015100.00090
Holm-adjusted p0.045300.00360

Both new CIs exceed 0.5, supporting positive association with successful semantic recovery.

4. Text attention and target relevance (W2)

Figure 3, Table 1; Appendix C.2, Figures 11–13. Target mass is normalized token–target overlap. Correlations use image-macro bin means.

Retained binsNPositions includedSpearman ρPearson r
1--10256100.00%0.5636-0.2606
1--923190.23%1.00000.7403
1--820680.47%1.00000.8335
1--718170.70%1.00000.8912
1--615660.94%1.00000.9414

Target mass rises through bins 1–9 and falls only in bin 10. Bins covering 90.23% of positions show strong positive bin-mean alignment. Figure 13 shows bottom-row attention maxima outside the target across all 32 layers in the example, echoing attention shift.

5. Proxy substitutions

Table 8; Appendix E.2. Acc₈ additionally includes MMB-CN; MME/20.

Replacement / methodAI2DPOPEHallBMMEMMB-ENMMB-CNMMStarSQA-IMGAcc₈Rel
MMTok (ICLR'26)76.384.243.32120.779.877.553.781.475.390.8%
Coverage → visual-mean similarity (both stages)70.1478.1035.801688.6975.6973.7147.2081.3168.3082.34%
Norm → hidden-state entropy (both stages)78.4784.4945.432152.6080.8478.2652.6083.0476.3592.04%
Text attention → text–visual similarity (S2)79.0885.5445.882142.0179.7378.9556.4784.2877.1392.98%
CLS attention → global-token similarity (S1)79.8384.5946.132207.7581.1078.9556.7383.3477.6393.59%
MiCo80.4485.4648.602252.3780.7679.1256.5384.5878.5194.65%

Alternative weights all exceed MMTok: effectiveness does not depend on one specific signal; MiCo’s combination performs best. PCA also shows larger hull ratios for MiCo: S1/independent S2 = 0.932/0.957 versus 0.490/0.905 without coverage (Figure 4; Appendix I.2).

AuthorsOpenReview ↗

Complete Coverage-Method Comparisons under a Shared Schedule (W3, W4, Q2)

1. Objective differences

The table summarizes MiCo and the original SCOPE and CoverPruner formulations.

MethodWeighting mechanism
MiCoCovered-source weight: H^v×R^v×P^v\widehat{\mathcal H}_v\times\widehat{\mathcal R}_v\times\widehat{\mathcal P}_v (information × reliability × relevance)
SCOPECandidate CLS saliency multiplies its marginal coverage gain.
CoverPrunerInstruction-attention-derived demand weights are applied to covered source tokens.

Original methods: SCOPE, CoverPruner.

2. Complete benchmark results

Two 32-layer backbones, K=7, three budgets, ten benchmarks, with the same data, prompts, and scoring. SCOPE, CoverPruner, and DivPrune are our two-stage extensions. T denotes the layer-average budget. Each cell is ordered MiCo / SCOPE / CoverPruner / DivPrune; MiCo values are bold. Acc₁₀ uses MME/20.

PDF: Tables 29–30, Appendix G.4.

LLaVA-1.5-7B

BenchmarkT=128T=64T=32
GQA60.0 / 59.37 / 59.68 / 59.1658.1 / 57.08 / 56.66 / 57.0855.9 / 54.49 / 55.02 / 54.25
SQA-IMG69.3 / 64.85 / 69.31 / 65.6969.7 / 64.75 / 69.41 / 65.4468.7 / 66.34 / 68.96 / 65.69
TextVQA57.6 / 57.75 / 55.60 / 56.3756.4 / 56.21 / 54.28 / 54.9255.2 / 54.48 / 51.36 / 52.82
POPE85.9 / 85.86 / 85.51 / 87.2084.7 / 83.16 / 83.61 / 85.0280.9 / 77.55 / 77.48 / 79.65
MME1474.8 / 1427.75 / 1396.44 / 1446.231426.8 / 1384.90 / 1331.27 / 1346.311355.0 / 1292.60 / 1224.61 / 1308.69
MMB-EN62.4 / 63.14 / 60.20 / 61.6860.4 / 62.71 / 58.50 / 59.2859.2 / 60.82 / 55.36 / 56.79
MMB-CN56.5 / 56.79 / 54.38 / 53.8755.7 / 54.38 / 50.34 / 50.1754.0 / 53.09 / 46.99 / 47.77
SEED-IMG63.7 / 62.99 / 62.19 / 61.9661.0 / 59.73 / 59.25 / 58.8858.2 / 56.55 / 55.36 / 54.64
AI2D54.9 / 55.28 / 54.44 / 54.0254.1 / 53.79 / 54.02 / 53.2153.6 / 53.27 / 53.40 / 52.30
MMMU36.3 / 35.11 / 35.56 / 36.2234.2 / 36.33 / 34.11 / 35.3334.1 / 35.78 / 33.67 / 34.89
Acc₁₀62.03 / 61.25 / 60.67 / 60.8560.56 / 59.74 / 58.67 / 58.6658.76 / 57.70 / 55.88 / 56.42

LLaVA-NeXT-7B

BenchmarkT=640T=320T=160
GQA61.9 / 61.69 / 61.51 / 61.1561.3 / 60.49 / 60.33 / 59.9359.5 / 58.63 / 58.89 / 57.63
SQA-IMG68.2 / 67.53 / 68.32 / 67.7268.5 / 67.67 / 68.37 / 66.3968.2 / 67.23 / 67.33 / 66.58
TextVQA58.9 / 59.24 / 53.71 / 54.7158.3 / 58.54 / 52.70 / 53.4156.8 / 56.94 / 51.17 / 52.41
POPE87.3 / 87.63 / 87.34 / 87.6286.4 / 85.12 / 85.28 / 85.2783.8 / 81.78 / 80.63 / 79.60
MME1446.8 / 1443.33 / 1398.30 / 1416.131437.0 / 1413.01 / 1390.73 / 1425.521446.1 / 1361.99 / 1341.97 / 1351.33
MMB-EN65.5 / 65.46 / 64.29 / 64.7865.5 / 65.03 / 63.01 / 64.0964.1 / 64.09 / 61.82 / 62.54
MMB-CN57.7 / 56.70 / 55.41 / 54.4757.7 / 56.70 / 54.04 / 54.0456.6 / 54.73 / 51.98 / 52.06
SEED-IMG68.4 / 67.18 / 67.41 / 66.6966.4 / 65.25 / 65.02 / 64.6864.4 / 62.82 / 62.26 / 60.41
AI2D65.3 / 65.28 / 63.92 / 65.5464.8 / 64.73 / 62.82 / 63.7663.5 / 63.37 / 62.73 / 63.47
MMMU35.0 / 36.11 / 36.11 / 37.6735.1 / 37.00 / 36.67 / 36.8935.2 / 36.22 / 35.22 / 36.56
Acc₁₀64.05 / 63.90 / 62.79 / 63.1263.59 / 63.12 / 61.78 / 61.9762.44 / 61.39 / 59.91 / 59.88

MiCo achieves the highest mean in all six settings: 62.03/60.56/58.76 and 64.05/63.59/62.44. These results show the aggregate benefit of its weighted-coverage objective under the shared schedule.

Single-stage cross-check (Tables 2–3; Appendices G.1–G.2): MiCo-S1 records eight wins/two ties/two losses against MMTok across 12 LLaVA settings at displayed retention precision. Retention improves by 1.9/1.2 percentage points on Qwen2.5-VL-7B/InternVL3-8B at T=128/256.

Reviewer_XF6LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MiCo, a training-free two-stage visual token pruning method for efficient MLLM inference. The authors derive a mutual information coverage objective from task log-loss under a semantic erasure model and approximate it using feature norms, attention scores, and cosine similarity. The resulting submodular objective is optimized greedily to preserve informative and task-relevant visual tokens. Extensive experiments across multiple MLLMs and image/video benchmarks demonstrate strong performance retention and inference acceleration under aggressive pruning.

Critical strengths
  1. Principled formulation. The paper introduces a mutual information coverage perspective that jointly considers information amount, representation reliability, task relevance, and coverage. The submodular formulation provides a clear optimization procedure with a theoretical approximation guarantee.
  2. Strong empirical results. Experiments cover diverse MLLM architectures, pruning budgets, and visual understanding tasks. MiCo consistently outperforms competitive baselines, particularly under aggressive pruning, while achieving substantial inference speedups.
Critical weaknesses
  1. Limited justification of theoretical proxies. The semantic erasure model relies on simplifying independence assumptions, and the proposed proxies are not sufficiently validated as estimates of their theoretical counterparts. In particular, the near-zero correlation between feature norms and token-level entropy raises concerns about the connection between the theoretical formulation and practical implementation.
  2. Insufficient isolation of the proposed objective's contribution. MiCo benefits from a two-stage pruning schedule, but the comparisons do not fully separate the advantages of this schedule from those of the mutual information coverage objective. Additional comparisons using identical pruning schedules and matched computational budgets would strengthen the conclusions.
Questions for authors
  1. How do the authors justify using feature norms and attention scores as proxies for semantic entropy, representation reliability, and task relevance, given the weak correlations observed in Appendix C.2? Could the authors further clarify how these empirical observations support the proposed information-theoretic interpretation?
  2. Can the authors compare MiCo against existing selection objectives under the same two-stage pruning schedule and matched computational budgets? This would help determine whether the reported improvements primarily originate from the proposed mutual information coverage objective or the two-stage design.
5 comments
AuthorsOpenReview ↗

Proxy Validation, Component Contributions, and Schedule-Controlled Comparisons

We appreciate your careful review and feedback on proxy validation and the two-stage schedule. We sincerely hope our new analyses and experiments answer your questions and clarify the contributions of our objective and proxy choices.

1. Independence assumptions. Our model does not assume that visual tokens are independent. It models semantic recovery through two gates: whether a source is covered and whether its representation is reliable. Conditional on the query and geometry, we assume these gates are independent of each other and of the source variable. These assumptions define the semantic-erasure channel and enable the factorization into information amount, reliability, and coverage (Appendix B.2). Token dependencies are captured through pairwise coverage in the practical objective. We further evaluate the resulting proxy assignments and their downstream contributions in Appendices C.2 and E.

2. Proxy rationale (W1, Q1). We derive weighted mutual-information coverage from task log-loss under semantic erasure modeling. Feature norm, CLS attention, and text-to-visual attention represent information amount, reliability, and task relevance, respectively. These choices draw on PACT, VisionZip/VisPruner, and FastV/SparseVLM; Appendix C.2 evaluates their roles.

3. Norm-entropy set agreement (W1, Q1; Appendix C.2, Figure 9). Original token correlation (0.00699) pools cross-image norms against position-level entropy. Cross-image variation, noise-sensitive fine rankings, and extreme scores can weaken score-based associations. We therefore revisited the purpose of token pruning: to retain a valuable token subset, rather than merely assign token scores. Different rankings can select identical Top-k sets despite different ordering within and outside them, motivating a set-level evaluation. We rank norms within each image and measure intersection/k with the entropy-selected set. Independent COCO-Stuff A/B pools give:

Measurek=32k=64k=128
A to B overlap30.28%46.98%38.98%
B to A overlap35.26%49.51%39.17%
Random, approx.6.04%12.07%24.14%
A to B / random5.02x3.89x1.61x
B to A / random5.84x4.10x1.62x

All six overlaps exceed 30% and five exceed 35%. Greater enrichment at smaller budgets supports stable agreement with high empirical category-entropy selection.

4. CLS attention and semantic recovery (W1, Q1; Appendix C.2, Figure 10, Table 10). Under matched conditions, Linear/MLP concordances between CLS attention and category-decoding success are 0.5741/0.5972, with 95% CIs [0.5142, 0.6318]/[0.5413, 0.6507]. Higher CLS attention is associated with successful semantic recovery, supporting its reliability role.

5. Text attention and target relevance (W1, Q1; Figure 3, Table 1). Target mass measures normalized token-target-box overlap. It rises through attention-rank bins 1-9 and falls only in bin 10.

Retained binsPositionsSpearman rhoPearson r
1-10100%0.5636-0.2606
1-990.23%1.00000.7403

Bins covering 90.23% of positions exhibit strong positive bin-mean correlations. The highest-attention 9.77% tail lowers full-range correlations. Appendix C.2, Figure 13 shows maximum attention outside the target in bottom-row positions across all 32 layers in the visualized example, echoing VisPruner's attention shift. MiCo combines attention with information and coverage; covering a high-weight position reduces its subsequent coverage gains, allowing other content to be selected.

6. Component contributions (W1, Q1). PCA ablations show broader projected coverage with MiCo (Figure 4; Appendix I.2). Removing norm/coverage/both attention signals lowers Acc7 (Figure 7; Appendix E.1). Alternative proxies remain effective, while our combination performs best (Table 8; Appendix E.2).

7. Objective and two-stage design (W2, Q2). On NeXT-13B at T=640/320/160, MiCo scores 67.6/66.5/65.3, versus two-stage VScan's 66.5/64.3/60.2 (Table 2). Single-stage MiCo-S1 achieves eight wins/two ties/two losses against MMTok across 12 LLaVA settings, with retention gains of 1.9/1.2 percentage points on Qwen2.5/InternVL3 at T=128/256 (Appendix G).

We extend SCOPE, CoverPruner, and DivPrune to MiCo's pruning layer and token schedule. Ten-task mean results are below; Appendix G.4, Tables 29-30 provide more.

ModelTMiCoSCOPECoverPrunerDivPrune
LLaVA-1.5-7B12862.0361.2560.6760.85
6460.5659.7458.6758.66
3258.7657.7055.8856.42
NeXT-7B64064.0563.9062.7963.12
32063.5963.1261.7861.97
16062.4461.3959.9159.88

MiCo leads all. MiCo-S1 and the refinement-layer analyses (Appendices F/G.3) further support our objective and two-stage design.

AuthorsOpenReview ↗

Complete Experimental Results for Proxy Validation and Component Contributions (W1, Q1)

1. Norm–entropy agreement

Appendix C.2, Figure 9. Independent COCO-Stuff A/B pools; overlap = intersection/k; matched random candidate sets.

Measurek=32k=64k=128
A→B overlap30.28%46.98%38.98%
B→A overlap35.26%49.51%39.17%
Random expectation, approx.6.04%12.07%24.14%
A→B / random5.02×3.89×1.61×
B→A / random5.84×4.10×1.62×

All overlaps exceed 30%; five exceed 35%. Enrichment reaches 5.02×/5.84×, supporting stable norm–entropy set agreement.

2. CLS attention and semantic recovery

Appendix C.2, Figure 10, Table 10. Concordance compares correct/incorrect decoding under matched conditions and is averaged within and across eligible images.

MetricLinearMLP
CLS-attention concordance0.57410.5972
95% CI[0.5142, 0.6318][0.5413, 0.6507]
Eligible images / 200116 / 200133 / 200
Raw p0.015100.00090
Holm-adjusted p0.045300.00360

Both probes support a positive association with semantic recovery.

3. Text attention and target relevance

Figure 3, Table 1; Appendix C.2, Figures 11–13. Target mass is normalized token–target overlap; correlations use image-macro bin means.

Retained binsNPositions includedSpearman ρPearson r
1--10256100.00%0.5636-0.2606
1--923190.23%1.00000.7403
1--820680.47%1.00000.8335
1--718170.70%1.00000.8912
1--615660.94%1.00000.9414

Bins covering 90.23% of positions show strong positive alignment. Figure 13 shows bottom-row maxima outside the target across all 32 layers in the example, echoing attention shift.

4. Component contributions

PCA: Figure 4; Appendix I.2. S1/independent S2 hull ratios: MiCo 0.932/0.957; no coverage 0.490/0.905; no attention 0.910/0.940; no norm 0.930/0.953.

Figure 7; Appendix E.1, Table 20. Acc₇ uses seven tasks with MME/20.

SettingAI2DPOPEHallBMMEMMB-ENMMStarSQA-IMGAcc₇
all-off70.567.834.61662.2472.446.777.664.7
only-H73.981.036.61933.9079.651.382.571.7
only-cls-attn73.881.439.11975.3777.352.681.672.1
random73.780.840.02047.5377.052.182.172.6
only-both-attn75.081.941.21958.5579.552.582.172.9
no-coverage77.582.640.92035.5179.955.083.074.4
only-text-visual-attn76.782.344.32137.3678.353.483.175.0
no-both-attn78.185.043.02099.8879.855.083.875.7
only-coverage77.584.443.22142.6080.254.783.875.8
no-text-visual-attn78.285.242.62149.0980.256.383.876.3
no-H79.684.445.62202.8180.455.384.077.1
no-cls-attn79.984.646.72206.6581.156.583.177.5
MiCo80.485.548.62252.3780.856.584.678.4

The full combination achieves the highest Acc₇. Random samples uniformly; all-off selects by index among equal scores.

5. Proxy substitutions

Table 8; Appendix E.2. Qwen2.5-VL-7B, T=128. Acc₈ includes MMB-CN and uses MME/20.

Replacement / methodAI2DPOPEHallBMMEMMB-ENMMB-CNMMStarSQA-IMGAcc₈Rel
MMTok (ICLR'26)76.384.243.32120.779.877.553.781.475.390.8%
Coverage → visual-mean similarity (both stages)70.1478.1035.801688.6975.6973.7147.2081.3168.3082.34%
Norm → hidden-state entropy (both stages)78.4784.4945.432152.6080.8478.2652.6083.0476.3592.04%
Text attention → text–visual similarity (S2)79.0885.5445.882142.0179.7378.9556.4784.2877.1392.98%
CLS attention → global-token similarity (S1)79.8384.5946.132207.7581.1078.9556.7383.3477.6393.59%
MiCo80.4485.4648.602252.3780.7679.1256.5384.5878.5194.65%

The three alternative weights exceed MMTok, showing that gains do not depend on one specific signal. MiCo’s combination performs best; the coverage replacement highlights the value of set coverage.

AuthorsOpenReview ↗

Complete Experimental Results for Schedule-Controlled Comparisons (W2, Q2)

1. Different selection objectives under the same schedule

PDF: Appendix G.4, Tables 29–30. We extend SCOPE, CoverPruner, and DivPrune to MiCo’s pruning layer K=7 and per-stage token counts on two backbones, three budgets, and ten benchmarks. All use the same datasets, prompts, and scoring. T is the layer-average token budget.

Each cell is ordered MiCo / SCOPE / CoverPruner / DivPrune. Acc₁₀ averages ten benchmark scores with MME/20.

Table 1. LLaVA-1.5-7B: all benchmark results.

BenchmarkT=128T=64T=32
GQA60.0 / 59.37 / 59.68 / 59.1658.1 / 57.08 / 56.66 / 57.0855.9 / 54.49 / 55.02 / 54.25
SQA-IMG69.3 / 64.85 / 69.31 / 65.6969.7 / 64.75 / 69.41 / 65.4468.7 / 66.34 / 68.96 / 65.69
TextVQA57.6 / 57.75 / 55.60 / 56.3756.4 / 56.21 / 54.28 / 54.9255.2 / 54.48 / 51.36 / 52.82
POPE85.9 / 85.86 / 85.51 / 87.2084.7 / 83.16 / 83.61 / 85.0280.9 / 77.55 / 77.48 / 79.65
MME1474.8 / 1427.75 / 1396.44 / 1446.231426.8 / 1384.90 / 1331.27 / 1346.311355.0 / 1292.60 / 1224.61 / 1308.69
MMB-EN62.4 / 63.14 / 60.20 / 61.6860.4 / 62.71 / 58.50 / 59.2859.2 / 60.82 / 55.36 / 56.79
MMB-CN56.5 / 56.79 / 54.38 / 53.8755.7 / 54.38 / 50.34 / 50.1754.0 / 53.09 / 46.99 / 47.77
SEED-IMG63.7 / 62.99 / 62.19 / 61.9661.0 / 59.73 / 59.25 / 58.8858.2 / 56.55 / 55.36 / 54.64
AI2D54.9 / 55.28 / 54.44 / 54.0254.1 / 53.79 / 54.02 / 53.2153.6 / 53.27 / 53.40 / 52.30
MMMU36.3 / 35.11 / 35.56 / 36.2234.2 / 36.33 / 34.11 / 35.3334.1 / 35.78 / 33.67 / 34.89
Acc₁₀62.03 / 61.25 / 60.67 / 60.8560.56 / 59.74 / 58.67 / 58.6658.76 / 57.70 / 55.88 / 56.42

Table 2. LLaVA-NeXT-7B: all benchmark results.

BenchmarkT=640T=320T=160
GQA61.9 / 61.69 / 61.51 / 61.1561.3 / 60.49 / 60.33 / 59.9359.5 / 58.63 / 58.89 / 57.63
SQA-IMG68.2 / 67.53 / 68.32 / 67.7268.5 / 67.67 / 68.37 / 66.3968.2 / 67.23 / 67.33 / 66.58
TextVQA58.9 / 59.24 / 53.71 / 54.7158.3 / 58.54 / 52.70 / 53.4156.8 / 56.94 / 51.17 / 52.41
POPE87.3 / 87.63 / 87.34 / 87.6286.4 / 85.12 / 85.28 / 85.2783.8 / 81.78 / 80.63 / 79.60
MME1446.8 / 1443.33 / 1398.30 / 1416.131437.0 / 1413.01 / 1390.73 / 1425.521446.1 / 1361.99 / 1341.97 / 1351.33
MMB-EN65.5 / 65.46 / 64.29 / 64.7865.5 / 65.03 / 63.01 / 64.0964.1 / 64.09 / 61.82 / 62.54
MMB-CN57.7 / 56.70 / 55.41 / 54.4757.7 / 56.70 / 54.04 / 54.0456.6 / 54.73 / 51.98 / 52.06
SEED-IMG68.4 / 67.18 / 67.41 / 66.6966.4 / 65.25 / 65.02 / 64.6864.4 / 62.82 / 62.26 / 60.41
AI2D65.3 / 65.28 / 63.92 / 65.5464.8 / 64.73 / 62.82 / 63.7663.5 / 63.37 / 62.73 / 63.47
MMMU35.0 / 36.11 / 36.11 / 37.6735.1 / 37.00 / 36.67 / 36.8935.2 / 36.22 / 35.22 / 36.56
Acc₁₀64.05 / 63.90 / 62.79 / 63.1263.59 / 63.12 / 61.78 / 61.9762.44 / 61.39 / 59.91 / 59.88

MiCo achieves the highest mean in all six settings: 62.03/60.56/58.76 on LLaVA-1.5-7B and 64.05/63.59/62.44 on LLaVA-NeXT-7B. These comparisons directly demonstrate the benefit of MiCo’s selection objective under the shared token schedule.

2. Additional single-stage and two-stage comparisons

  • Single-stage MiCo-S1 — Tables 2–3; Appendices G.1–G.2. Against MMTok across 12 LLaVA settings, MiCo-S1 records eight wins, two ties, and two losses at displayed relative-retention precision. On Qwen2.5-VL-7B at T=128 and InternVL3-8B at T=256, retention improves by 1.9 and 1.2 percentage points, respectively.
  • Two-stage VScan — Table 2; Appendix G.1. On NeXT-13B at T=640/320/160, MiCo scores 67.6/66.5/65.3, versus VScan’s 66.5/64.3/60.2, giving displayed Acc gains of 1.1/2.2/5.1 points.

The single-stage results support the selector before decoder refinement; the shared-schedule results above establish its aggregate advantage when both stages are retained.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Summary of Our Revisions, Further Clarification of Our Motivation and Contributions, and Overall Response to All Reviewers

We appreciate all reviewers' feedback deeply and conduct extensive experiments, demonstrating MiCo's effectiveness and revising our PDF with many new analyses to answer your concerns. We sincerely hope that our efforts can be recognized and help you better understand MiCo. We remain committed to further analyses and revisions to address your concerns, and would be grateful if you would consider raising your scores in light of the strengthened evidence. Our new additions are:

  1. Proxy checks: Figure 3, Tables 1 and 8, Appendices C.2 and E.2.
  2. PCA ablations: Figure 4, Appendix I.2.
  3. Adaptive token budgets--MiCo-adaptive: Figure 8, Appendix I.1.
  4. Two-stage schedule comparisons: Tables 2-3, Appendix G.4, Tables 29-30.
  5. MiCo-S1 comparisons: Tables 2-3, Appendix G.
  6. Unified-backend efficiency: Appendix D.4.
  7. Resolution/model scaling: Appendices D.5-D.6.
  8. Video-specific baselines: Appendix H.5.
  9. Two-stage pruning visualizations: Appendix I.3.
  10. Code is included in the supplementary material for your inspection.

1. Motivation and Contributions.

  • Given that most current token-pruning methods rely on heuristic rules, we derive a general pruning objective from task loss. Through a semantic-erasure model and necessary approximations, we obtain source-weighted mutual-information coverage. We instantiate this objective using readily available signals computed during a single forward pass, making MiCo training-free, lightweight, efficient and effective. These proxies are replaceable, giving the framework flexibility for different backbones and improved estimators. MiCo achieves strong performance while reducing latency and memory, and the general objective supports further research into better proxies and more accurate task-loss estimation.

2. Proxy Rationale and Contributions.

  • Semantic entropy H(Uv∣q,g)H(U_v\mid q,g): feature norm. PACT [1] (Sec. 3.1, Eq. 2) uses hidden-state magnitude to reflect accumulated information, motivating our norm-based approximation to semantic entropy.
  • Reliability Pr⁡(Cv=1∣q,g)\Pr(C_v=1\mid q,g): CLS attention. VisionZip [2] and VisPruner [3] use CLS attention to identify important tokens. We use participation in global visual aggregation as a reliability signal.
  • Relevance Pr⁡(J=v∣Q=q)\Pr(J=v\mid Q=q): text-to-visual attention. Following FastV [4] and SparseVLM [5], we use decoder attention to weight query-relevant visual information.
  • PCA ablations (Figure 4; Appendix I.2, Figure 24). Stage-1/independent Stage-2-only hull ratios are 0.932/0.957, versus 0.490/0.905 without coverage, showing broader projected coverage in these examples.
  • Component ablations (Figure 7; Appendix E.1, Table 20). MiCo achieves Acc7=78.4; removing norm, coverage, or both attention signals gives 77.1/74.4/75.7, demonstrating complementary benefits.
  • Proxy substitutions (Table 8; Appendix E.2). Alternative information/reliability/relevance weights score 76.35/77.63/77.13, all above MMTok's 75.3, showing that its gains do not depend entirely on one specific signal. Our combination scores highest at 78.51.

3. Effectiveness of MiCo's General Pruning Objective and Two-Stage Design.

  • Two-stage VScan (Tables 2-3; Appendix G.1). MiCo leads VScan at all three budgets on NeXT-13B, reaching 65.3 versus 60.2 at T=160.
  • Single-stage MiCo-S1 (Tables 2-3; Appendices G.1-G.2). Against MMTok, MiCo-S1 has eight wins, two ties, and two losses across the LLaVA settings; retention improves by 1.9/1.2 percentage points on Qwen2.5-VL-7B/InternVL3-8B at T=128/256, supporting single-stage effectiveness.
  • Same-schedule two-stage comparison (Appendix G.4, Tables 29-30). We match two-stage SCOPE, CoverPruner, and DivPrune to MiCo's layer and token schedule. MiCo achieves the highest mean in all six model-budget settings; at T=32 on LLaVA-1.5-7B, scores are 58.76/57.70/55.88/56.42 for MiCo/SCOPE/CoverPruner/DivPrune, supporting its weighted-coverage objective.

References [1] Dhouib et al. PACT. CVPR 2025. https://arxiv.org/abs/2504.08966 [2] Yang et al. VisionZip. CVPR 2025. https://arxiv.org/abs/2412.04467 [3] Zhang et al. VisPruner: Beyond Text-Visual Attention. ICCV 2025. https://arxiv.org/abs/2412.01818 [4] Chen et al. FastV. ECCV 2024. https://arxiv.org/abs/2403.06764 [5] Zhang et al. SparseVLM. ICML 2025. https://proceedings.mlr.press/v267/zhang25s.html

We have polished our method section to present MiCo more clearly and concisely. We have also added a new task loss analysis in Appendix J to further investigate pruning process and MiCo's implications for future research, supplementing the additions summarized in our previous comment.

AuthorsOpenReview ↗

Follow up on Our Rebuttal and Revised Manuscript

Dear Reviewer,

As the discussion deadline approaches, we would be grateful if you could review our responses and revised manuscript when your schedule permits. We would welcome any remaining questions and would be happy to provide further clarification before deadline.

Thank you for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.