- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies training-free visual token pruning for multimodal large language models (MLLMs) from an information-theoretic perspective. The authors formulate token subset selection through four factors: information amount, representation reliability, task relevance, and coverage. Based on this formulation, they propose MiCo, a two-stage pruning framework. In Stage 1, MiCo performs task-agnostic selection on vision-encoder features, using feature norm and CLS/pooled visual attention to weight visual tokens while encouraging representative coverage. In Stage 2, after several LLM layers, MiCo performs query-aware refinement using contextualized visual features and text-to-visual attention, again selecting a representative subset under a fixed token budget. The resulting objective is instantiated as a weighted coverage function and optimized greedily. Experiments across multiple MLLMs, image/video benchmarks, pruning ratios, and efficiency settings show consistently strong performance and substantial inference acceleration.
Critical strengths
1. The two-stage pruning design is well motivated and natural.
It is reasonable to first remove visually redundant tokens before language–vision interaction, while retaining a relatively large candidate pool, and then perform more aggressive query-aware pruning after the model has access to the textual context. This design combines visual representativeness and task relevance in a clean way.
2. The experimental evaluation is extensive.
The paper compares against a broad set of pruning baselines across multiple MLLM architectures and many benchmarks, including general visual understanding, fine-grained, hallucination-related evaluation, and video understanding. Importantly, the paper also reports practical efficiency metrics such as latency, memory consumption, and FLOPs rather than only comparing accuracy under fixed pruning ratios.
Critical weaknesses
1. The presentation is unnecessarily theory-heavy and obscures a relatively simple algorithmic idea.
The information-theoretic formulation is mathematically reasonable under the stated assumptions and does motivate the factorized weighted-coverage structure. However, the latent quantities are not directly observable, and the executable method ultimately depends on empirically chosen proxies. The connection from task log-loss to the concrete pruning rule therefore involves several layers of abstraction, including the semantic-query model, the semantic-erasure assumption, and the observable proxy assignments. Given that a substantial portion of the method section is devoted to this derivation while the practical algorithm is comparatively simple, I find the presentation harder to follow than necessary and the theoretical framing somewhat stronger than the evidence supporting the specific proxy instantiation.
2. The theoretical proxies are not sufficiently validated, especially in Stage 1.
MiCo relies on feature norm, visual attention, and text-to-visual attention as proxies for information amount, representation reliability, and task relevance. However, the reported statistics do not provide strong support for these assumptions. In Stage 1, the per-token Spearman correlation between feature norm and the proposed information reference is only , indicating essentially no useful token-level ranking relationship, while the confidence intervals for the CLS-attention reliability test include the chance level. This is particularly concerning given prior observations that high-norm ViT tokens can correspond to representation artifacts rather than informative regions [1]. Since Stage-1 pruning is irreversible and occurs before query-aware interaction, errors at this stage cannot be recovered later. Stage 2 shows a statistically significant but still very weak association between text-to-visual attention and task relevance ().
3. Scalability to larger visual inputs and larger MLLMs is insufficiently evaluated.
The experiments cover diverse architectures, but do not systematically evaluate scaling with input resolution or model size. This is particularly important for MiCo because its coverage selector requires pairwise visual-token similarities, whose overhead grows quadratically with the number of visual tokens. A controlled evaluation with increasing image resolution/token count would clarify whether the reported speedups remain meaningful for high-resolution inputs. Similarly, the largest evaluated MLLM is 13B, leaving it unclear whether the method and its proxy choices generalize to substantially larger VLMs.
4. The video comparison could be more complete.
MiCo shows particularly strong gains on video benchmarks, but the reported baselines are mostly general visual-token pruning methods. There are video-specific approaches that explicitly exploit temporal or inter-frame redundancy, e.g., through frame similarity, key-frame selection, or cross-frame token merging/pruning. This comparison is especially relevant because MiCo performs Stage-1 pruning independently for each frame, while cross-frame redundancy is only handled later in Stage 2.
[1]. Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2024, May). Vision transformers need registers. In International conference on learning representations (Vol. 2024, pp. 2632-2652).
Questions for authors
-
How should the pruning budget be chosen in practice? Different tasks may require very different amounts of visual information; for example, OCR/grounding and reasoning-heavy tasks may have different sensitivity to pruning. Can the authors provide a task-adaptive strategy or clearer guidance for selecting the token budget?
-
See weakness 3
-
See weakness 4
8 comments
Complete MiCo-Adaptive Results at tau=0.990–0.999 (Q1)
We introduce MiCo-adaptive, a new adaptive-budget design that uses a shared coverage-retention threshold tau instead of a fixed token count. At each stage, selection stops when retained/full weighted coverage reaches tau. Because visual features and query-dependent attention vary across inputs, the retained count adapts to image complexity and the visual information demanded by the question. Tau provides a simple control over the accuracy–computation trade-off.
PDF: Figure 8; Appendix I.1, Figure 23. The tables below report all ten thresholds from 0.990 to 0.999 on LLaVA-1.5-7B across ten benchmarks.
1. Mean tokens: Stage 1 / Stage 2 / layer average
Vanilla: 576 tokens. Each cell averages the benchmark’s samples.
| Benchmark | 0.990 | 0.991 | 0.992 | 0.993 | 0.994 | 0.995 | 0.996 | 0.997 | 0.998 | 0.999 |
|---|---|---|---|---|---|---|---|---|---|---|
| GQA | 158.2/50.8/74.3 | 168.0/57.3/81.6 | 178.9/65.4/90.2 | 191.0/74.5/100.0 | 204.7/85.8/111.8 | 220.3/100.5/126.7 | 238.6/119.7/145.7 | 260.6/145.0/170.3 | 288.2/180.7/204.2 | 326.8/239.5/258.6 |
| SQA-IMG | 78.4/27.0/38.3 | 84.9/30.4/42.3 | 92.4/34.6/47.2 | 101.2/39.8/53.2 | 111.7/46.4/60.7 | 124.2/55.4/70.4 | 139.8/67.3/83.2 | 160.0/84.3/100.8 | 187.6/110.9/127.7 | 231.1/161.3/176.5 |
| TextVQA | 144.5/55.8/75.2 | 153.1/61.8/81.7 | 162.7/68.8/89.3 | 173.5/77.2/98.2 | 185.9/87.6/109.1 | 200.2/100.7/122.4 | 217.1/117.5/139.3 | 237.9/140.0/161.4 | 264.7/172.2/192.4 | 303.5/225.6/242.6 |
| POPE | 154.2/46.2/69.8 | 163.9/52.1/76.5 | 174.6/59.4/84.6 | 186.5/68.4/94.3 | 200.1/79.8/106.2 | 215.8/94.4/121.0 | 234.0/113.5/139.8 | 256.0/138.8/164.4 | 283.8/175.1/198.9 | 322.7/233.9/253.3 |
| MME | 139.3/41.6/63.0 | 148.3/46.8/69.0 | 158.4/53.2/76.2 | 169.6/61.1/84.8 | 182.4/71.2/95.5 | 197.2/84.2/108.9 | 214.5/101.5/126.2 | 235.5/125.0/149.2 | 262.2/159.4/181.9 | 300.5/216.0/234.5 |
| MMB-EN | 107.1/31.7/48.2 | 115.0/35.8/53.2 | 124.1/40.9/59.1 | 134.4/47.7/66.6 | 146.3/56.0/75.7 | 160.3/66.8/87.2 | 177.2/81.4/102.4 | 198.1/102.0/123.1 | 225.6/132.5/152.9 | 266.4/185.4/203.1 |
| MMB-CN | 107.1/31.5/48.0 | 115.1/35.7/53.0 | 124.2/40.8/59.1 | 134.5/47.6/66.6 | 146.4/55.9/75.7 | 160.4/66.7/87.2 | 177.3/81.5/102.5 | 198.2/102.2/123.2 | 225.7/132.7/153.0 | 266.5/185.7/203.4 |
| SEED-IMG | 139.4/45.8/66.3 | 148.5/51.4/72.6 | 158.6/58.3/80.3 | 170.1/66.6/89.2 | 183.1/77.2/100.3 | 198.2/90.5/114.1 | 216.0/108.0/131.6 | 237.6/131.6/154.8 | 265.4/165.8/187.6 | 305.2/222.2/240.4 |
| AI2D | 118.9/49.3/64.6 | 126.1/54.7/70.3 | 134.2/60.9/77.0 | 143.2/68.5/84.9 | 153.7/78.0/94.5 | 165.8/89.7/106.4 | 180.1/104.6/121.1 | 197.9/125.0/140.9 | 221.1/153.5/168.3 | 255.7/199.5/211.8 |
| MMMU | 90.9/35.3/47.4 | 97.3/39.5/52.1 | 104.6/44.4/57.6 | 113.1/50.5/64.2 | 122.9/58.3/72.4 | 134.7/68.1/82.6 | 149.1/81.2/96.0 | 167.4/99.0/114.0 | 192.3/125.8/140.4 | 231.1/171.5/184.6 |
2. Relative performance (% of each benchmark’s Vanilla)
| Benchmark | 0.990 | 0.991 | 0.992 | 0.993 | 0.994 | 0.995 | 0.996 | 0.997 | 0.998 | 0.999 |
|---|---|---|---|---|---|---|---|---|---|---|
| GQA | 93.68 | 94.57 | 94.95 | 95.63 | 95.96 | 96.60 | 96.83 | 97.04 | 98.08 | 98.55 |
| SQA-IMG | 99.00 | 99.57 | 99.21 | 98.86 | 99.57 | 99.43 | 98.50 | 98.86 | 98.64 | 98.43 |
| TextVQA | 97.79 | 98.00 | 98.04 | 98.06 | 98.82 | 99.03 | 99.31 | 99.36 | 98.81 | 99.59 |
| POPE | 97.96 | 99.12 | 99.29 | 100.00 | 100.25 | 100.25 | 99.80 | 100.28 | 100.45 | 100.06 |
| MME | 91.42 | 92.75 | 93.87 | 94.61 | 96.02 | 96.32 | 96.79 | 96.42 | 97.02 | 97.51 |
| MMB-EN | 93.40 | 92.61 | 91.82 | 93.80 | 95.25 | 95.65 | 95.65 | 96.04 | 97.36 | 99.21 |
| MMB-CN | 92.92 | 93.36 | 93.51 | 93.95 | 95.72 | 97.05 | 97.20 | 98.23 | 100.00 | 98.53 |
| SEED-IMG | 92.21 | 92.55 | 93.10 | 93.75 | 94.65 | 95.38 | 96.10 | 96.56 | 97.81 | 98.51 |
| AI2D | 100.00 | 100.23 | 99.94 | 99.53 | 99.65 | 99.24 | 99.47 | 98.94 | 99.18 | 99.30 |
| MMMU | 97.26 | 100.00 | 98.48 | 99.70 | 100.00 | 99.70 | 98.78 | 99.09 | 97.87 | 97.56 |
Across ten benchmarks (equal weights), lowering tau from 0.999 to 0.990 reduces the mean layer-average count from 220.88 to 59.51 tokens, a 73.06% reduction, while relative performance remains 95.56%, compared with 98.72% at 0.999. These results support MiCo’s ability to retain a core set of task-useful visual tokens: adjusting tau substantially reduces computation while preserving most performance. MiCo-adaptive therefore combines a controllable retention threshold with input-dependent token budgets, allocating tokens according to image complexity and question-specific information needs. Figure 23 reports the wider sweep of all 19 thresholds (0.900–0.999).
Complete Proxy-Validation and Substitution Results (W2)
1. Norm–entropy agreement
Appendix C.2, Figure 9. The original token-level correlation is 0.00699; the position-mean correlation is 0.35943. These measure different associations. Pruning retains a subset, so we additionally test Top-k agreement: different full rankings can retain identical sets. Norms are ranked within each image using independent COCO-Stuff A/B pools. Overlap = intersection/k; random uses the same candidate set.
| Measure | k=32 | k=64 | k=128 |
|---|---|---|---|
| A→B overlap | 30.28% | 46.98% | 38.98% |
| B→A overlap | 35.26% | 49.51% | 39.17% |
| Random expectation, approx. | 6.04% | 12.07% | 24.14% |
| A→B / random | 5.02× | 3.89× | 1.61× |
| B→A / random | 5.84× | 4.10× | 1.62× |
All overlaps exceed 30%; five exceed 35%. Enrichment reaches 5.02×/5.84×, supporting stable norm–entropy set agreement.
2. CLS attention and semantic recovery
Appendix C.2, Figure 10, Table 10. Concordance compares correct/incorrect decoding under matched conditions and is averaged within and across eligible images.
| Metric | Linear | MLP |
|---|---|---|
| CLS-attention concordance | 0.5741 | 0.5972 |
| 95% CI | [0.5142, 0.6318] | [0.5413, 0.6507] |
| Eligible images / 200 | 116 / 200 | 133 / 200 |
| Raw p | 0.01510 | 0.00090 |
| Holm-adjusted p | 0.04530 | 0.00360 |
Both new confidence intervals lie above 0.5, addressing the concern about intervals including chance and supporting positive association with successful semantic recovery.
3. Text attention and target relevance
Figure 3, Table 1; Appendix C.2, Figures 11–13. Target mass is normalized token–target overlap; correlations use image-macro bin means.
| Retained bins | N | Positions included | Spearman ρ | Pearson r |
|---|---|---|---|---|
| 1--10 | 256 | 100.00% | 0.5636 | -0.2606 |
| 1--9 | 231 | 90.23% | 1.0000 | 0.7403 |
| 1--8 | 206 | 80.47% | 1.0000 | 0.8335 |
| 1--7 | 181 | 70.70% | 1.0000 | 0.8912 |
| 1--6 | 156 | 60.94% | 1.0000 | 0.9414 |
Target mass rises through bins 1–9 and falls only in bin 10. Bins covering 90.23% of positions show strong positive alignment. Figure 13 shows bottom-row maxima outside the target across all 32 layers in the example, echoing attention shift.
4. PCA component ablations
Figure 4; Appendix I.2, Figure 24. Selected-to-full convex-hull area ratios in the illustrated example. S2 is an independent Stage-2-only diagnostic with Stage 1 disabled.
| Setting | Stage 1 | Independent Stage 2 |
|---|---|---|
| MiCo | 0.932 | 0.957 |
| Without coverage | 0.490 | 0.905 |
| Without corresponding attention | 0.910 | 0.940 |
| Without norm | 0.930 | 0.953 |
Removing each component reduces projected coverage in this example.
5. Proxy substitutions
Table 8; Appendix E.2. Qwen2.5-VL-7B, T=128. Acc₈ includes MMB-CN and uses MME/20.
| Replacement / method | AI2D | POPE | HallB | MME | MMB-EN | MMB-CN | MMStar | SQA-IMG | Acc₈ | Rel |
|---|---|---|---|---|---|---|---|---|---|---|
| MMTok (ICLR'26) | 76.3 | 84.2 | 43.3 | 2120.7 | 79.8 | 77.5 | 53.7 | 81.4 | 75.3 | 90.8% |
| Coverage → visual-mean similarity (both stages) | 70.14 | 78.10 | 35.80 | 1688.69 | 75.69 | 73.71 | 47.20 | 81.31 | 68.30 | 82.34% |
| Norm → hidden-state entropy (both stages) | 78.47 | 84.49 | 45.43 | 2152.60 | 80.84 | 78.26 | 52.60 | 83.04 | 76.35 | 92.04% |
| Text attention → text–visual similarity (S2) | 79.08 | 85.54 | 45.88 | 2142.01 | 79.73 | 78.95 | 56.47 | 84.28 | 77.13 | 92.98% |
| CLS attention → global-token similarity (S1) | 79.83 | 84.59 | 46.13 | 2207.75 | 81.10 | 78.95 | 56.73 | 83.34 | 77.63 | 93.59% |
| MiCo | 80.44 | 85.46 | 48.60 | 2252.37 | 80.76 | 79.12 | 56.53 | 84.58 | 78.51 | 94.65% |
The three alternative weights exceed MMTok, showing that gains do not depend on one specific signal. MiCo’s combination performs best; the coverage replacement highlights the value of set coverage.
Theoretical Explanation, Proxy Validation, Scalability, Video Comparisons, and Adaptive Budgets
We appreciate your careful review and questions on theory, proxies, scalability, and budget selection. We hope our expanded analyses and experiments address your concerns and deepen your understanding of MiCo's motivation and effectiveness.
1. Theory and practical algorithm (W1). Most current token-pruning methods use heuristic rules, motivating us to derive a general pruning objective from theory. Starting from task log-loss, we introduce a semantic-erasure model and necessary approximations to obtain a source-weighted mutual-information coverage objective (Section 3; Appendix B). For efficient single-pass estimation, we follow PACT, VisionZip/VisPruner, and FastV/SparseVLM, using norm/CLS attention/text-to-visual attention for information/reliability/relevance, respectively. We acknowledge that the derivation is involved and will streamline its presentation. In practice, these proxies draw on established pruning methods and are readily available during inference, keeping MiCo training-free, simple to implement, and easy to use.
2. Proxy validation and early pruning (W2).
3. Resolution and model scalability (W3, Q2; Appendices D.5-D.6). At fixed T=320 on NeXT-7B:
Stage 1 covers fixed-size crops; at fixed T, Stage 2 receives about 640 candidates, limiting selector growth. Fixed 88.9% compression additionally yields 2.23x/2.57x/2.92x speedups.
With unchanged proxy forms at approximately 89% compression, larger models give:
MiCo leads all three comparisons by 105-311 MME points, retaining 98.9% on NeXT-34B.
4. Video-specific baselines (W4, Q3; Appendix H.5). We add FastVID, HoliTom, and FlashVID, which explicitly exploit temporal redundancy. Three-task means on VideoMME/MVBench/MLVU are:
At T=32, MiCo retains 97.3% of Vanilla performance and leads VideoMME/MLVU. At T=16, it retains 93.1%, with the fastest measured complete-answer latency (835.0 ms, 2.26x) and 21.32 GiB peak memory. Stage 1 preserves per-frame candidates; Stage 2 selects over the concatenated, contextualized pool, allowing cross-frame redundancy to be covered after text interaction.
5. Adaptive token budget (Q1). We introduce MiCo-adaptive, which adapts both stages' token counts to each image-question pair using a shared information-retention threshold tau. Figure 8 evaluates tau=0.990-0.999; Appendix I.1, Figure 23 reports all 19 thresholds from 0.900 to 0.999 across ten benchmarks. As tau decreases, retained-token counts fall substantially, while relative scores generally decline much less, showing that MiCo preserves task-useful visual information under substantial compression. Thus tau provides a flexible accuracy–computation trade-off, allowing both stages to adapt their budgets to visual complexity and query relevance.
If our responses can address your questions and concerns, we would be very grateful if you would consider raising your score.