- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TACI, a training-free, two-stage visual-token pruning method. Before decoder processing, it greedily constructs a candidate pool by minimizing target-weighted posterior uncertainty over visual tokens. At an intermediate decoder layer, it constructs a Gaussian-process surrogate over the surviving visual tokens and contextualized instruction tokens, then greedily retains visual tokens according to their conditional mutual information with the instruction given the visual tokens already selected.
Critical strengths
-
The paper identifies a meaningful limitation of static relevance and diversity scores. A token can be individually relevant to a question while being redundant with evidence already retained. Evaluating marginal contribution conditioned on the selected set is a sensible way to address this problem.
-
Early selection reduces the number of visual tokens processed by the decoder, while the intermediate selection uses instruction representations that have already attended to the surviving visual context. This is a reasonable compromise between selection quality and efficiency.
Critical weaknesses
- The paper avoids explicit combinatorial enumeration through greedy selection, but both stages have strict sequential dependencies. After every selected token, the posterior covariance changes and all remaining marginal gains must be recomputed.
For stage 1, in the complexity equation (Eq. 36), the first term can become large for high-resolution images, multi-image inputs, and videos. These are precisely the scenarios as motivation for token pruning.
For stage 2, appendix B states that the actual implementation instead refactorizes the instruction covariance at every selection step. The paper reports analytical FLOPs but no wall-clock latency, time-to-first-token, throughput, or scaling with (N_0). Low FLOPs do not guarantee low latency.
It is unclear whether the accuracy gains compensate for the selection overhead in practical deployment.
- The main empirical gains are mixed and often small.
For example, on LLaVA: At 192 tokens, TACI clearly wins mainly on MME; several baselines outperform it on most other benchmarks. At 128 tokens, its GQA advantage over PyramidDrop is only 0.02, while it trails the best method by 1–2 on POPE, MMBench, MMStar, and VizWiz. At 64 tokens, it wins GQA, MME, and SEED, but loses by 2.69 on POPE, 2.92 on MMBench, 1.93 on MMStar, and 1.36 on VizWiz. Differences of 0.01–0.2 should not be presented as meaningful improvements without uncertainty estimates.
TACI performs very poorly on VizWiz for Qwen: at 192 tokens, TACI 56.08 versus CDPruner 69.69; at 128 tokens, TACI 56.12 versus CDPruner 67.84; at 64 tokens, TACI 55.57 versus CDPruner 64.30. This is a very large and persistent gap.
The paper also reports no aggregate metric across benchmarks. The discussion counts per-column wins but does not quantify average normalized performance.
-
The Gaussian-process conditional information is a surrogate whose connection to task information is weak. The visual and instruction representations are deterministic neural-network states. TACI constructs an auxiliary scalar GP over token positions using a fixed exponential kernel and then computes mutual information under that artificial Gaussian model. Thus, it is not the mutual information between a visual token and the correct answer, nor is it the empirical information that the token contributes to the VLM’s prediction. It is conditional dependence under a chosen kernel surrogate.
-
The motivation of paper emphasizes high-resolution images, multi-image inputs, videos, etc. The experiments consist of seven single-image benchmarks on two 7B-scale backbones. There is no video, multi-image, long-context, or explicit high-resolution stress test. These omitted settings are precisely where the quadratic covariance construction and sequential greedy updates are most likely to become problematic.
Questions for authors
see the above "Critical Weaknesses" part.
10 comments
Response to Reviewer kQJL (Part 2)
TACI performs very poorly on VizWiz for Qwen: at 192 tokens, TACI 56.08 versus CDPruner 69.69; at 128 tokens, TACI 56.12 versus CDPruner 67.84; at 64 tokens, TACI 55.57 versus CDPruner 64.30. This is a very large and persistent gap.
The substantial Qwen2.5-VL regression on VizWiz remains a limitation of the pruning schedule used in our main experiments. We investigate this case through an additional schedule ablation in Table C-1. Retaining visual tokens until the final decoder layer substantially improves VizWiz performance. However, we also observe that the same schedule reduces performance on other benchmarks (e.g.,POPE), indicating a cross-benchmark trade-off in the pruning schedule. We therefore retain a common pruning schedule across benchmarks in the main comparisons and present the alternative schedule as an analysis of the VizWiz failure case.
[Table C-1] VizWiz accuracy (%) under different average visual token budgets. TACI (Original) uses the pruning schedule adopted in the main paper, while TACI (Revised) retains visual tokens until the final decoder layer. Numbers in parentheses indicate the token budgets for Stages 1, 2, and 3, respectively.
| Method | 64 tokens | 128 tokens | 192 tokens |
|---|---|---|---|
| LLaVA-1.5-7B | |||
| CDPruner | 57.04 | 55.50 | 54.50 |
| TACI (Original) | 55.68 (102, 51, 0) | 54.44 (205, 102, 0) | 53.75 (307, 153, 0) |
| TACI (Revised) | 58.07 (85, 43, 43) | 54.94 (171, 85, 85) | 55.84 (231, 153, 153) |
| Qwen2.5-VL-7B | |||
| CDPruner | 64.30 | 67.84 | 69.69 |
| TACI (Original) | 55.57 (102, 51, 0) | 56.12 (205, 102, 0) | 56.08 (307, 153, 0) |
| TACI (Revised) | 67.01 (85, 43, 43) | 69.12 (171, 85, 85) | 70.88 (231, 153, 153) |
Response to Reviewer kQJL (Part 3)
The paper also reports no aggregate metric across benchmarks. The discussion counts per-column wins but does not quantify average normalized performance.
Thank you for the suggestion. We have added Avg. (%) to Tables 1 and 2 of the main paper (reproduced below as Tables C-2 and C-3) to present the average normalized performance across the seven benchmarks. Each benchmark score is normalized by its corresponding full-token baseline, and the normalized scores are then averaged across all seven benchmarks. The aggregate results consolidate our main findings: TACI is competitive at the higher token budgets and achieves the highest normalized average on both backbones at the most aggressive 64-token budget.
[Table C-2] Performance on LLaVA-1.5-7B under different token-reduction budgets. Within each budget, the best score in each column is shown in bold and the second-best score is .
| Method | GQA | MME | POPE | MMB | MMStar | VizWiz | SEED | Avg. (%) |
|---|---|---|---|---|---|---|---|---|
| Full visual tokens (100.0%) | ||||||||
| Original | 61.89 | 1814.28 | 87.01 | 64.52 | 33.73 | 53.65 | 65.72 | 100.00 |
| Average 192 tokens (66.67%) | ||||||||
| FastV | 57.92 | 1785.81 | 81.34 | 63.57 | 32.20 | 55.24 | 63.14 | 96.93 |
| PyramidDrop | 61.88 | 86.73 | 33.80 | 53.74 | 65.49 | 99.92 | ||
| HiMAP | 56.30 | 1726.27 | 81.12 | 61.08 | 31.93 | 53.85 | 59.27 | 94.18 |
| SparseVLM | 60.26 | 1754.98 | 86.99 | 63.57 | 53.17 | 64.52 | 98.41 | |
| VisionZip | 59.06 | 1710.50 | 86.14 | 63.23 | 33.07 | 53.59 | 62.71 | 97.15 |
| CDPruner | 60.52 | 1761.87 | 86.67 | 63.06 | 32.73 | 64.05 | 98.33 | |
| TransPrune | 60.65 | 1768.59 | 86.67 | 64.86 | 32.67 | 52.19 | 64.80 | 98.34 |
| DUET-VLM | 59.83 | 1740.68 | 87.16 | 64.00 | 32.87 | 51.83 | 64.76 | 97.80 |
| TACI (Ours) | 1811.07 | 63.32 | 32.73 | 53.75 | ||||
| Average 128 tokens (77.78%) | ||||||||
| FastV | 55.64 | 1701.41 | 77.08 | 63.23 | 32.33 | 61.35 | 94.68 | |
| PyramidDrop | 86.50 | 34.40 | 53.48 | 99.22 | ||||
| HiMAP | 54.42 | 1642.37 | 77.86 | 59.45 | 31.53 | 53.81 | 56.90 | 91.49 |
| SparseVLM | 58.68 | 1724.37 | 86.22 | 63.49 | 54.08 | 63.34 | 97.48 | |
| VisionZip | 57.18 | 1693.81 | 85.00 | 62.29 | 32.67 | 54.33 | 61.32 | 95.92 |
| CDPruner | 59.87 | 1733.72 | 87.97 | 62.03 | 32.40 | 55.50 | 63.20 | 97.89 |
| TransPrune | 59.95 | 1772.49 | 64.43 | 32.53 | 51.63 | 63.89 | 97.74 | |
| DUET-VLM | 58.68 | 1728.36 | 86.76 | 62.97 | 32.20 | 51.49 | 63.79 | 96.56 |
| TACI (Ours) | 60.47 | 1834.87 | 86.43 | 62.29 | 32.80 | 54.44 | 64.31 | |
| Average 64 tokens (88.89%) | ||||||||
| FastV | 51.71 | 1495.55 | 69.88 | 59.54 | 31.47 | 55.24 | 55.92 | 88.56 |
| PyramidDrop | 57.43 | 1647.45 | 81.67 | 52.44 | 58.69 | 94.09 | ||
| HiMAP | 50.09 | 1390.37 | 68.87 | 54.90 | 30.67 | 53.95 | 51.78 | 84.58 |
| SparseVLM | 55.75 | 1659.73 | 84.94 | 61.86 | 33.73 | 53.43 | 60.77 | 95.30 |
| VisionZip | 54.90 | 1636.30 | 81.08 | 59.62 | 31.13 | 55.10 | 57.29 | 92.38 |
| CDPruner | 1731.37 | 87.60 | 60.22 | 30.73 | 57.04 | 61.55 | 96.53 | |
| TransPrune | 58.44 | 63.49 | 32.67 | 51.81 | ||||
| DUET-VLM | 55.67 | 1706.02 | 84.58 | 62.29 | 32.20 | 51.29 | 60.88 | 94.49 |
| TACI (Ours) | 59.18 | 1769.94 | 84.91 | 60.57 | 31.80 | 63.00 | 96.94 |
Response to Reviewer kQJL (Part 1)
Thank you for your thoughtful comments. We respond to each point below.
We agree that both selection stages are sequential: after each token is selected, the posterior covariance is updated and the marginal gains of the remaining candidates are recomputed. Our method avoids combinatorial subset enumeration, but it does not eliminate this sequential dependence. We therefore evaluated the resulting overhead directly through wall-clock latency and memory measurements, in addition to the analytical complexity reported in the paper. As shown in Table B-4 of our response to Reviewer VE8W, TACI requires 155.77 ms per inference (on POPE at an average budget of 192 visual tokens), while the uncompressed model requires 311.89 ms, corresponding to a 2.0× speedup. We further isolate the Stage 2 selection cost in Table B-5: the current implementation, which refactorizes the instruction covariance at each selection step as described in Appendix B, requires 8.27 ms in total, corresponding to about 5.3% of TACI’s end-to-end latency. Thus, while the sequential updates introduce measurable overhead, this cost remains small relative to the overall inference time, and the complete TACI system still achieves substantial end-to-end latency and memory savings.
Thank you for pointing this out. This is a mistake: Our final implementation uses the rank-one Cholesky update. The implementation description in Appendix B and the corresponding step in Algorithm 2 were not updated to reflect this change. We have thoroughly rechecked the paper for related inconsistencies and did not find any additional instances. We apologize for this error and will correct both locations in the revised manuscript.
Our main efficiency evaluation focuses on FLOPs as they are less dependent on implementation details and experimental environments. To complement this, Table B-4 in our response to Reviewer VE8W presents end-to-end latency. At an average budget of 192 visual tokens, TACI requires 155.77 ms per inference, while the uncompressed model requires 311.89 ms (corresponding to a 2.0× speedup), confirming that the reduction in analytical computation also translates into substantial wall-clock savings.
We agree that very small numerical differences, should not be interpreted as meaningful improvements without uncertainty estimates. Following the evaluation settings used in prior work, our inference is deterministic on fixed benchmark sets, so repeated runs do not provide run-to-run variance. We will ensure that such near-ties are not presented as substantive improvements.
At the higher token budgets, several pruning methods already preserve performance close to the full-token model on many benchmarks, leaving limited headroom for further improvement. This naturally results in small numerical differences among methods. As the token budget becomes more restrictive, the differences become more pronounced, and TACI’s relative performance is strongest in this regime.
We also ask the reviewer to consider that visual token pruning inherently trades off computational efficiency and task performance. Figure 2 of the main paper shows that TACI achieves a favorable accuracy-computation trade-off.