acceptodds
Under review as a conference paper at ICLR 2027

From Targeted Coverage to Conditional Information for Visual Token Pruning

Abstract

Vision-language models (VLMs) process long sequences of visual tokens, resulting in substantial inference cost. Existing visual token pruning methods typically rely on instruction relevance, visual redundancy, or their combination, but can retain tokens whose evidence relevant to the instruction is already represented by the selected set. We propose TACI (Target-Aware Coverage and Conditional Information), a training-free framework that selects visual evidence according to the additional information it provides about the instruction beyond the tokens already retained. Once the instruction representations have incorporated visual context, TACI evaluates each candidate using conditional mutual information. To avoid carrying the full visual sequence until this point, it first constructs a compact candidate pool using target-weighted visual coverage. Both stages use Gaussian process (GP) models over token representations, providing a unified probabilistic framework for the selection process. Experiments on seven benchmarks with LLaVA and Qwen backbones demonstrate the effectiveness of TACI across different token budgets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Model, Token Reduction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kQJLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TACI, a training-free, two-stage visual-token pruning method. Before decoder processing, it greedily constructs a candidate pool by minimizing target-weighted posterior uncertainty over visual tokens. At an intermediate decoder layer, it constructs a Gaussian-process surrogate over the surviving visual tokens and contextualized instruction tokens, then greedily retains visual tokens according to their conditional mutual information with the instruction given the visual tokens already selected.

Critical strengths
  1. The paper identifies a meaningful limitation of static relevance and diversity scores. A token can be individually relevant to a question while being redundant with evidence already retained. Evaluating marginal contribution conditioned on the selected set is a sensible way to address this problem.

  2. Early selection reduces the number of visual tokens processed by the decoder, while the intermediate selection uses instruction representations that have already attended to the surviving visual context. This is a reasonable compromise between selection quality and efficiency.

Critical weaknesses
  1. The paper avoids explicit combinatorial enumeration through greedy selection, but both stages have strict sequential dependencies. After every selected token, the posterior covariance changes and all remaining marginal gains must be recomputed.

For stage 1, in the complexity equation (Eq. 36), the first term can become large for high-resolution images, multi-image inputs, and videos. These are precisely the scenarios as motivation for token pruning.

For stage 2, appendix B states that the actual implementation instead refactorizes the instruction covariance at every selection step. The paper reports analytical FLOPs but no wall-clock latency, time-to-first-token, throughput, or scaling with (N_0). Low FLOPs do not guarantee low latency.

It is unclear whether the accuracy gains compensate for the selection overhead in practical deployment.

  1. The main empirical gains are mixed and often small.

For example, on LLaVA: At 192 tokens, TACI clearly wins mainly on MME; several baselines outperform it on most other benchmarks. At 128 tokens, its GQA advantage over PyramidDrop is only 0.02, while it trails the best method by 1–2 on POPE, MMBench, MMStar, and VizWiz. At 64 tokens, it wins GQA, MME, and SEED, but loses by 2.69 on POPE, 2.92 on MMBench, 1.93 on MMStar, and 1.36 on VizWiz. Differences of 0.01–0.2 should not be presented as meaningful improvements without uncertainty estimates.

TACI performs very poorly on VizWiz for Qwen: at 192 tokens, TACI 56.08 versus CDPruner 69.69; at 128 tokens, TACI 56.12 versus CDPruner 67.84; at 64 tokens, TACI 55.57 versus CDPruner 64.30. This is a very large and persistent gap.

The paper also reports no aggregate metric across benchmarks. The discussion counts per-column wins but does not quantify average normalized performance.

  1. The Gaussian-process conditional information is a surrogate whose connection to task information is weak. The visual and instruction representations are deterministic neural-network states. TACI constructs an auxiliary scalar GP over token positions using a fixed exponential kernel and then computes mutual information under that artificial Gaussian model. Thus, it is not the mutual information between a visual token and the correct answer, nor is it the empirical information that the token contributes to the VLM’s prediction. It is conditional dependence under a chosen kernel surrogate.

  2. The motivation of paper emphasizes high-resolution images, multi-image inputs, videos, etc. The experiments consist of seven single-image benchmarks on two 7B-scale backbones. There is no video, multi-image, long-context, or explicit high-resolution stress test. These omitted settings are precisely where the quadratic covariance construction and sequential greedy updates are most likely to become problematic.

Questions for authors

see the above "Critical Weaknesses" part.

10 comments
AuthorsOpenReview ↗

Response to Reviewer kQJL (Part 1)

Thank you for your thoughtful comments. We respond to each point below.

The paper avoids explicit combinatorial enumeration through greedy selection, but both stages have strict sequential dependencies. After every selected token, the posterior covariance changes and all remaining marginal gains must be recomputed. For stage 1, in the complexity equation (Eq. 36), the first term can become large for high-resolution images, multi-image inputs, and videos. These are precisely the scenarios as motivation for token pruning.

We agree that both selection stages are sequential: after each token is selected, the posterior covariance is updated and the marginal gains of the remaining candidates are recomputed. Our method avoids combinatorial subset enumeration, but it does not eliminate this sequential dependence. We therefore evaluated the resulting overhead directly through wall-clock latency and memory measurements, in addition to the analytical complexity reported in the paper. As shown in Table B-4 of our response to Reviewer VE8W, TACI requires 155.77 ms per inference (on POPE at an average budget of 192 visual tokens), while the uncompressed model requires 311.89 ms, corresponding to a 2.0× speedup. We further isolate the Stage 2 selection cost in Table B-5: the current implementation, which refactorizes the instruction covariance at each selection step as described in Appendix B, requires 8.27 ms in total, corresponding to about 5.3% of TACI’s end-to-end latency. Thus, while the sequential updates introduce measurable overhead, this cost remains small relative to the overall inference time, and the complete TACI system still achieves substantial end-to-end latency and memory savings.

For stage 2, appendix B states that the actual implementation instead refactorizes the instruction covariance at every selection step. The paper reports analytical FLOPs but no wall-clock latency, time-to-first-token, throughput, or scaling with (N_0). Low FLOPs do not guarantee low latency. It is unclear whether the accuracy gains compensate for the selection overhead in practical deployment.

Thank you for pointing this out. This is a mistake: Our final implementation uses the rank-one Cholesky update. The implementation description in Appendix B and the corresponding step in Algorithm 2 were not updated to reflect this change. We have thoroughly rechecked the paper for related inconsistencies and did not find any additional instances. We apologize for this error and will correct both locations in the revised manuscript.

Our main efficiency evaluation focuses on FLOPs as they are less dependent on implementation details and experimental environments. To complement this, Table B-4 in our response to Reviewer VE8W presents end-to-end latency. At an average budget of 192 visual tokens, TACI requires 155.77 ms per inference, while the uncompressed model requires 311.89 ms (corresponding to a 2.0× speedup), confirming that the reduction in analytical computation also translates into substantial wall-clock savings.

The main empirical gains are mixed and often small. For example, on LLaVA: At 192 tokens, TACI clearly wins mainly on MME; several baselines outperform it on most other benchmarks. At 128 tokens, its GQA advantage over PyramidDrop is only 0.02, while it trails the best method by 1–2 on POPE, MMBench, MMStar, and VizWiz. At 64 tokens, it wins GQA, MME, and SEED, but loses by 2.69 on POPE, 2.92 on MMBench, 1.93 on MMStar, and 1.36 on VizWiz. Differences of 0.01–0.2 should not be presented as meaningful improvements without uncertainty estimates.

We agree that very small numerical differences, should not be interpreted as meaningful improvements without uncertainty estimates. Following the evaluation settings used in prior work, our inference is deterministic on fixed benchmark sets, so repeated runs do not provide run-to-run variance. We will ensure that such near-ties are not presented as substantive improvements.

At the higher token budgets, several pruning methods already preserve performance close to the full-token model on many benchmarks, leaving limited headroom for further improvement. This naturally results in small numerical differences among methods. As the token budget becomes more restrictive, the differences become more pronounced, and TACI’s relative performance is strongest in this regime.

We also ask the reviewer to consider that visual token pruning inherently trades off computational efficiency and task performance. Figure 2 of the main paper shows that TACI achieves a favorable accuracy-computation trade-off.

AuthorsOpenReview ↗

Response to Reviewer kQJL (Part 2)

TACI performs very poorly on VizWiz for Qwen: at 192 tokens, TACI 56.08 versus CDPruner 69.69; at 128 tokens, TACI 56.12 versus CDPruner 67.84; at 64 tokens, TACI 55.57 versus CDPruner 64.30. This is a very large and persistent gap.

The substantial Qwen2.5-VL regression on VizWiz remains a limitation of the pruning schedule used in our main experiments. We investigate this case through an additional schedule ablation in Table C-1. Retaining visual tokens until the final decoder layer substantially improves VizWiz performance. However, we also observe that the same schedule reduces performance on other benchmarks (e.g.,POPE), indicating a cross-benchmark trade-off in the pruning schedule. We therefore retain a common pruning schedule across benchmarks in the main comparisons and present the alternative schedule as an analysis of the VizWiz failure case.

[Table C-1] VizWiz accuracy (%) under different average visual token budgets. TACI (Original) uses the pruning schedule adopted in the main paper, while TACI (Revised) retains visual tokens until the final decoder layer. Numbers in parentheses indicate the token budgets for Stages 1, 2, and 3, respectively.

Method64 tokens128 tokens192 tokens
LLaVA-1.5-7B
CDPruner57.0455.5054.50
TACI (Original)55.68 (102, 51, 0)54.44 (205, 102, 0)53.75 (307, 153, 0)
TACI (Revised)58.07 (85, 43, 43)54.94 (171, 85, 85)55.84 (231, 153, 153)
Qwen2.5-VL-7B
CDPruner64.3067.8469.69
TACI (Original)55.57 (102, 51, 0)56.12 (205, 102, 0)56.08 (307, 153, 0)
TACI (Revised)67.01 (85, 43, 43)69.12 (171, 85, 85)70.88 (231, 153, 153)
AuthorsOpenReview ↗

Response to Reviewer kQJL (Part 3)

The paper also reports no aggregate metric across benchmarks. The discussion counts per-column wins but does not quantify average normalized performance.

Thank you for the suggestion. We have added Avg. (%) to Tables 1 and 2 of the main paper (reproduced below as Tables C-2 and C-3) to present the average normalized performance across the seven benchmarks. Each benchmark score is normalized by its corresponding full-token baseline, and the normalized scores are then averaged across all seven benchmarks. The aggregate results consolidate our main findings: TACI is competitive at the higher token budgets and achieves the highest normalized average on both backbones at the most aggressive 64-token budget.

[Table C-2] Performance on LLaVA-1.5-7B under different token-reduction budgets. Within each budget, the best score in each column is shown in bold and the second-best score is underlined‾\underline{underlined}.

MethodGQAMMEPOPEMMBMMStarVizWizSEEDAvg. (%)
Full visual tokens (100.0%)
Original61.891814.2887.0164.5233.7353.6565.72100.00
Average 192 tokens (66.67%)
FastV57.921785.8181.3463.5732.2055.2463.1496.93
PyramidDrop61.881804.39‾\underline{1804.39}86.7364.69‾\underline{64.69}33.8053.7465.4999.92
HiMAP56.301726.2781.1261.0831.9353.8559.2794.18
SparseVLM60.261754.9886.9963.5733.40‾\underline{33.40}53.1764.5298.41
VisionZip59.061710.5086.1463.2333.0753.5962.7197.15
CDPruner60.521761.8786.6763.0632.7354.50‾\underline{54.50}64.0598.33
TransPrune60.651768.5986.6764.8632.6752.1964.8098.34
DUET-VLM59.831740.6887.1664.0032.8751.8364.7697.80
TACI (Ours)60.82‾\underline{60.82}1811.0787.03‾\underline{87.03}63.3232.7353.7565.02‾\underline{65.02}98.92‾\underline{98.92}
Average 128 tokens (77.78%)
FastV55.641701.4177.0863.2332.3355.42‾\underline{55.42}61.3594.68
PyramidDrop60.45‾\underline{60.45}1797.53‾\underline{1797.53}86.5063.92‾\underline{63.92}34.4053.4864.18‾\underline{64.18}99.22
HiMAP54.421642.3777.8659.4531.5353.8156.9091.49
SparseVLM58.681724.3786.2263.4933.00‾\underline{33.00}54.0863.3497.48
VisionZip57.181693.8185.0062.2932.6754.3361.3295.92
CDPruner59.871733.7287.9762.0332.4055.5063.2097.89
TransPrune59.951772.4986.87‾\underline{86.87}64.4332.5351.6363.8997.74
DUET-VLM58.681728.3686.7662.9732.2051.4963.7996.56
TACI (Ours)60.471834.8786.4362.2932.8054.4464.3198.76‾\underline{98.76}
Average 64 tokens (88.89%)
FastV51.711495.5569.8859.5431.4755.2455.9288.56
PyramidDrop57.431647.4581.6762.37‾\underline{62.37}32.87‾\underline{32.87}52.4458.6994.09
HiMAP50.091390.3768.8754.9030.6753.9551.7884.58
SparseVLM55.751659.7384.9461.8633.7353.4360.7795.30
VisionZip54.901636.3081.0859.6231.1355.1057.2992.38
CDPruner58.93‾\underline{58.93}1731.3787.6060.2230.7357.0461.5596.53
TransPrune58.441746.27‾\underline{1746.27}86.30‾\underline{86.30}63.4932.6751.8162.40‾\underline{62.40}96.66‾\underline{96.66}
DUET-VLM55.671706.0284.5862.2932.2051.2960.8894.49
TACI (Ours)59.181769.9484.9160.5731.8055.68‾\underline{55.68}63.0096.94
Reviewer_VE8WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TACI, a training-free visual token pruning framework for vision-language models. The method first identifies a target-aware set of candidate visual tokens and then uses conditional mutual information (CMI) to iteratively select tokens that contribute the most new instruction-relevant information beyond what has already been chosen. Unlike prior approaches that focus on relevance or diversity, TACI explicitly models redundancy with respect to the task, seeking tokens that provide complementary evidence rather than repeated information. Experiments on LLaVA-1.5 and Qwen2.5-VL across multiple benchmarks show competitive performance, especially under aggressive token budgets.

Critical strengths
  • The conditional mutual information (CMI) is a new criterion for visual token selection (as far as I'm concerned for this task), focusing on information gain rather than only relevance or diversity.

  • The information-theoretic framework based on Gaussian-process covariance modeling provides a coherent mathematical justification for token pruning.

  • The proposed method requires no retraining or fine-tuning, making it easy to integrate into existing VLMs.

  • The method achieves competitive results across multiple benchmarks, particularly under aggressive token budgets.

Critical weaknesses
  • The paper shows that TACI improves performance, but does not demonstrate that CMI is better than simpler relevance-and-diversity objectives, such as: MMR (Rel−λRedundancy\text{Rel}-\lambda\text{Redundancy}) or Coverage + Relevance (αRel+(1−α)Div\alpha \text{Rel}+(1-\alpha)\text{Div})

  • The contribution of the individual components (target-weighted coverage, CMI selection, GP modeling) is not completely isolated, making it difficult to identify the true source of the gains.

  • The method estimates information using covariance in the embedding space under a Gaussian-process assumption, but provides little evidence that this accurately reflects semantic information relevant to the task.

  • The paper provides limited empirical analysis of latency and memory usage of the CMI computation itself.

Questions for authors
  • What evidence shows that CMI identifies semantically informative tokens better than simpler relevance- and diversity-based metrics?

  • How much does each component (target-weighted coverage, GP covariance modeling, and CMI selection) contribute to the overall gain, and does CMI still help when compared against simpler relevance-diversity objectives under the same setup?

  • What empirical evidence supports the use of embedding covariance as a proxy for task-relevant semantic information rather than mere geometric similarity?

  • Given the overhead of covariance estimation and iterative CMI computation, what are the actual gains in latency and memory usage compared with simpler pruning methods?

3 comments
AuthorsOpenReview ↗

Response to Reviewer VE8W (Part 1)

Thank you for your thoughtful comments. We respond to each point below.

The paper shows that TACI improves performance, but does not demonstrate that CMI is better than simpler relevance-and-diversity objectives, such as MMR (Rel−λRedundancy\text{Rel}-\lambda \text{Redundancy}) or Coverage + Relevance (αRel+(1−α)Div\alpha \text{Rel}+(1-\alpha) \text{Div}).

What evidence shows that CMI identifies semantically informative tokens better than simpler relevance- and diversity-based metrics?

To assess whether CMI provides benefits beyond simpler relevance–diversity objectives, we replaced TACI's Stage 2 selection criterion with MMR and Coverage + Relevance while keeping the Stage 1 candidate pool and pruning schedule fixed. We evaluated these alternatives on Qwen2.5-VL-7B at an average budget of 64 visual tokens, using four values of λ\lambda or α\alpha (0.2, 0.3, 0.5, and 0.7) for each method. As shown in Table B-1, CMI achieves the highest performance on all seven benchmarks, with a normalized average score of 92.26%, compared with 87.13% for MMR and 89.07% for Coverage + Relevance under their best evaluated settings.

[Table B-1] Comparison with simpler Stage 2 selectors on Qwen2.5-VL-7B at an average budget of 64 visual tokens.

Stage 2 SelectorGQAPOPEMMEMMBenchMMStarVizWizSEEDAvg. (%)
MMR (λ=0.2\lambda = 0.2)54.5978.341936.6381.5353.2055.3871.5887.13
MMR (λ=0.3\lambda = 0.3)52.8175.881902.4781.5353.2755.0170.9185.92
MMR (λ=0.5\lambda = 0.5)50.8673.191813.9681.0153.3354.9070.4284.31
MMR (λ=0.7\lambda = 0.7)49.6972.401805.7580.7652.6054.9069.6583.51
Coverage + Relevance (α=0.2\alpha = 0.2)55.8379.972110.4581.7954.5355.3671.5189.07
Coverage + Relevance (α=0.3\alpha = 0.3)54.8878.461994.1881.5353.3355.4271.1887.53
Coverage + Relevance (α=0.5\alpha = 0.5)52.9075.621930.7181.7952.7355.2170.4085.95
Coverage + Relevance (α=0.7\alpha = 0.7)51.1873.321872.9681.7053.2754.9870.1284.82
TACI (Ours)58.9485.932167.5683.5156.9355.5773.5392.26

The contribution of the individual components (target-weighted coverage, CMI selection, GP modeling) is not completely isolated, making it difficult to identify the true source of the gains.

How much does each component (target-weighted coverage, GP covariance modeling, and CMI selection) contribute to the overall gain, and does CMI still help when compared against simpler relevance-diversity objectives under the same setup?

Thank you for raising this point. Appendix E (Table 3) of the original submission already provides controlled ablations of the Stage 1 relevance weighting, Stage 2 conditioning objective, and covariance construction. To strengthen this analysis, Table A-2 in our response to Reviewer uNtG extends these ablations to Qwen2.5 across all seven benchmarks at an average budget of 64 visual tokens, while changing one design component at a time. We further compare CMI against the simpler relevance–diversity objectives requested here in Table B-1.

AuthorsOpenReview ↗

Response to Reviewer VE8W (Part 2)

The method estimates information using covariance in the embedding space under a Gaussian-process assumption, but provides little evidence that this accurately reflects semantic information relevant to the task.

What empirical evidence supports the use of embedding covariance as a proxy for task-relevant semantic information rather than mere geometric similarity?

We agree that GP-CMI is a surrogate measure and do not claim that embedding covariance directly represents semantic information or mutual information with the correct answer. Establishing such a direct correspondence is inherently difficult. What can be evaluated empirically is whether the dependence captured by the surrogate is associated with information used by the VLM for prediction.

To examine this, Table A-3 in our response to Reviewer uNtG presents a token-deletion experiment on 300 questions per benchmark. Removing eight visual tokens prioritized by GP-CMI causes substantially larger accuracy drops than removing eight randomly selected tokens across all four benchmarks. This provides empirical support for the connection between the GP-CMI ranking and correct-answer prediction.

Table A-2 provides complementary downstream evidence. Target-weighted coverage followed by CMI outperforms Instruction-Relevance Top-K, which selects tokens solely based on their direct similarity to the instruction. In addition, Table B-1 shows that CMI outperforms simpler relevance–diversity objectives under the same Stage 1 candidate pool and pruning schedule. Although these comparisons do not establish that the GP covariance itself is a direct measure of semantic information, they provide evidence that the resulting GP-CMI criterion captures task-relevant structure beyond direct relevance or simple relevance-diversity heuristics.

The paper provides limited empirical analysis of latency and memory usage of the CMI computation itself. Given the overhead of covariance estimation and iterative CMI computation, what are the actual gains in latency and memory usage compared with simpler pruning methods?

Thank you for the question. We measured end-to-end latency, KV-cache memory, and peak GPU memory on POPE at an average budget of 192 visual tokens. As shown in Table B-4, TACI requires 155.77 ms per inference, while the uncompressed model requires 311.89 ms (2.0× speedup). It also has the lowest reported KV-cache usage among the methods with KV caching enabled and the lowest peak GPU memory among the compared methods.

To quantify the overhead of CMI itself, we separately measured the latency and memory of the Stage 2 selector (Table B-5). CMI requires 8.27 ms for selection, versus 5.02 ms for Static MI and 1.84 ms for Cosine FPS. Its peak selection memory is 39.47 MB, essentially the same as Static MI (39.37 MB), although its retained allocation is higher (3.90 MB versus 1.16 MB for Static MI and 0.76 MB for Cosine FPS).

Covariance construction (4.50 ms) and iterative recomputation of conditional information gains (3.32 ms) account for most of the CMI selection time. While CMI introduces measurable overhead relative to simpler selectors, its 8.27 ms Stage 2 selection cost corresponds to only about 5.3% of TACI’s total end-to-end latency, and the complete system still achieves substantial latency and memory savings.

[Table B-4] Total inference latency and memory on POPE with LLaVA-1.5-7B

MethodLatency (ms)KV cache (MB)Peak GPU (GB)
Full311.89 (1.00)331.8416.64
Average 192 tokens
FastV531.81 (0.59)130.3215.54
PyramidDrop387.68 (0.80)130.5116.28
SparseVLM475.71 (0.66)130.5120.37
VisionZip486.62 (0.64)130.5116.15
CDPruner705.56 (0.44)130.5117.52
TransPrune407.68 (0.77)130.5116.25
DUET-VLM556.58 (0.56)130.3815.65
TACI (Ours)155.77 (2.00)29.8514.52

[Table B-5] Isolated Stage 2 latency and memory

Stage 2 selectorMatrix construction (ms)Initialization (ms)Iteration & ranking (ms)Total selection (ms)Peak memory (MB)Retained allocation (MB)
Cosine FPS1.310.030.401.8420.180.76
Static MI4.520.360.035.0239.371.16
CMI (Ours)4.500.373.328.2739.473.90
AuthorsOpenReview ↗

Follow-up to Reviewer VE8W

Dear Reviewer VE8W,

Thank you again for your valuable feedback.

We hope that our responses and additional experiments, including the comparisons with simpler objectives and the latency and memory analysis, have adequately addressed your comments.

Please let us know if you have any additional questions or issues that we should clarify or elaborate on.

Sincerely yours,
The Authors

Reviewer_uNtGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TACI, a training-free visual token pruning method. It first constructs a candidate pool through target-weighted visual coverage, then selects tokens at an intermediate decoder layer using Gaussian conditional mutual information between visual tokens and instruction representations. The main mathematical derivations appear sound. However, the central motivation lacks direct validation, and the distinction from classical subset selection methods is not sufficiently supported experimentally. Performance varies substantially across tasks and budgets, without establishing a consistent accuracy or efficiency advantage.

Critical strengths
  1. The two selection objectives are explicitly defined, with derivations and algorithmic details provided in the appendix. The authors acknowledge that the surrogate objective guarantees neither answer correctness nor global optimality.

  2. Experiments cover two backbones and seven benchmarks, showing performance trade-offs across token budgets.

Critical weaknesses
  1. The central research gap is not sufficiently established. The paper argues that existing methods retain instruction-relevant tokens that provide little additional evidence beyond the selected set. However, CDPruner’s relevance-weighted DPP already updates marginal gains based on the selected set and suppresses redundancy. Accounting for previously selected tokens is therefore not itself a distinguishing contribution. The authors need to demonstrate when existing task-aware subset selection fails to identify redundant evidence and whether this failure materially affects performance.

  2. The method largely adopts classical criteria, while its incremental contribution is not isolated. Stage 1 corresponds to weighted predictive variance reduction, and Stage 2 to Gaussian mutual information maximization with a fixed target. The derivations primarily follow standard Gaussian conditioning and mutual information identities. The pruning schedule also follows DUET-VLM. The connection between GP information gain and task-relevant evidence remains unvalidated.

  3. Performance gains are inconsistent, with small improvements in some settings and substantial regressions in others.

Questions for authors

Please refer to weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer uNtG (Part 1)

Thank you for your thoughtful comments. We respond to each point below.

The central research gap is not sufficiently established. The paper argues that existing methods retain instruction-relevant tokens that provide little additional evidence beyond the selected set. However, CDPruner’s relevance-weighted DPP already updates marginal gains based on the selected set and suppresses redundancy. Accounting for previously selected tokens is therefore not itself a distinguishing contribution. The authors need to demonstrate when existing task-aware subset selection fails to identify redundant evidence and whether this failure materially affects performance.

Thank you for raising this. While CDPruner jointly considers instruction relevance and visual diversity, it does not explicitly evaluate whether a candidate's association with the instruction is already explained by the selected tokens. TACI evaluates the dependence that remains through conditional mutual information (under its Gaussian process surrogate).

CDPruner combines a fixed instruction-relevance weight with a marginal diversity gain that is recomputed as tokens are selected. It therefore favors candidates that are individually relevant to the instruction and add visual diversity. However, a candidate may remain relevant because it repeats instruction-related evidence already retained, while its visual differences concern details that do not help with the instruction. Such a candidate adds visual diversity without necessarily adding useful evidence for the task.

TACI assigns little gain when a candidate’s remaining dependence on the instruction is weak, even if the candidate is individually relevant and visually distinct. This can preserve the limited token budget for candidates that contribute complementary instruction-related evidence. We will clarify this motivation and distinction.

Tables 1 and 2 already show that TACI outperforms CDPruner on most benchmarks at each token budget on both backbones. To further assess the contribution of conditional information selection, we replaced the Stage 2 selector with a relevance-weighted DPP selector (from CDPruner) while keeping all others fixed. Table A-1 demonstrates that conditional information selection achieves higher average normalized performance at all token budgets, with larger gains under tighter budgets.

[Table A-1] Stage 2 selection ablation on Qwen2.5-VL-7B. Avg. (%) corresponds to the average across benchmarks.

Stage 2 SelectorGQAPOPEMMEMMBenchMMStarVizWizSEEDAvg. (%)
Average 192 tokens
Relevance-weighted DPP60.5987.082302.6283.4260.2055.8275.1894.71
TACI (Ours)60.3987.742297.5984.6260.6056.0875.8595.21
Average 128 tokens
Relevance-weighted DPP60.3286.302224.3682.9957.4055.8374.1693.17
TACI (Ours)60.4287.132256.5984.6259.4056.1275.3894.52
Average 64 tokens
Relevance-weighted DPP58.3184.792147.4581.7955.0055.5272.2790.85
TACI (Ours)58.9485.932167.5683.5156.9355.5773.5392.26
AuthorsOpenReview ↗

Response to Reviewer uNtG (Part 2)

The method largely adopts classical criteria, while its incremental contribution is not isolated. Stage 1 corresponds to weighted predictive variance reduction, and Stage 2 to Gaussian mutual information maximization with a fixed target. The derivations primarily follow standard Gaussian conditioning and mutual information identities. The pruning schedule also follows DUET-VLM.

We address this concern from two perspectives: 1) What TACI adds beyond established techniques and 2) how its individual design choices contribute to performance.

1) The underlying Gaussian conditioning and mutual information formulations are established, and the pruning schedule follows DUET-VLM, as stated in Section 3.1. Our contribution lies in integrating these techniques into a GP-based visual token pruning framework that combines early instruction-weighted coverage with conditional information selection using visually contextualized instruction targets.

This design addresses a practical challenge: Conditional selection uses instruction representations that have incorporated visual context by an intermediate decoder layer. To reduce the computation required to reach that layer, instruction-weighted coverage first forms a compact candidate pool before decoder processing. At the intermediate layer, TACI constructs a joint covariance from hidden and query-key representations and evaluates each candidate’s additional instruction-related information given the selected evidence. This integration enables the use of contextualized instruction targets while limiting the visual tokens carried to that stage.

2) Appendix E (Table 3) already evaluates instruction weighting, conditional selection, and covariance construction via component ablations. To strengthen this, Table A-2 extends the analysis to Qwen2.5-VL-7B:

a) Stage 1 selection: We keep Stage 2 fixed to CMI with the mixed covariance, and compare our target-weighted coverage against Uniform coverage and Instruction-relevance Top-K. Uniform coverage assigns equal weights to all visual locations in Eq. (11), removing instruction-dependent weighting. Instruction-relevance Top-K selects tokens solely by individual relevance without considering coverage. Table A-2(a) shows that target-weighted coverage achieves the highest average normalized performance.

b) Stage 2 conditioning: We keep Stage 1 fixed to target-weighted coverage. Static MI scores each candidate once by its mutual information with the instruction targets and keeps these scores fixed throughout selection. CMI recomputes each candidate’s additional information after conditioning on the selected tokens. Both methods use the same token representations and the same initial mixed covariance, isolating the effect of conditioning. Table A-2(b) shows that CMI achieves higher average normalized performance than Static MI.

c) Stage 2 covariance construction: We keep Stage 1 and the CMI selection rule fixed and compare hidden covariance, Q/K covariance, and the proposed mixed covariance. Table A-2(c) shows that the mixed covariance achieves the highest average normalized performance among these variants.

[Table A-2] Component ablation on Qwen2.5-VL-7B at an average of 64 visual tokens. Avg. (%) denotes the average performance across benchmarks after normalization by the corresponding full-token baseline.

Selection VariantGQAPOPEMMEMMBenchMMStarVizWizSEEDAvg. (%)
(a) Stage 1 selection
Uniform coverage59.7085.822164.3981.5356.3355.6173.2291.88
Instruction-relevance Top-K55.5782.791975.5681.6252.9355.0969.8887.82
Target-weighted coverage (Ours)58.9485.932167.5683.5156.9355.5773.5392.26
(b) Stage 2 selection objective
Static MI58.9885.422134.3882.3956.0055.6673.5091.60
CMI (Ours)58.9485.932167.5683.5156.9355.5773.5392.26
(c) Stage 2 covariance construction
CMI with hidden covariance58.8585.412171.9882.8255.4755.5673.2691.69
CMI with Q/K covariance58.9585.422190.6782.9055.5355.6373.0391.83
CMI with mixed covariance (Ours)58.9485.932167.5683.5156.9355.5773.5392.26
AuthorsOpenReview ↗

Response to Reviewer uNtG (Part 3)

The connection between GP information gain and task-relevant evidence remains unvalidated.

To examine whether tokens prioritized by GP-CMI contribute to correct-answer prediction, we performed a token-deletion experiment (300 questions per benchmark). For each question, we removed either eight tokens prioritized by GP-CMI or eight randomly selected visual tokens. We repeated random deletion 20 times per question and averaged the resulting accuracies.

Table A-3 presents accuracy after deletion. Removing the tokens prioritized by GP-CMI yields substantially lower accuracy than random deletion across all four benchmarks, providing empirical support for the connection between the GP-CMI ranking and correct-answer prediction.

[Table A-3] Prediction accuracy (%) after deleting eight visual tokens with high GP-CMI or random.

BenchmarkRandom deletionHigh GP-CMI deletionSignificant difference
GQA97.2589.00Yes
MMStar93.5788.00Yes
SEED-Bench96.9592.00Yes
POPE99.0094.67Yes

Performance gains are inconsistent, with small improvements in some settings and substantial regressions in others.

We agree that accuracy improvements vary across individual benchmarks and token budgets.

Such variation is not unique to TACI: It is also observed for every pruning method in Tables 1 and 2. Overall, TACI’s relative performance is strongest under aggressive token reduction. At the higher average budget of 192 tokens, TACI ranks among the top three pruning methods on six of seven benchmarks on Qwen2.5, remaining within 2.6% of the corresponding full-model score of the best method on each of these six benchmarks; VizWiz is the clear exception.

We would also note that visual token pruning aims to reduce computation while preserving task performance. Figure 2 of the main paper provides complementary evidence of a favorable accuracy computation trade-off on POPE.

The substantial Qwen2.5-VL regression on VizWiz remains a limitation of the pruning schedule used in our main experiments. We examine this case through an additional schedule ablation (in Table C-1 of our response to Reviewer kQJL). The alternative allocation substantially improves VizWiz performance. However, we also observed that the same schedule reduces performance on other benchmarks (e.g., POPE). We therefore retain a common pruning schedule across benchmarks in the main comparisons and present the alternative schedule as an analysis of the VizWiz failure case.

AuthorsOpenReview ↗

Follow-up to Reviewer uNtG

Dear Reviewer uNtG,

Thank you again for your valuable feedback.

We hope that our responses and additional experiments have adequately addressed your concerns regarding the distinction from existing approaches, the contributions of the individual components, and the connection between GP-CMI and task-relevant information.

Please let us know if you have any additional questions or issues that we should clarify or elaborate on.

Sincerely yours,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.