acceptodds
Withdrawn from ICLR 2027

Pruning by Consequence: Intervention-Guided Visual Token Selection for Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training-free visual token selection reduces the inference cost of vision-language models by retaining a small subset of image tokens. Existing selectors rank tokens with attention, similarity, saliency, or diversity signals—inexpensive proxies for the quantity that actually matters: whether the compressed model still behaves like the uncompressed one. We measure that quantity directly. Our method, Cluster Intervention Gain (CIG), scores a candidate subset by how faithfully it preserves the full model's answer distribution, maximizing the negative KL divergence between the full-token and selected-token answer distributions through real subset prefills and a cluster-level greedy search; no labels or training are required. Under a controlled exact-576 protocol on InternVL3-8B, Qwen2.5-VL-7B, InternVL3.5-4B, and Qwen3-VL-4B, CIG is the strongest architecture-portable selector in all 24 backbone-budget settings, and paired intervals resolve the advantage for every backbone at k ≤ 64. Its mean margin over the best heuristic grows from +0.99 percentage points at k = 192 to +11.57 percentage points at k = 8, where it still retains at least 93.3% of full-token accuracy with 72× fewer visual tokens. A batched variant stays within 0.65 percentage points of the sequential selector while cutting selection time by 1.6–4.1×, and at serving-scale batching the shorter KV cache lowers end-to-end latency by up to 17.9% on a memory-pressured 8B workload, selection cost included. Ablations attribute most of the gain to the intervention objective itself rather than to the clustering heuristics that make the search tractable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Visual Token Selection, Token Pruning, Causal Intervention, Efficient Multimodal Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_V4WZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Cluster Intervention Gain (CIG), a training-free visual token selector for vision-language models. It obtains a reference answer distribution from the full model, clusters visual tokens, and evaluates candidate subsets through additional prefills. A greedy procedure selects subsets that minimize KL divergence from the reference while satisfying a token budget. Importantly, the implemented objective concerns a normalized first-answer-token distribution over a limited support, rather than complete generated answers. Evaluation includes four backbones under a fixed 576-token protocol, LLaVA comparisons, held-out checks, short free-form generation, ablations, and latency measurements. The strongest finding is improved accuracy under aggressive compression; efficiency and high-resolution transfer are more limited.

Critical strengths
  • Accuracy gains at very small token budgets. Table 1 reports positive paired intervals against the strongest eligible heuristic for every backbone at k≤64k \leq 64. At k=8k=8, the mean advantage is 11.57 percentage points. The partial held-out checks and short free-form generation results provide additional support.
Critical weaknesses

1. Compression requires executing the full-reference prefill first, and the practical benefit remains inadequately established. Algorithm 1 requires a full-reference prefill, a blind prefill, and multiple candidate prefills. For the short-answer tasks dominating the accuracy evaluation, the reference already supplies the next-token information used to answer the question. Reducing the final KV cache therefore does not establish reduced work for the complete request. The paper's own estimator gives about 14.1 TFLOPs for CIG at k=32k=32, versus about 8.1 TFLOPs for full, as shown in Appendix O. End-to-end latency is 0.6519s versus 0.1888s for Full and 0.1599s for CDpruner, as shown in Table 30, which is approximately 3.45 and 4.08 times slower. The 128-token caption workload is also approximately 24% slower than Full in Table 31. I am skeptical that these results demonstrate an optimal trade-off between performance and efficiency. The serving experiments leave important questions unresolved. They use eager attention and do not remeasure quality on the timed workloads. Table 28 reports identical selection costs at batch sizes 4 and 16 without explaining whether independent selection for every request is included. Furthermore, Section 5.4 and Table 5 categorically state that Qwen3-VL-4B shows no crossover and batch 4 does not cross, although Table 28 contains positive speedups in both regimes. These accounting and interpretation issues directly affect the efficiency claim. Despite detailed empirical reporting, I am not convinced that the paper meets the standard required for acceptance at ICLR.

2. The objective preserves a resticted readout, while important experiments substantially narrow the claimed advantage. Matching a normalized first-token distribution cannot constrain probability mass outside its support or preserve later generated tokens. The unspecified construction of the open-ended support further prevents independent reproduction. Short-answer generation checks are useful, but do not establish fidelity for the long-generation workloads used to motivate the probing cost. The unfavorable compressions are consequential. On LLaVA-1.5, the seven-task advantage over CDPruner is only 0.163 precentage points, with a 95% interval of [-0.305, 0.636]. At high resolution, its macro is 62.59 versus 65.68 for CDPruner, and TextVQA is 40.36 versus 50.13 in Table 25. On held-out TextVQA at k=128k=128, MMTok exceeds CIG on every backbone, as shown in Table 17. The oracle interpretation is additionally unsupported. Append Q infers that at most approximately one percentage point of the oracle gap comes from candidate-space and search limitations. Achieved scores under different objectives and candidate spaces cannot establish this bound: changing the objective changes the greedy trajectory, and no exact optimum or search guarantee is provided. To substantiate the claimed attribution of the oracle gap, the authors should provide quantitative comparisons of the greedy search trajectories and final scores under different objectives, holding the candidate space and search budget fixed.

Questions for authors

1. Does Table 28 include independent subset selection for every request in each batch? Why is selection cost identical at batch sizes 4 and 16? Please specify the execution and accounting, including reuse of the reference prefill's output or KV cache.

2. What is the exact open-ended answer-support construction, including its size, treatment of multi-token forms, and whether it remains fixed across candidate subsets? What λ\lambda is used in J3J_3, and how is δi\delta _i defined for the highest-density point?

3. Is there existing evidence that the serving advantage persists with optimized attention and preserves quality on the actual timed long-generation workloads? Appendix N uses eager attention and borrows accuracy from the short-answer grid.

4. Please reconcile the crossover statements with Table 28. Does Appendix Q's approximately one-percentage-point search-gap claim rely on exact optimization over a common candidate space? If it uses heuristic achieved scores, what justifies treating it as a bound?

Reviewer_EAFcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CIG, a training-free visual token selector that chooses which tokens to keep by directly measuring how closely the pruned model’s output distribution matches that of the full model, rather than relying on attention or similarity proxies. A cluster-level greedy search with real subset prefills makes this objective tractable. On four VLM backbones, CIG consistently outperforms existing pre-LLM selectors, especially at very low token budgets.

Critical strengths
  1. The writing is clear and easy to follow.

  2. The motivation and analysis of the problem are well-grounded.

  3. The evaluation is diverse and includes analysis of negative results.

Critical weaknesses
  1. From my understanding, all vision tokens will go through a full forward pass in each selection procedure. If this is true, why we don't directly use this forward pass? Is a full forward pass over all vision tokens required every time a user provides images? Could you also compare against methods with post-hoc KV eviction after a single full prefill?

  2. Following the above question, the full forward pass means the method can get full results when selecting tokens. Is this a fair comparison when comparing this method with other pre-LLM methods that don't have full results when selecting?

  3. The latency gain appears only for InternVL3-8B at batch 16, not on Qwen3-VL-4B or at batch 1 and 4. Is this a property of the method or of the specific GPU and memory configuration?

Questions for authors

See weaknesses.

Reviewer_p9voOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes a training-free efficient visual tokenization strategy for vision-language reasoning with large vision-language models (LVLMs). In particular, the authors build their method based on the differences between a base model's output in its vanilla setting vs under compression, and propose to prune visual tokens deemed "excessive". The authors further attempt to demonstrate the usefulness of their method primarily on 22 VQA benchmarks with supplementary results on several other benchmarks with the LLaVA 1.5 backbone only.

Critical strengths

The work has the following primary strengths:

  • Modeling token compression as a distributional distance minimization problem w.r.t. the original model's visual tokens is interesting and is a good fit as a general direction to pursue.

  • Some experimental results seem to support the proposed method's usefulness over the benchmarks it includes, e.g. in Tables 1 and 2.

Critical weaknesses

The work has the following primary weaknesses:

  • W1: Readability, Presentation and Narrative
    • Unfortunately, the vast majority work is very difficult to read and understand.
    • Throughout the work, there are multiple sentences spanning over 5 rows of text with many logical jumps in-between. This pattern starts from the abstract, persists through the first two sections and gets extremely exacerbated by the time the reader reaches the methodology and results sections.
    • Furthermore, the work has many ad-hoc definitions, an overly-granular description language (one look at the Algorithm 1 is sufficient to see the extent) which includes many vaguely defined statements, which further hinders the readability.
    • Even more importantly, the work contains "intriguing artifacts", very likely originating from AI agents. Although this pattern is consistent across almost all tables of the work, it is more visible in Tables 10, 13 and 18. In these tables, the authors respectively consider some "pass/fail gate”, count "Cells won: 24/24" and assigns different methods “PASS” or “NA” status under some odd protocoI.
    • Overall, I strongly believe that the work is very far away from being in a state that is acceptable, and its narrative hinders the value of any other contribution of the work substantially.

  • W2: Novelty Concerns, Missing Cited Works
    • The work leverages the existing model's outputs for visual token pruning, though there are several existing works from the literature that follow a similar strategy in essence, leveraging the difference between a base model and a model with a different number of visual tokens [A, B]. Although these works do not have the identical training settings, the authors should have at least cited and ideally quantitatively compared their approach against these works. The authors should also clarify in precise terms how their objective differs from the prior works.

  • W3: Problems with Experiments, Lack of Benchmarks and Unfair Comparisons
    • The work only reports results on 22 VQA benchmarks with its diverse backbone setting, which is very limited for the capabilities of LVLMs used in the work. However, based on the backbones utilized by the authors, and also considering the contemporary works in the literature [C, D, E]. Notably among these works, [D, E] note that many existing generic VQA benchmarks are not sensitive to visual token reduction/compression in the first place. Accordingly, the proposed method should be compared against the baselines under more benchmarks, including but not limited to ChartQA, DocVQA, RefCOCO and CV-Bench.
    • Furthermore, many results in the work provide against the usefulness of CIG, e.g. in Table 4 with LLaVA 1.5 backbone, CIG merely performs on-par with the baselines, while taking major hits at vision-centric benchmarks such as TextVQA.
    • The gains of the proposed method are also limited to particular settings. To exemplify, Table 5 shows the gains over the base model under a particular backbone, whereas another baseline has comparable or even better latency. In another table with the LLaVA 1.5 backbone at Table 29's Wall p50 column, the authors report 0.5701s for the proposed approach against 0.0982s for the primary baseline and 0.1328s for the base model at the matched 32 visual tokens. On top of these, the additional computation for the visual token strategy is also higher than that of the base model. Accordingly, comparing methods only at the same number of visual tokens does not imply efficiency improvements while instead showing that the proposed method comes at further added costs.

[A] Suo, W., Ma, J., Sun, M., Wu, L. Y., Wang, P., & Zhang, Y. (2025, October). Pruning all-rounder: Rethinking and improving inference efficiency for large vision language models. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 20247-20256). IEEE.

[B] Ji, H., Sun, Y., Jin, Y., Deng, C., Ding, J., Fu, L., & Wang, X. (2026, July). VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 9281-9301).

[C] Huang, W., Zhai, Z., Shen, Y., Cao, S., Zhao, F., Xu, X., ... & Lin, S. (2025, May). Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification. In International Conference on Learning Representations (Vol. 2025, pp. 69927-69955).

[D] Endo, M., Wang, X., & Yeung-Levy, S. (2025, October). Feather the throttle: Revisiting visual token pruning for vision-language model acceleration. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 22826-22835). IEEE.

[E] Kuzucu, S., Tonioni, A., Lup, V., Schiele, B., Tombari, F., & Naeem, M. F. (2026). PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding. arXiv preprint arXiv:2605.30126.

Questions for authors
  • Could the authors clarify the extent of AI usage in their work? As it stands, the work's AI use statement does not seem to reflect the true depth of AI usage in the work.

  • Could the authors report more results on more benchmarks, including but not limited to ChartQA, DocVQ, CV-Bench and RefCOCO? Ideally, having at least some video results is also useful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.