acceptodds
Withdrawn from ICLR 2027

CORGI: Concept-Oriented Redundancy-Guided Image Token Compression for Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

23 views

Abstract

Visual token compression aims to reduce the computational cost of Vision-Language Models (VLMs) by removing redundant or non-informative visual tokens before or during multimodal processing. Existing visual token pruning methods suffer from two major limitations. First, their pruning decisions are semantically opaque: practitioners cannot determine what information the kept or discarded tokens represent or control generation by selecting which visual tokens should be preserved for a particular task or application. Second, many methods require layer-specific design choices and hyperparameter tuning, such as selecting the transformer layers at which pruning should be performed. We introduce the first interpretable and controllable visual token pruning framework for VLMs. Our method identifies the semantic meaning of visual tokens in natural language by leveraging the model’s shared vision-language representation space. This allows an easy, human-friendly way of selecting visual tokens for pruning according to their textual meaning. The resulting token selection is performed before the visual tokens are provided to the VLM, eliminating the need for additional layer-wise token dropping or pruning schedules. Experiments on image and video benchmarks across both natural and medical domains demonstrate that our method often outperforms the state-of-the-art while offering greater interpretability, explicit control, and a simpler pruning pipeline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Language Model, Visual Token Pruning, MLLMs, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BqGzOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CORGI, a concept-based visual token compression framework. It uses a distilled predictor to assign vocabulary labels to visual tokens, filters tokens using a common-word vocabulary, and optionally retains only a few representatives per concept. The labels also enable manual concept-level intervention. Experiments cover image captioning, VQA, and medical report generation, demonstrating compression benefits.

Critical strengths
  1. The paper provides an approach to visual token selection by assigning readable vocabulary labels to tokens. This makes pruning decisions easier to inspect and provides an interface for concept-level intervention.
  2. The proposed pipeline is clearly described and easy to follow.
Critical weaknesses
  1. The distinction from closely related interpretability work is insufficiently discussed. "Towards Interpreting Visual Information Processing in Vision-Language Models" (ICLR 2025) already applies Logit Lens to decode visual token representations into vocabulary entries. CORGI’s main extension appears to be adapting this mechanism to token compression through concept-based selection. The additional methodological contribution should be clarified.
  2. The reported efficiency does not cover the complete pipeline. GFLOPs account only for LLM prefill and explicitly exclude pre-LLM processing, including the concept predictor. The predictor uses a d→2d→Vd\rightarrow2d\rightarrow V architecture, whose vocabulary-sized output layer may introduce substantial computation and parameters. Without end-to-end latency and memory measurements, the practical efficiency advantage remains unclear.
  3. Readable vocabulary labels do not establish faithful token interpretations. A top-1 vocabulary prediction may capture only part of a visual token’s information, while punctuation or an uncommon word does not necessarily indicate that the token is uninformative.
  4. The efficient variant relies on an additionally trained concept predictor whose transfer fidelity is insufficiently evaluated. The predictor is trained using 80k COCO 2014 images, split into training and validation sets. Although downstream results include medical images and videos, the accuracy of its concept predictions on these domains is not directly evaluated.
  5. Concept-level controllability is supported mainly by qualitative examples. The paper does not quantify suppression success or preservation of unrelated content.
  6. Performance preservation is uneven, and most comparisons focus on a single baseline. CORGI underperforms Duet-VLM on LLaVA-1.5 GQA and across all reported caption-quality metrics in the 32B LLaVA-Bench setting. Moreover, general image-VQA evaluation is limited to GQA, restricting the evidence for its effectiveness across broader question-answering tasks.
Questions for authors

see weakness

Reviewer_MqbdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces CORGI, a concept-oriented visual-token compression framework for vision-language models (VLMs). The key idea is to project each visual token into the language model's vocabulary space and use the resulting lexical token as a human-readable concept annotation. To avoid running all visual tokens through the full language-model decoder before pruning, the authors train a lightweight distillation network to predict the final-layer vocabulary distribution directly from the projected visual features. CORGI first removes tokens whose predicted lexical labels do not belong to a predefined list of 20K common English words, and then performs redundancy reduction by retaining only the top-k highest-confidence visual tokens assigned to the same predicted concept. Since pruning is performed before the visual tokens enter the LLM, the method avoids layer-wise pruning schedules and also allows users to intervene by explicitly retaining or suppressing selected concepts.

The paper evaluates CORGI on image captioning, visual question answering, video question answering, and medical report generation using Qwen2.5-VL and LLaVA backbones. The results suggest that CORGI can achieve aggressive visual-token reduction while remaining competitive with Duet-VLM and several previously reported pruning baselines. The concept-oriented formulation is interesting and potentially useful for building more transparent and controllable VLM inference systems. However, the paper's central claim that the vocabulary projection reveals the "semantic meaning" of individual visual tokens is not yet supported by sufficiently direct evidence. Important implementation details concerning BPE tokenization and concept matching are also unclear, and the empirical comparison should be expanded and standardized before strong state-of-the-art claims can be established.

Critical strengths
  1. The method is interesting and relatively novel. Instead of treating visual-token selection as a black-box ranking problem based only on attention, feature similarity, or learned importance scores, CORGI attempts to expose the lexical concept associated with each retained or discarded patch. This is a distinctive perspective on visual-token compression and creates a natural interface for concept-level inspection and intervention.

  2. The paper is well organized and contains substantial analysis. The presentation is generally clear, and the paper covers a broad set of issues, including the cost of obtaining final-layer concept labels, layer selection, soft versus hard distillation, image-text alignment across backbones, the choice of the per-concept token budget, domain-specific vocabularies, and concept-level intervention. The authors have anticipated many natural concerns and provide explanations or preliminary analyses for them. Although some of these analyses are not yet sufficiently conclusive, their inclusion makes the paper informative and shows that the authors have thought carefully about the limitations of the proposed approach.

Critical weaknesses

1.The claim that the predicted vocabulary token represents the semantic meaning of a visual token is insufficiently validated. The central premise of CORGI is that a final-layer visual-token representation can be projected through the LLM unembedding matrix and interpreted as a semantically meaningful natural-language concept. It is dimensionally valid to apply the unembedding matrix to a visual hidden state, but this alone does not establish that the highest-probability vocabulary token is the correct semantic label for the corresponding image patch. In a decoder-only VLM, visual-token positions are not necessarily trained with a direct next-token prediction objective, so a reliable patch-to-word correspondence is not guaranteed by the training objective.

The current evidence is indirect. Lower entropy and higher confidence only show that the vocabulary distribution is more concentrated, not that the predicted concept is correct. Likewise, improved downstream performance at the final layer does not directly measure the semantic accuracy of the concept annotations. The Visual Genome coverage analysis is useful, but coverage alone does not report concept precision, recall, localization correctness, or calibration. Consequently, terms such as "the semantic meaning of a visual token" and "fully interpretable concept" currently appear stronger than the presented evidence supports.

The paper would benefit from a direct concept-grounding evaluation. For example, the authors could obtain object and region annotations using ground-truth semantic-segmentation datasets or a strong external segmentation and recognition model. Each visual patch could then be assigned the object category or semantic region covering that patch, and this reference label could be compared with the concept produced by CORGI. The evaluation should report exact or synonym-aware concept accuracy, precision and recall for retained concepts, localization consistency, and calibration as a function of CORGI's confidence. It would also be useful to separately evaluate object categories, attributes, stuff/background regions, and patches containing multiple objects. Such an experiment would directly test whether CORGI discovers genuine visual semantics rather than merely producing confident lexical projections. If the correspondence is only approximate, the claims should be revised to describe the outputs as vocabulary-aligned lexical proxies rather than the semantic meaning of visual tokens.

2.The treatment of BPE tokenization and the 20K-word vocabulary is underspecified. The method appears to assign each visual patch a single vocabulary ID using an argmax over the predicted vocabulary distribution, decode that ID into text, and retain the patch if the decoded string belongs to a list of 20K common English words. However, modern VLMs use BPE or related subword tokenization, under which a word or concept may consist of multiple tokens, while an individual token may represent a word, a stem, a suffix, a whitespace-marked form, or another fragment. The appendix argues that frequent words and meaningful stems often exist as standalone tokens, but this heuristic does not fully specify or justify the matching procedure.

The authors should clarify the following details:

  • What happens to valid concepts that require multiple tokens, such as compound nouns, or that are split differently by different model tokenizers?
  • The 20K list contains a large number of concepts that are absent from any particular image. Does the list merely act as a whitelist after each patch independently predicts its top-1 vocabulary token, or is there an explicit image-level mechanism for determining which concepts are actually present?

3.The empirical comparison and benchmark coverage are not sufficiently comprehensive or standardized. Most experiments on Qwen, video question answering, and medical report generation compare CORGI primarily with Duet-VLM. Additional pruning methods such as FastV, HiRED, PruMerge, SparseVLM, PyramidDrop, and VisionZip are included only in the LLaVA/GQA setting, and several of those results are taken from prior work rather than reproduced under a unified protocol. This is not sufficient to establish a broad state-of-the-art claim in a rapidly developing visual-token compression literature. The evaluation should include more recent and representative methods from the last several years, covering pre-LLM pruning, token merging, progressive decoder-side pruning, and hybrid approaches.

Questions for authors

Just described in Critical Weaknesses.

Reviewer_xXXQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CORGI compresses visual tokens by assigning each patch a vocabulary-level concept. A separately trained two-layer distiller predicts the final-layer vocabulary distribution from projected visual features. Patches whose predicted concepts fall outside a common-English-word list are discarded; an optional per-concept top-k rule removes repeated concepts. The retained tokens enter the language decoder without layer-wise pruning schedules. Evaluation covers image captioning, image and video question answering, and medical report generation, primarily with Qwen2.5-VL-7B, with additional Qwen2.5-VL-32B and LLaVA-1.5-7B experiments. The paper reports large decoder-prefill reductions, gains over Duet-VLM in several settings, and qualitative concept interventions.

Critical strengths
  1. Vocabulary-level labels make token selection easy to inspect and allow users to exclude specific concepts. Predicting these labels before the decoder avoids an additional full-decoder pass in the main inference configuration. This design offers a clear interface for inspecting and controlling token selection, although the faithfulness of the labels requires further validation.
  2. The evaluation covers image captioning, video QA and medical report generation. On MSVD-QA, CORGI achieves 31.9% accuracy versus 28.0% for Duet-VLM, with pruning rates of 94.0% and 93.5%, respectively (Table 3). Gains in several captioning and medical-report settings further support its practical potential, though performance varies across settings.
Critical weaknesses
  1. Appendix A.3.2 excludes the distiller from the reported FLOPs, despite its estimated 1.1 billion parameters. Omitting this cost may overstate efficiency gains. The accounting also needs clarification, since the no-distillation ablation includes the cost of an additional forward pass.
  2. The paper reports estimated LLM-prefill FLOPs but no measured end-to-end latency, throughput or peak GPU memory usage. Its practical speed and memory benefits therefore remain unclear.
  3. Comparisons focus on Duet-VLM. Without additional baselines such as VisionZip and random pruning at matched token counts or compute budgets, the contribution of semantic selection is difficult to isolate.
  4. Controllability is supported by seven qualitative examples, without systematic measurements of target-concept suppression or effects on descriptions of unrelated content.
Questions for authors
  1. What are the end-to-end latency, throughput and peak GPU memory usage of CORGI, the unpruned VLM and Duet-VLM under identical hardware, inputs and decoding settings? Please include the distiller and all preprocessing overhead, and apply a consistent cost-accounting scope across the distillation ablations.
  2. Does CORGI outperform VisionZip and random pruning when retained-token counts or compute budgets are matched? Please provide quality–cost comparisons that isolate the benefit of semantic selection from that of token reduction alone. 3.On a larger, clearly defined image set, how often does concept deletion suppress the target concept while preserving unrelated content? Please report target-concept occurrence before and after deletion, and quantify changes in descriptions of unrelated visual content to assess unintended information loss.
  3. Could CORGI use a smaller output head covering approximately 20,000 allowed words plus a rejection class, or a low-rank output head? Please evaluate these alternatives and report their parameter, runtime and memory savings, together with any changes in token selection and pruning quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.