- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CORGI, a concept-based visual token compression framework. It uses a distilled predictor to assign vocabulary labels to visual tokens, filters tokens using a common-word vocabulary, and optionally retains only a few representatives per concept. The labels also enable manual concept-level intervention. Experiments cover image captioning, VQA, and medical report generation, demonstrating compression benefits.
Critical strengths
- The paper provides an approach to visual token selection by assigning readable vocabulary labels to tokens. This makes pruning decisions easier to inspect and provides an interface for concept-level intervention.
- The proposed pipeline is clearly described and easy to follow.
Critical weaknesses
- The distinction from closely related interpretability work is insufficiently discussed. "Towards Interpreting Visual Information Processing in Vision-Language Models" (ICLR 2025) already applies Logit Lens to decode visual token representations into vocabulary entries. CORGI’s main extension appears to be adapting this mechanism to token compression through concept-based selection. The additional methodological contribution should be clarified.
- The reported efficiency does not cover the complete pipeline. GFLOPs account only for LLM prefill and explicitly exclude pre-LLM processing, including the concept predictor. The predictor uses a architecture, whose vocabulary-sized output layer may introduce substantial computation and parameters. Without end-to-end latency and memory measurements, the practical efficiency advantage remains unclear.
- Readable vocabulary labels do not establish faithful token interpretations. A top-1 vocabulary prediction may capture only part of a visual token’s information, while punctuation or an uncommon word does not necessarily indicate that the token is uninformative.
- The efficient variant relies on an additionally trained concept predictor whose transfer fidelity is insufficiently evaluated. The predictor is trained using 80k COCO 2014 images, split into training and validation sets. Although downstream results include medical images and videos, the accuracy of its concept predictions on these domains is not directly evaluated.
- Concept-level controllability is supported mainly by qualitative examples. The paper does not quantify suppression success or preservation of unrelated content.
- Performance preservation is uneven, and most comparisons focus on a single baseline. CORGI underperforms Duet-VLM on LLaVA-1.5 GQA and across all reported caption-quality metrics in the 32B LLaVA-Bench setting. Moreover, general image-VQA evaluation is limited to GQA, restricting the evidence for its effectiveness across broader question-answering tasks.
Questions for authors
see weakness