- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LAVIC, a training-free framework for visual token compression in vision-language models (VLMs). The paper argues that token importance estimated from visual features or attention may not accurately reflect which tokens are most useful to the downstream LLM after vision-language projection. To address this issue, LAVIC reframes visual token compression as preserving LLM-aligned visual evidence and introduces three components: (1) LLM-Expressive Evidence, which measures changes in projected representations induced by local perturbations and prioritizes tokens with higher scores; (2) Query-Anchored Evidence, which groups tokens selected for removal according to query-conditioned anchors and aggregates each group separately to retain information relevant to different queries; and (3) Evidence-Aware Capacity, which adaptively allocates token budgets across layers based on the distribution of query-conditioned relevance scores. Experiments across multiple VLM architectures, benchmarks, and input settings show that LAVIC achieves a competitive accuracy-efficiency trade-off. For example, on LLaVA-v1.5-7B, it reduces the number of visual tokens from 576 to 60 while achieving 96.4% of the uncompressed baseline's normalized aggregate performance. The paper also provides component-level analyses and runtime measurements.
Critical strengths
1. A meaningful problem formulation and a coherent framework. The paper highlights a potential mismatch between token importance estimated from visual features and the usefulness of projected tokens to the downstream LLM. To address this issue, LAVIC introduces three complementary components: a scoring method based on local perturbations in the projected representation space, an aggregation method that groups removed tokens around query related anchors, and an entropy based strategy for allocating token budgets across layers. Together, these components form a coherent pipeline that integrates token selection, information preservation, and adaptive compression.
2. Broad empirical evaluation with competitive results. The paper evaluates LAVIC across multiple VLM architectures and benchmarks, with different token budgets and visual input settings. LAVIC achieves competitive results across these settings, including under aggressive compression. For example, on LLaVA-v1.5-7B, it reduces the visual token count from 576 to 60 while retaining 96.4% of the uncompressed baseline's normalized aggregate performance. The paper also includes ablation studies of individual components, robustness analyses under different perturbation settings.
Critical weaknesses
1. The novelty of the proposed evidence criterion is not fully established. The paper proposes a scoring method that measures local changes in projected visual representations under perturbations. Although the experiments demonstrate its effectiveness, the paper does not sufficiently explain why these changes provide a principled measure of task relevant visual evidence. It is also unclear how this criterion differs conceptually from existing methods that use sensitivity to guide token pruning. A deeper analysis would help clarify the novelty and conceptual contribution of the proposed approach.
2. The necessity and individual contributions of the three components are not sufficiently established. The paper provides several useful ablation studies. Figure 5 compares different token scoring criteria, Figure 6 and Table 2 examine grouping and aggregation strategies, and Table 3 compares adaptive capacity allocation with fixed allocation. However, the paper does not systematically evaluate how performance and efficiency change as the components are added one by one under the same final token budget. For example, it would be useful to compare the initial token selection module alone, the first two modules with fixed capacity allocation, and the full framework. Table 6 reports the runtime overhead of individual components, but does not directly relate these costs to their improvements in task performance. Without such comparisons, it remains unclear whether the additional components provide enough benefit to justify their computational costs.
3. Compatibility with optimized attention implementations requires clarification. The aggregation module that uses query anchors relies on attention information to estimate the relevance of visual tokens to each anchor. However, the paper does not clearly explain how this information is obtained when using optimized implementations such as FlashAttention, which typically avoid explicitly storing the full attention matrix. The authors should clarify whether LAVIC can work efficiently with these implementations or requires additional computation, changes to the attention kernels, or a less efficient implementation. This information is important for assessing whether the reported speedups can be achieved in practical inference settings.
4. The handling of positional information and the preservation of spatial grounding are insufficiently characterized. The aggregation module combines visual representations from potentially different image locations into a single summary. However, the paper does not sufficiently explain how positional information is handled for these summaries in subsequent LLM layers, including how position indices or spatial coordinates are assigned, preserved, or updated. Preserving semantic information does not necessarily guarantee that spatial relationships are retained. The current evaluation focuses primarily on general visual understanding benchmarks and does not directly establish whether the method preserves detailed localization and spatial grounding capabilities. Further clarification of the positional encoding mechanism and evaluation on tasks that require spatial precision would strengthen the evidence for the paper's claims about information preservation.
Questions for authors
1. Compatibility with optimized attention implementations. How is the attention information required by the aggregation module obtained in practice? Specifically, can LAVIC work efficiently with FlashAttention or other optimized attention implementations, or does it require additional computation, changes to the attention kernels, or a less efficient implementation? If available, a latency comparison with a baseline using the same optimized attention implementation would help clarify whether the reported speedups can be achieved in practical inference settings.
2. Individual contributions of the three components. Could the authors clarify how much each component contributes to the accuracy and efficiency of LAVIC? In particular, is there an ablation comparing (i) the initial token selection module alone, (ii) the first two modules with fixed capacity allocation, and (iii) the full framework, under the same final visual token budget and evaluation settings? Table 6 reports the runtime overhead of individual components, but these costs are not directly linked to their improvements in task performance. Such a comparison would help determine whether the additional components provide sufficient benefits to justify their computational costs.
3. Positional information and spatial grounding after aggregation. How is positional information handled when visual tokens from different image locations are combined into a single summary and passed to subsequent LLM layers? For example, how are position indices or spatial coordinates assigned or preserved for these summaries, particularly in architectures that use M-RoPE or explicit spatial positional encodings? Clarifying the implementation and providing experimental evidence would help assess whether LAVIC adequately preserves both semantic and spatial information for position-sensitive visual understanding.
6 comments
Follow-up to Reviewer w9xv
Dear Reviewer w9xv,
We sincerely thank Reviewer w9xv for appreciating our problem formulation, the coherent design of LAVIC, and its competitive performance under aggressive compression across diverse settings.
We hope these clarifications adequately address all your concerns, and we are glad to clarify anything further.
Best, The Authors
Thank you for your detailed responses. However, I believe that the following two concerns have not yet been adequately addressed, and I would appreciate further clarification from the authors.
-
Spatial reasoning and visual grounding. I believe there is an important distinction between spatial reasoning and visual grounding. Could the authors provide experimental results on dedicated visual grounding benchmarks (e.g., RefCOCO), where models are required to explicitly localize objects in images by predicting point coordinates or bounding boxes? In particular, my original concern regarding positional encoding was motivated by the potential impact of token aggregation on such localization capabilities. It would therefore be helpful to separately quantify the performance degradation on these tasks after visual token compression.
-
Compatibility with optimized attention implementations. Could the authors clarify whether the Vanilla baseline in Fig. 7(a) was evaluated using standard SDPA, FlashAttention, or another attention implementation? I believe that a newly proposed acceleration method should ideally either provide additional speedups over widely adopted inference optimization techniques or be efficiently compatible with them. If the proposed method neither achieves a speed advantage over existing optimization techniques nor supports efficient integration with them, its practical applicability would be considerably limited.
Response to Reviewer w9xv
We thank the reviewer for the constructive feedback. We have addressed the key concerns through additional analyses, experiments, and clarifications, and improved the figures for clarity.
1. Distinction and Validation of LLM-Expressive Evidence.
Conceptual Distinction. Rather than simply measuring visual-token sensitivity to perturbations, LAVIC evaluates how distinguishable local visual variations remain in the LLM-facing representation after the vision-to-LLM projector. Although this distinguishability does not directly guarantee semantic importance, it provides a practical criterion for preserving visual evidence accessible to the LLM.
Empirical Validation (Fig. 5; Appendix E.1). Removing visual tokens with high LLM-expressive evidence scores causes greater performance degradation than removing an equal number of low-scoring or randomly selected tokens (Fig. 5(b)), suggesting that our score identifies visual information important for downstream LLM processing. Furthermore, our criterion retains more answer-relevant OCR regions (Fig. 5(c)) and achieves stronger answer–distractor discrimination (Appendix E.1, Table 13), supporting its ability to preserve evidence useful for answering the query. Consistent with these findings, replacing our score with ZOO-Prune's sensitivity score under the same compression pipeline reduces MME performance from 1795 to 1776 (Fig. 5(a)), demonstrating more effective visual-token selection.
2. Compatibility with Optimized Attention Implementations.
Backbone-Specific Attention Implementation. LAVIC obtains the attention weights required for query-anchor aggregation according to each backbone's implementation. LLaVA-v1.5 explicitly computes attention weights during prefill and uses optimized SDPA for decoding. LLaVA-NeXT and LLaVA-OneVision retain optimized attention in non-pruning layers while computing attention weights explicitly at pruning layers. Other backbones use separate attention computations or eager attention. No FlashAttention kernel modifications are required, although attention extraction incurs only modest additional computational overhead.
Measured Runtime Overhead (Table 4, Fig. 7(a)). These costs are included in the reported wall-clock latency and component-wise overhead measurements. In Fig. 7(a), LAVIC reduces prefill latency from 58.70 ms to 28.95 ms, including 3.011 ms of additional overhead. Thus, the reported efficiency gains account for the cost of attention extraction.
3. Component-Wise Contributions and Efficiency.
Component Ablation (Table 4). Table 4 reports both performance and latency for incremental component configurations. On TextVQA, LLM-expressive evidence alone achieves 51.8 (45.4 ms), while adding evidence-aware capacity without query-anchored evidence achieves 52.1 (46.2 ms). The full LAVIC achieves 55.1 (48.5 ms). Replacing LLM-expressive scoring with random selection or removing or replacing query-anchored summarization also reduces performance.
Performance–Cost Analysis (Table 4, Fig. 7(a)). Compared with initial filtering alone, the full LAVIC improves TextVQA performance by 3.3 points with only 3.1 ms of additional latency. Fig. 7(a) further breaks down the overhead of the three components, relating their performance contributions to their computational costs.
4. Positional Information and Spatial Grounding.
Backbone-Specific Position Handling (Appendix A, Table 7). LAVIC preserves each backbone's positional encoding scheme. For LLaVA-family models, position indices are reindexed to match the compressed 1D sequence. InternVL2.5 preserves the original positions of retained tokens and assigns subsequent positions to summary tokens. For M-RoPE-based Qwen3.5-4B, retained tokens preserve their original 3D position indices, while summary tokens use the mean positions of the tokens they summarize.
Evaluation Related to Spatial Understanding (Tables 1 and 5). BLINK includes object-localization and spatial-relation tasks, while GQA evaluates visual reasoning involving spatial relationships. On BLINK, LAVIC achieves 45.6, compared with 43.3 for SparseVLM, at 92.7% token reduction on InternVL2.5-8B (Table 5). On GQA, LLaVA-v1.5-7B retains 97.1% of full-token performance at 88.9% token reduction (Table 1). These results demonstrate competitive performance on benchmarks involving spatial understanding.