acceptodds
Under review as a conference paper at ICLR 2027

Mind the Evidence: LLM-Aligned Visual Token Compression for VLMs

14 views

Abstract

Vision-language models increasingly rely on large numbers of visual tokens, making visual token compression essential for efficient inference. However, existing token pruning methods typically estimate token importance from the vision encoder or from attention/relevance patterns, without explicitly considering how visual information is transformed before reaching the language model. This creates a fundamental mismatch: a token that appears important in the visual representation may contribute little to the representation actually processed by the LLM. We introduce LAVIC, a training-free framework for compressing visual tokens from the perspective of the LLM-facing representation. LAVIC measures how distinguishable local visual variations remain in the LLM-facing representation, using this distinguishability as a signal of the visual evidence exposed to the downstream language model. Based on this evidence, LAVIC preserves informative tokens, groups redundant evidence around query-relevant anchors, and adaptively allocates the limited token budget according to the concentration of the evidence. Across multiple vision-language models and benchmarks, LAVIC consistently preserves model performance under aggressive token compression while substantially reducing visual tokens and inference cost. On LLaVA-v1.5, LAVIC retains 96.4% of the full-token performance using only 60 of 576 visual tokens, while reducing prefill latency from 58.70 ms to 28.95 ms. At 128 and 64 tokens, LAVIC also maintains strong performance and consistently outperforms competing compression methods. These results suggest that visual token compression can be better understood as preserving LLM-facing visual evidence rather than simply selecting visually salient tokens.

Primary area:
optimization
Keywords:
Token Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_w9xvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LAVIC, a training-free framework for visual token compression in vision-language models (VLMs). The paper argues that token importance estimated from visual features or attention may not accurately reflect which tokens are most useful to the downstream LLM after vision-language projection. To address this issue, LAVIC reframes visual token compression as preserving LLM-aligned visual evidence and introduces three components: (1) LLM-Expressive Evidence, which measures changes in projected representations induced by local perturbations and prioritizes tokens with higher scores; (2) Query-Anchored Evidence, which groups tokens selected for removal according to query-conditioned anchors and aggregates each group separately to retain information relevant to different queries; and (3) Evidence-Aware Capacity, which adaptively allocates token budgets across layers based on the distribution of query-conditioned relevance scores. Experiments across multiple VLM architectures, benchmarks, and input settings show that LAVIC achieves a competitive accuracy-efficiency trade-off. For example, on LLaVA-v1.5-7B, it reduces the number of visual tokens from 576 to 60 while achieving 96.4% of the uncompressed baseline's normalized aggregate performance. The paper also provides component-level analyses and runtime measurements.

Critical strengths

1. A meaningful problem formulation and a coherent framework. The paper highlights a potential mismatch between token importance estimated from visual features and the usefulness of projected tokens to the downstream LLM. To address this issue, LAVIC introduces three complementary components: a scoring method based on local perturbations in the projected representation space, an aggregation method that groups removed tokens around query related anchors, and an entropy based strategy for allocating token budgets across layers. Together, these components form a coherent pipeline that integrates token selection, information preservation, and adaptive compression.

2. Broad empirical evaluation with competitive results. The paper evaluates LAVIC across multiple VLM architectures and benchmarks, with different token budgets and visual input settings. LAVIC achieves competitive results across these settings, including under aggressive compression. For example, on LLaVA-v1.5-7B, it reduces the visual token count from 576 to 60 while retaining 96.4% of the uncompressed baseline's normalized aggregate performance. The paper also includes ablation studies of individual components, robustness analyses under different perturbation settings.

Critical weaknesses

1. The novelty of the proposed evidence criterion is not fully established. The paper proposes a scoring method that measures local changes in projected visual representations under perturbations. Although the experiments demonstrate its effectiveness, the paper does not sufficiently explain why these changes provide a principled measure of task relevant visual evidence. It is also unclear how this criterion differs conceptually from existing methods that use sensitivity to guide token pruning. A deeper analysis would help clarify the novelty and conceptual contribution of the proposed approach.

2. The necessity and individual contributions of the three components are not sufficiently established. The paper provides several useful ablation studies. Figure 5 compares different token scoring criteria, Figure 6 and Table 2 examine grouping and aggregation strategies, and Table 3 compares adaptive capacity allocation with fixed allocation. However, the paper does not systematically evaluate how performance and efficiency change as the components are added one by one under the same final token budget. For example, it would be useful to compare the initial token selection module alone, the first two modules with fixed capacity allocation, and the full framework. Table 6 reports the runtime overhead of individual components, but does not directly relate these costs to their improvements in task performance. Without such comparisons, it remains unclear whether the additional components provide enough benefit to justify their computational costs.

3. Compatibility with optimized attention implementations requires clarification. The aggregation module that uses query anchors relies on attention information to estimate the relevance of visual tokens to each anchor. However, the paper does not clearly explain how this information is obtained when using optimized implementations such as FlashAttention, which typically avoid explicitly storing the full attention matrix. The authors should clarify whether LAVIC can work efficiently with these implementations or requires additional computation, changes to the attention kernels, or a less efficient implementation. This information is important for assessing whether the reported speedups can be achieved in practical inference settings.

4. The handling of positional information and the preservation of spatial grounding are insufficiently characterized. The aggregation module combines visual representations from potentially different image locations into a single summary. However, the paper does not sufficiently explain how positional information is handled for these summaries in subsequent LLM layers, including how position indices or spatial coordinates are assigned, preserved, or updated. Preserving semantic information does not necessarily guarantee that spatial relationships are retained. The current evaluation focuses primarily on general visual understanding benchmarks and does not directly establish whether the method preserves detailed localization and spatial grounding capabilities. Further clarification of the positional encoding mechanism and evaluation on tasks that require spatial precision would strengthen the evidence for the paper's claims about information preservation.

Questions for authors

1. Compatibility with optimized attention implementations. How is the attention information required by the aggregation module obtained in practice? Specifically, can LAVIC work efficiently with FlashAttention or other optimized attention implementations, or does it require additional computation, changes to the attention kernels, or a less efficient implementation? If available, a latency comparison with a baseline using the same optimized attention implementation would help clarify whether the reported speedups can be achieved in practical inference settings.

2. Individual contributions of the three components. Could the authors clarify how much each component contributes to the accuracy and efficiency of LAVIC? In particular, is there an ablation comparing (i) the initial token selection module alone, (ii) the first two modules with fixed capacity allocation, and (iii) the full framework, under the same final visual token budget and evaluation settings? Table 6 reports the runtime overhead of individual components, but these costs are not directly linked to their improvements in task performance. Such a comparison would help determine whether the additional components provide sufficient benefits to justify their computational costs.

3. Positional information and spatial grounding after aggregation. How is positional information handled when visual tokens from different image locations are combined into a single summary and passed to subsequent LLM layers? For example, how are position indices or spatial coordinates assigned or preserved for these summaries, particularly in architectures that use M-RoPE or explicit spatial positional encodings? Clarifying the implementation and providing experimental evidence would help assess whether LAVIC adequately preserves both semantic and spatial information for position-sensitive visual understanding.

6 comments
AuthorsOpenReview ↗

Response to Reviewer w9xv

We thank the reviewer for the constructive feedback. We have addressed the key concerns through additional analyses, experiments, and clarifications, and improved the figures for clarity.

 

1. Distinction and Validation of LLM-Expressive Evidence.

  • Conceptual Distinction. Rather than simply measuring visual-token sensitivity to perturbations, LAVIC evaluates how distinguishable local visual variations remain in the LLM-facing representation after the vision-to-LLM projector. Although this distinguishability does not directly guarantee semantic importance, it provides a practical criterion for preserving visual evidence accessible to the LLM.

  • Empirical Validation (Fig. 5; Appendix E.1). Removing visual tokens with high LLM-expressive evidence scores causes greater performance degradation than removing an equal number of low-scoring or randomly selected tokens (Fig. 5(b)), suggesting that our score identifies visual information important for downstream LLM processing. Furthermore, our criterion retains more answer-relevant OCR regions (Fig. 5(c)) and achieves stronger answer–distractor discrimination (Appendix E.1, Table 13), supporting its ability to preserve evidence useful for answering the query. Consistent with these findings, replacing our score with ZOO-Prune's sensitivity score under the same compression pipeline reduces MME performance from 1795 to 1776 (Fig. 5(a)), demonstrating more effective visual-token selection.

 

2. Compatibility with Optimized Attention Implementations.

  • Backbone-Specific Attention Implementation. LAVIC obtains the attention weights required for query-anchor aggregation according to each backbone's implementation. LLaVA-v1.5 explicitly computes attention weights during prefill and uses optimized SDPA for decoding. LLaVA-NeXT and LLaVA-OneVision retain optimized attention in non-pruning layers while computing attention weights explicitly at pruning layers. Other backbones use separate attention computations or eager attention. No FlashAttention kernel modifications are required, although attention extraction incurs only modest additional computational overhead.

  • Measured Runtime Overhead (Table 4, Fig. 7(a)). These costs are included in the reported wall-clock latency and component-wise overhead measurements. In Fig. 7(a), LAVIC reduces prefill latency from 58.70 ms to 28.95 ms, including 3.011 ms of additional overhead. Thus, the reported efficiency gains account for the cost of attention extraction.

 

3. Component-Wise Contributions and Efficiency.

  • Component Ablation (Table 4). Table 4 reports both performance and latency for incremental component configurations. On TextVQA, LLM-expressive evidence alone achieves 51.8 (45.4 ms), while adding evidence-aware capacity without query-anchored evidence achieves 52.1 (46.2 ms). The full LAVIC achieves 55.1 (48.5 ms). Replacing LLM-expressive scoring with random selection or removing or replacing query-anchored summarization also reduces performance.

  • Performance–Cost Analysis (Table 4, Fig. 7(a)). Compared with initial filtering alone, the full LAVIC improves TextVQA performance by 3.3 points with only 3.1 ms of additional latency. Fig. 7(a) further breaks down the overhead of the three components, relating their performance contributions to their computational costs.

 

4. Positional Information and Spatial Grounding.

  • Backbone-Specific Position Handling (Appendix A, Table 7). LAVIC preserves each backbone's positional encoding scheme. For LLaVA-family models, position indices are reindexed to match the compressed 1D sequence. InternVL2.5 preserves the original positions of retained tokens and assigns subsequent positions to summary tokens. For M-RoPE-based Qwen3.5-4B, retained tokens preserve their original 3D position indices, while summary tokens use the mean positions of the tokens they summarize.

  • Evaluation Related to Spatial Understanding (Tables 1 and 5). BLINK includes object-localization and spatial-relation tasks, while GQA evaluates visual reasoning involving spatial relationships. On BLINK, LAVIC achieves 45.6, compared with 43.3 for SparseVLM, at 92.7% token reduction on InternVL2.5-8B (Table 5). On GQA, LLaVA-v1.5-7B retains 97.1% of full-token performance at 88.9% token reduction (Table 1). These results demonstrate competitive performance on benchmarks involving spatial understanding.

AuthorsOpenReview ↗

Follow-up to Reviewer w9xv

Dear Reviewer w9xv,

We sincerely thank Reviewer w9xv for appreciating our problem formulation, the coherent design of LAVIC, and its competitive performance under aggressive compression across diverse settings.

We hope these clarifications adequately address all your concerns, and we are glad to clarify anything further.

Best, The Authors

Thank you for your detailed responses. However, I believe that the following two concerns have not yet been adequately addressed, and I would appreciate further clarification from the authors.

  1. Spatial reasoning and visual grounding. I believe there is an important distinction between spatial reasoning and visual grounding. Could the authors provide experimental results on dedicated visual grounding benchmarks (e.g., RefCOCO), where models are required to explicitly localize objects in images by predicting point coordinates or bounding boxes? In particular, my original concern regarding positional encoding was motivated by the potential impact of token aggregation on such localization capabilities. It would therefore be helpful to separately quantify the performance degradation on these tasks after visual token compression.

  2. Compatibility with optimized attention implementations. Could the authors clarify whether the Vanilla baseline in Fig. 7(a) was evaluated using standard SDPA, FlashAttention, or another attention implementation? I believe that a newly proposed acceleration method should ideally either provide additional speedups over widely adopted inference optimization techniques or be efficiently compatible with them. If the proposed method neither achieves a speed advantage over existing optimization techniques nor supports efficient integration with them, its practical applicability would be considerably limited.

Reviewer_3AFoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LAVIC is a training-free visual token compression framework that integrates visual feature response analysis, LLM query anchor-guided token selection and summarization, and an adaptive layer-wise budget allocation. The paper evaluates the proposed method across various vision-language models (VLMs), multiple input modalities (single-image, multi-image, and video), and several benchmark datasets, demonstrating its effectiveness in maintaining high performance even under high compression ratios.

Critical strengths
  • Training-Free and Effective Framework: The paper proposes a training-free visual token compression method that achieves solid performance on test benchmarks while significantly reducing the number of tokens.

  • Evaluation Across Multiple Settings: The method is evaluated under diverse model architectures, input modalities, and configuration settings, demonstrating its versatility.

Critical weaknesses
  • Incomplete and Inconsistent Benchmark Evaluation: The benchmark selection across different backbones is insufficient and inconsistent. While baseline methods like VisionZip evaluate up to 11 benchmarks, this paper tests LLaVA-v1.5 on only 5 datasets and LLaVA-NeXT on only 2 (MME and ScienceQA). Furthermore, the evaluated datasets vary significantly across different model architectures and baseline tables, making cross-model comparisons inconsistent. Because the reported averages are heavily influenced by a narrow set of datasets (e.g., gains on ScienceQA skewing the average), the claims of outperforming existing baselines are not convincingly supported by the provided results.

  • Limited Novelty: Each core component of the proposed pipeline closely resembles prior work. Specifically, the projector perturbation scoring is very similar to ZOO-Prune, the anchor selection and token recycling/summarization mirror SparseVLM, and the adaptive layer-wise budget allocation has been explored in PDrop and ATP-LLaVA. The paper lacks ablation studies that swap out individual components with these published alternatives within the same pipeline, making it unclear where the performance gains actually originate.

Questions for authors
  • Benchmark Consistency and Expansion: Can you evaluate LAVIC across a uniform and broader set of standard benchmarks (matching VisionZip’s protocol with datasets like GQA, VQAv2, MMMU, etc.) across all backbones to ensure a consistent and solid comparison?

  • Component-Level Ablation: Can you conduct ablation experiments replacing each module with its closest precedent (e.g., ZOO-Prune's scoring, SparseVLM's anchor selection and recycling) under the same pipeline to demonstrate the specific advantages of your design choices?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 3AFo

We thank the reviewer for the constructive feedback. We have addressed the key concerns through additional analyses, experiments, and clarifications, and improved the figures for clarity.

 

1. Benchmark Consistency and Expanded Evaluation.

  • Unified Benchmark Protocol (Table 1). We revised the main evaluation to use the same four benchmarks—MME, TextVQA, GQA, and MMMU—for both LLaVA-v1.5-7B and LLaVA-NeXT-7B. We also report per-benchmark scores and average relative performance under comparable visual-token budgets.

  • Performance Across Compression Budgets (Table 1). The expanded evaluation shows that LAVIC maintains competitive performance across compression settings, with clearer advantages under aggressive compression. At the highest evaluated reduction levels, LAVIC retains 96.0% and 83.4% of full-token performance on LLaVA-v1.5 and LLaVA-NeXT, respectively, outperforming the strongest compared baselines.

 

2. Component-Level Comparisons with Existing Methods.

We clarify the existing component-level comparisons in Fig. 5(a), Fig. 6(b), and Table 3, and additionally include a PDrop-style progressive allocation baseline in Table 3.

  • LLM-Expressive Evidence (Fig. 5(a)). We replaced our scoring criterion with sensitivity-based scoring from ZOO-Prune and attention-based scoring from VisionZip. LAVIC achieves an MME score of 1795, compared with 1776 and 1788, respectively.

  • Query-Anchored Evidence (Fig. 6). We compared our query-anchor grouping with visual-similarity grouping from SparseVLM and text-guided grouping from VisionTrim. LAVIC achieves higher performance across POPE, SQA, and TextVQA under the same compression pipeline.

  • Evidence-Aware Capacity (Table 3). We compared our adaptive allocation with fixed allocation and a PDrop-style progressive schedule at matched token-reduction levels. LAVIC achieves the highest MME scores on both Qwen-VL-Chat and LLaVA-v1.5, supporting the effectiveness of evidence-aware layer-wise allocation.

AuthorsOpenReview ↗

Follow-up to Reviewer 3AFo

Dear Reviewer 3AFo,

We sincerely thank Reviewer 3AFo for recognizing the solid performance of our training-free framework with significantly fewer tokens, as well as its versatility across diverse models, input modalities, and settings.

We also appreciate the constructive comments on soundness, and we hope these clarifications address your concerns, and we would welcome any follow-up during the discussion period! If there are any non-technical concerns that need further clarification, please let us know specifically so that we can address them.

Best, The Authors

~Yusen_Peng1OpenReview ↗

More OCR-heavy benchmarks would be great

Thanks for your interesting work! Other than TextVQA, I wonder how this proposed method performs compared to baseblines, in more challenging OCR-heavy benchmarks such OCRBench, DocVQA, etc?

Reviewer_cWhfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LAVIC, a training-free visual-token compression method. It first ranks tokens by the separation of locally perturbed features after projection, then groups discarded information around query anchors and adapts layer-wise capacity using attention entropy. The evaluation covers multiple VLMs, image and video tasks, runtime, component analyses, and random-seed stability.

Critical strengths
  1. I am glad to see the paper examine which visual distinctions survive the projector instead of treating vision-side saliency as the final measure of importance. Although I have concerns about the interpretation of LLM-aligned evidence, the question itself is worthwhile. Figure 5 compares alternative scoring rules and the effects of removing high-score, low-score, and random tokens, which is more informative than a few qualitative examples alone.

  2. The reported 60-token configuration retaining 96.4% of average relative performance on LLaVA-v1.5 in Table 1 is an appealing result. The authors also test different architectures, high-resolution and multi-image settings, and video rather than tuning only one backbone. Tables 4–5 show several substantial gains at tight budgets, making the method worth further investigation.

Critical weaknesses

I have presented the weaknesses together with the relevant details to make it easier for the authors to address them during the rebuttal period. If my concerns are adequately addressed, I would be willing to raise my score.

  1. In my view, Equation (1) needs a stronger interpretation. Under a small-perturbation approximation, its expected score is proportional to the squared Frobenius norm of the projector’s local Jacobian. It directly measures how strongly the projector amplifies local variation, without measuring how the language model reads those directions. In particular, for a tokenwise linear projector g(v)=Wv and fixed isotropic noise, every token has the same expected score. This does not invalidate the nonlinear-projector experiments, but a larger score is not inherently more useful semantic evidence. Figure 5 is helpful; I would still like a direct comparison with Jacobian/local-sensitivity baselines at the same interface and whether the score predicts per-token, task-conditioned removal effects beyond the aggregate high/low-group comparison.

  2. I cannot reconstruct the layer-wise budget from the main method description. Equation (5) defines a scale but not how it determines the next token count, enforces the overall budget, or accounts for summary tokens. Figure 11 also labels one Mini-Gemini setting “Avg 887” while its final retained portion is 251, and Table 9 reports 887 in the Tokens column. These numbers therefore cannot simply be read as final retained counts. Please define whether counts are averaged over layers, examples, or something else, and provide the initial filtering budget, pruning layers, per-layer counts, and positional-encoding treatment. Otherwise, “using only 60 tokens” and budget matching against single-stage methods are easy to misinterpret.

  3. Appendix A says baselines are rerun when possible and otherwise taken from published results, but the tables do not identify provenance and configuration row by row. For a method combining staged pruning, adaptive budgets, and summary tokens, similar token counts alone do not guarantee similar compute; checkpoints, decoding settings, and attention implementations can also matter. Table 11’s FLOPs comparison with SparseVLM is useful but does not cover the main competing methods. I would like the strongest baselines rerun with a common model and evaluation pipeline at least in the central LLaVA setting, together with accuracy–FLOPs or accuracy–latency curves rather than reliance on one token-count column.

  4. Figure 6 and Table 2 provide useful grouping and summarization comparisons, but the Mean, Max, and Weighted variants in Table 2a do not have identical compute. Attention received by summaries in Table 2b also does not by itself establish that they preserve correct answer information. Appendix E.4 already shows that retained answer regions can lose discriminability when mixed with other OCR regions. I would hold the initial retained set, output token count, and layer-wise budget fixed, then compare query summaries with the same number of additional raw tokens and with randomized or zeroed summaries. This would more directly test whether anchor-specific organization helps beyond simply retaining additional content.

Minor Weaknesses:

  1. Equation (2) can produce no anchors for a one-token query or uniform associations, and Equation (4) has log N=0 when only one visual token remains. Figure 7 also includes M=1, although Equation (1) then has no pairwise differences and gives all-zero scores. Please state the fallbacks, numerical safeguards, and tie-breaking rules for these cases.

  2. The eight-seed results in Appendix E.3 are a strength, but Table 12 does not identify the budget configuration or provide paired intervals against baselines. Please add these details for the smaller aggregate gains at moderate compression and avoid implying uniform superiority across every configuration.

Questions for authors

For clarity and to facilitate the rebuttal process, I have already outlined the key strengths and weaknesses that informed my rating in the sections above. Please refer to the Strengths and Weaknesses sections for the detailed rationale. I would be willing to reconsider my rating if the authors adequately address my main concerns during the rebuttal.

3 comments
AuthorsOpenReview ↗

Response to Reviewer cWhf

We thank the reviewer for the constructive feedback. We have addressed the key concerns through additional analyses, experiments, and clarifications, and improved the figures for clarity.

 

1. Interpretation and Validation of LLM-Expressive Evidence.

  • Interpretation of Eq. (1). Eq. (1) measures the distinguishability of locally perturbed visual tokens after projection into the LLM-facing space. As the reviewer correctly notes, under a first-order approximation with small perturbations, the expected score is proportional to the squared Frobenius norm of the projector's local Jacobian. We therefore interpret the score as a measure of local distinguishability, rather than a direct measure of semantic importance or how the LLM utilizes visual information.

  • Comparison with Jacobian and Existing Criteria (Fig. 5(a)). We added a Jacobian Frobenius baseline and compared it with sensitivity-based scoring from ZOO-Prune and attention-based scoring from VisionZip under the same compression pipeline. LAVIC achieves an MME score of 1795, outperforming Jacobian Frobenius (1790), VisionZip (1788), and ZOO-Prune (1776).

  • Answer–Distractor Discrimination (Appendix E.1, Table 13). We compare correct and incorrect OCR candidates on 100 TextVQA samples using teacher-forced log-likelihoods, with 128 visual tokens selected by initial LLM-facing filtering only. LAVIC achieves the highest Answer Top-1 (76.0%) and Answer–Distractor Gap (1.9686), indicating that our score helps retain visual evidence useful for distinguishing correct answers from distractors.

 

2. Layer-Wise Budget Allocation and Visual-Token Accounting.

  • Visual-Token Accounting (Appendix A, Eq. 6). Reported average visual-token counts include retained and query-anchor summary tokens, averaged across decoder layers and evaluation samples.

  • Layer-Wise Budget Allocation (Sec. 3.3, Eq. 5; Appendix A). Eq. (5) specifies removals per pruning layer using evidence spread, the remaining removal budget, and remaining pruning layers. Appendix A clarifies how remaining removals are handled at the final layer.

  • Backbone-Specific Settings (Appendix A, Table 7). We added the initial visual-token counts, LLM-facing retention ratios, pruning layers, and positional encoding strategies for each backbone.

  • Clarification of Mini-Gemini Token Counts (Appendix A, Fig. 12). The reviewer-noted counts 887 and 251 differ: 887 is averaged across decoder layers, whereas 251 is retained after final compression (Appendix A, Fig. 12). We also corrected LLaVA-OneVision accounting from a single-tile reference (729 tokens) to the multi-tile average (1,918 tokens; Appendix Table 9).

 

3. Baseline Provenance and Performance–FLOPs Trade-off.

  • Baseline Provenance (Appendix Tables 8 and 9). Published baseline results are marked with ‡ to distinguish them from our evaluations.

  • Performance–FLOPs Analysis (Fig. 7(c), Appendix Table 12). We added a performance–TFLOPs trade-off analysis and report benchmark-specific accuracy and TFLOPs. TFLOPs are calculated using the actual layer-wise sequence lengths after pruning.

  • Aggressive Compression (64-Token Budget). LAVIC achieves 96.4% average relative performance at 1.06 TFLOPs, indicating a favorable trade-off under aggressive compression.

 

4. Effect of Query-Anchored Summarization.

  • Component Ablation (Table 4). We added ablations to test whether query-anchored summarization helps beyond retaining extra visual tokens, comparing removal of query-anchored evidence with replacement by zero vectors or raw Top-K tokens. On TextVQA, replacement reduces performance from 55.1 to 53.3 (zero vectors) or 54.9 (raw Top-K).

 

5. Edge Cases and Numerical Stability.

We added an Edge Cases and Numerical Stability subsection in Appendix A to clarify the following cases:

  • Fallback Rules (Appendix A). When M = 1, we use the difference between the original and perturbed projector outputs. If no query anchor is selected in Eq. (2), we select the query token with the highest association score.

  • Numerical Stability and Tie-Breaking (Appendix A). In Eq. (4), we use log(max(N, 2)) and a small ε for stability; ties favor the smaller original visual-token index.

 

6. Random-Seed Stability and Performance Claims.

  • Random-Seed Stability (Appendix Table 14). At 77.8% visual-token reduction, matching our main setting, the eight-seed evaluation shows standard deviations ≤ 0.14 and ranges ≤ 0.44 across three benchmarks, indicating limited seed variability.

  • Refinement of Performance Claims. We revised our claims to emphasize competitive overall performance, with more pronounced advantages under aggressive compression, rather than uniform superiority across compression settings.

AuthorsOpenReview ↗

Follow-up to Reviewer cWhf

Dear Reviewer cWhf,

We sincerely thank Reviewer cWhf for the structured and actionable feedbacks.

We hope these clarifications address your concerns, and we are glad to run any further tests you would find useful. We are happy to add any of the supplementary analyses above. If the new evidence adequately resolves the concerns, we would be grateful if you would consider it in your final assessment.

Best, The Authors

Thank you for the detailed response and additional experiments. The clarifications on token accounting, layer-wise budget allocation, and baseline provenance address some of my concerns, and the new experiment provides useful support for the scoring method. In my view, these results are encouraging, but they do not yet establish a clear advantage. While the rebuttal has clarified several issues, I will therefore maintain my original score.

Comments on the paper

2 comments
Area_Chair_V3VUOpenReview ↗

With the authors' rebuttal now posted, I would ask each reviewer to respond in the discussion thread.

Now that the authors' rebuttal has been posted, I ask each reviewer to respond in the discussion thread—especially on whether your original concerns have been addressed or remain unresolved—so that our discussion can meaningfully inform the final recommendation before the discussion period closes.

AuthorsOpenReview ↗

General Response

We thank all reviewers for their time and encouraging feedback. We appreciate their recognition of LAVIC's meaningful problem formulation and coherent framework (cWhf, w9xv), its effective training-free design (3AFo), and its competitive performance under aggressive compression across diverse VLM architectures and input settings (cWhf, 3AFo, w9xv).

Below, we summarize the main contribution and revisions made in response to their comments. All section, table, and figure references refer to the revised PDF, with changes highlighted in blue.

Problem Formulation and Evidence Validation (cWhf, w9xv). We frame visual-token compression as preserving LLM-aligned visual evidence, rather than relying solely on vision-side token importance. We clarified that our LLM-Expressive Evidence criterion measures the distinguishability of local visual variations after projection, rather than semantic importance directly, and further validated its effectiveness through Jacobian comparisons, token-removal analyses, and answer–distractor discrimination (Fig. 5; Appendix E.1).

Component-Level Validation (cWhf, 3AFo, w9xv). We expanded comparisons of our three components—LLM-Expressive Evidence, Query-Anchored Evidence, and Evidence-Aware Capacity—with alternative strategies from existing methods, and added component-wise ablations reporting both task performance and latency (Figs. 5–6; Tables 2–4).

Unified and Expanded Evaluation (3AFo). We unified the main evaluation across LLaVA-v1.5 and LLaVA-NeXT and expanded our evaluations to additional benchmarks and recent VLMs (Table 1; Tables 5–6; Appendices B–C). The results demonstrate LAVIC's clearer advantages under aggressive visual-token compression.

Efficiency (cWhf, w9xv). We added performance–FLOPs analyses, clarified baseline provenance, and reported computational overhead and inference latency (Fig. 7; Appendices B, D). We also confirmed through CUDA profiling that the Vanilla baseline uses PyTorch SDPA with the built-in Flash Attention CUDA kernel, against which LAVIC achieves a 2.03× prefill speedup, including its additional overhead. The results show that LAVIC achieves the highest average relative performance under aggressive compression while maintaining competitive computational cost.

Implementation and Robustness (cWhf, w9xv). We clarified visual-token accounting, layer-wise budgets, positional encoding, and optimized attention compatibility, and expanded robustness analyses (Sec. 3; Fig. 8; Appendices A, E).

Additional Evaluations (3AFo, w9xv). Encouraged by Reviewer 3AFo's suggestion to expand benchmark coverage and Reviewer w9xv's request for visual grounding evaluation, we commit to conducting additional experiments on VQA v2 and VizWiz and including the results in the camera-ready version.

 

We hope these additional results and revisions address the reviewers' concerns.

Best, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.