acceptodds
Under review as a conference paper at ICLR 2027

Looking Beyond Tokens: Context-Aware Visual Token Compression for MLLMs

Abstract

Multimodal large language models (MLLMs) suffer from heavy computational and memory overheads due to the excessive number of visual patch tokens. Existing visual token compression approaches typically rely on raw token importance scores or raw feature similarities, leading to redundant budget allocation or invalid merging across distinct objects. In this paper, we argue that reliable visual token compression should be guided by multimodal context, where token relations are jointly characterized by spatial structure and text-conditioned semantics rather than isolated visual features. Motivated by this insight, we propose Context-aware Visual Token Compression (CVTC), a novel training-free compression paradigm for MLLMs. CVTC employs a cascaded context coarsening mechanism to construct a multimodal context graph, progressively integrating spatial topology and text-conditioned relevance to guide context-aware token merging. Based on the propagated multimodal context, CVTC performs greedy contextual merging to identify tokens sharing coherent contextual relations, while aggregating their original visual features to construct the compressed representation. Extensively evaluated on various benchmarks across different backbones, CVTC achieves a favorable balance among performance, efficiency, and robustness. The codes are available in supplementary materials will be publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
token compression, efficient VLM, multi modal large language model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kcYWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Context-aware Visual Token Compression (CVTC), a training-free visual token compression method for MLLMs. The method addresses the limitations of existing token pruning and merging approaches that rely primarily on individual token importance or raw feature similarity, potentially causing information loss or incorrect merging across distinct semantic regions. CVTC introduces a two-stage compression framework. First, Spatial Context Coarsening (SCC) constructs a spatial graph and performs context-aware greedy merging through iterative graph contraction. Second, Text-conditioned Context Coarsening (TCC) incorporates text-conditioned relevance to preserve task-critical visual details while further compressing the spatial clusters. Experiments on multiple MLLM backbones and visual understanding benchmarks demonstrate that CVTC achieves competitive performance under different token budgets, with particularly notable improvements under aggressive compression ratios.

Critical strengths

The paper evaluates CVTC on three representative MLLM architectures, covering fixed-resolution, multi-resolution, and dynamic-resolution visual tokenization. The experimental results demonstrate competitive accuracy across various benchmarks, particularly under aggressive token compression. The ablation studies on the spatial and text-conditioned components, as well as graph topology, provide additional evidence supporting the effectiveness of the proposed design.

Critical weaknesses

The proposed contextual greedy merging requires iterative graph contraction, including repeated neighbor matching, graph updates, and cluster feature aggregation. This process introduces additional computational complexity that may become increasingly expensive as the number of visual tokens grows. In Table 4, even on LLaVA-v1.5-7B with only 576 original visual tokens, CVTC exhibits higher prefill latency than several competing methods. The paper does not provide sufficient evidence regarding the scalability of this iterative procedure to substantially longer visual sequences, such as those encountered in high-resolution images.

Questions for authors
  1. How does the inference efficiency of CVTC compare with existing methods on Qwen2.5-VL-7B? Table 4 reports efficiency results only on LLaVA-v1.5-7B, which has a relatively short visual token sequence. It is important to understand how the computational overhead scales with longer visual sequences. Could the authors provide prefill latency and end-to-end inference latency comparisons on Qwen2.5-VL-7B under the same token budgets as competing methods?

  2. What explains the inconsistent experimental results reported in Tables 5 and 6? Tables 5 and 6 report different results for the full CVTC method under the same LLaVA-v1.5-7B setting with 64 retained tokens. Specifically, Table 5 reports an MME score of 1674 and a three-benchmark average retention of 93.6%, whereas Table 6 reports an MME score of 1640 and an average retention of 92.9%. Could the authors clarify the reasons for these discrepancies?

Reviewer_N6pHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses visual token compression for MLLMs, arguing that token importance or raw feature similarity alone leads to redundant budget allocation and to merging across objects. To address this, it proposes CVTC, a training-free compressor that merges spatially adjacent patches based on contextual states obtained by propagation on a spatial graph (SCC), and then preserves patches relevant to the question through a hybrid graph (TCC).

Critical strengths
  • S1. CVTC decides which tokens to merge using the contextual states, while it constructs the merged tokens from the original features. Restricting merges to spatially adjacent clusters keeps each token a connected region, making the design simple without training or attention maps of the LLM.

  • S2. The ablations isolate the main design choices of SCC. Replacing the spatial graph with a kNN graph in the feature space and removing the propagation both degrade the accuracy.

Critical weaknesses

W1. Justification of SCC.

  • This paper does not explain theoretically why contextual propagation improves the merging decision. Since the edge weight is the exponential of the cosine similarity, a dissimilar neighbor still receives a large share of the transition weight, and the propagation may mix features across object boundaries rather than separate them.
  • The effect of SCC is verified only through the downstream accuracy. The authors are encouraged to measure how faithfully the merged tokens represent their patches, compared with other merging methods.

W2. Notion of context and contribution of TCC.

  • Apart from the spatial neighborhood, the context conditioned on the text in TCC is not distinguished from preserving tokens relevant to the question. CDPruner [A], which selects tokens with similarity conditioned on the instruction, should be discussed and compared.
  • Removing the propagation in TCC changes the accuracy only marginally (Table A4), and the results are insensitive to the ratio of protected patches (Table A5). This leaves it unclear whether TCC captures meaningful context, which replacing the protected patches with the tokens selected by CDPruner would clarify.

[A] Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs, NeurIPS 2025

Questions for authors
  • Q1. In Table 4, CVTC with the smaller budget is slower and less accurate than HoloV with the larger budget. Why does reducing the budget barely reduce the total inference time of CVTC, and what advantage does CVTC offer in the trade-off between accuracy and latency?

  • Q2. What are the latency and the FLOPs of the compressor itself, and how do they scale on LLaVA-NeXT and Qwen2.5-VL with more visual tokens?

  • Q3. Why does CVTC perform worse than the compared methods on TextVQA, although it is designed to preserve fine details relevant to the task?

  • Q4. Does the improvement on VizWiz come from a change in the rate of answering unanswerable questions, and does the advantage under moderate budgets hold when VizWiz is excluded from the average?

  • Q5. The reported results of VisionTrim are lower than those in the original paper, while the uncompressed results match. Does VisionTrim compress tokens only before the LLM, or does it follow the averaged budget of the original paper?

Reviewer_Xsr8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Context-aware Visual Token Compression (CVTC), a training-free method that guides token compression using contextual relations rather than raw feature importance or pairwise similarity. CVTC combines spatial context coarsening (SCC) and text-conditioned context coarsening (TCC) to preserve spatial structure and task-relevant visual details. Experiments across three MLLM backbones and multiple benchmarks show competitive results, achieving 96.0% performance retention on LLaVA-v1.5-7B with 64 tokens, 94.1% on LLaVA-NeXT-7B with 320 tokens, and 79.7% on Qwen2.5-VL-7B at a 5% token retention ratio.

Critical strengths
  1. Extensive experiments. The paper evaluates CVTC across multiple MLLM architectures, including fixed-resolution, AnyRes, and dynamic-resolution models. Comparisons with existing methods under different token budgets provide a relatively comprehensive evaluation.

  2. Effective component design. The ablation studies in Table 5 show that both SCC and TCC improve performance over the baseline. Combining them achieves the highest average performance on GQA, POPE, and MME, supporting the effectiveness of the two components.

  3. Training-free design. CVTC requires no additional training data or parameter updates and can be directly applied to existing MLLMs, making it practical for deployment across different models.

Critical weaknesses
  1. Unclear figures. Some figures, particularly the overall framework illustration, are difficult to follow. The roles of individual components and their interactions are not clearly presented, making the proposed method less intuitive.

  2. Limited novelty. CVTC shares similarities with VisionTrim, as both leverage text-related global information and spatially aware local detail preservation. Although their implementations differ, the paper does not clearly distinguish its key contributions from VisionTrim, raising concerns about its novelty.

  3. Limited practical efficiency gains. Although CVTC reduces the number of visual tokens, graph construction, context propagation, and iterative greedy merging introduce additional computational overhead. As shown in Table 4, CVTC does not consistently achieve better inference efficiency than existing methods.

  4. Numerous hyperparameters. CVTC involves several hyperparameters, including spatial neighborhood structure, propagation depth, random-walk weights, protected patch ratio, and token budget allocation. More comprehensive sensitivity analyses are needed to assess its robustness across different settings.

  5. Missing related work. Several relevant studies on visual token compression are not discussed, including:

    • TimeChat-Online: 80% Visual Tokens Are Naturally Redundant in Streaming Videos
    • StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understanding
Questions for authors

Have the authors considered evaluating CVTC on multi-image reasoning or video understanding tasks? The current experiments mainly focus on single-image benchmarks. For multi-image and video inputs, visual tokens involve not only spatial relationships but also cross-image and temporal dependencies. It remains unclear whether the proposed local spatial graph can effectively capture these relationships.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.